Lead/Sr AI Engineer, Azure Data Bricks

Enable Data

. Build production-ready AI applications and deploy them on Azure Databricks and Azure cloud infrastructure .

Posted 9/19/2026full-timeRemote • IndiaSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and deploying AI applications on Azure Databricks, with strong proficiency in Python, SQL, and MLOps best practices. Capable of designing end-to-end data pipelines and optimizing performance while ensuring responsible AI practices.

Highest-signal resume keywords

Azure DatabricksPython/PySparkSQL ProficiencyMLOps Best PracticesCI/CD Pipeline Implementation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

ETL/ELT WorkflowsData Pipeline DesignMachine Learning DeploymentGenerative AI ModelsVector Search ImplementationLogging and MonitoringPerformance OptimizationScenario-Based TestingPrompt EngineeringData Ingestion

Soft Skills

CollaborationProblem-SolvingCommunication

Tools & Technologies

Azure CloudAzure Data LakeAzure FunctionsAzure OpenAIAzure DevOpsMLflowTerraformFastAPIReactClaude Vision API

Industry Keywords

MLOpsResponsible AIData GovernanceAI Safety GuardrailsRetrieval Augmented Generation

Tech Stack

Tools & technologies

AirflowAzureCloudETLPandasPySparkPythonReactSQLTerraformUnityVault

About the role

Key responsibilities & impact

Build production-ready AI applications and deploy them on Azure Databricks and Azure cloud infrastructure
Design end-to-end data and AI pipelines using Azure Databricks
Develop ETL/ELT workflows with Python/PySpark and SQL
Implement CI/CD pipelines for Databricks jobs, notebooks, and workflows
Integrate Databricks with Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions, and related services
Optimize jobs for performance, cost, and reliability
Build reusable, modular code
Collaborate with data scientists, platform engineers, and business stakeholders to move models from experimentation to production
Implement logging, monitoring, and error handling for production pipelines
Develop and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines, for NLP, computer vision, and predictive analytics
Fine-tune LLMs using LoRA/QLoRA and integrate them with Azure OpenAI or Hugging Face models
Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search
Apply responsible AI practices, including bias detection and model governance
Ingest data into Databricks from sources such as SAP and website scraping
Write scenario-based and edge-case tests and finalize code changes and acceptance criteria with stakeholders
Send custom notifications from Databricks using APIs and custom Teams channel webhooks
Build chat UI interfaces and Claude-based applications
Use graphs to detect rule dependencies and circular references
Reengineer code and map it to user requirements
Implement AI safety guardrails and caching mechanisms

Requirements

What you’ll need

Experience with Azure Databricks (jobs, workflows, clusters, Unity Catalog preferred)
Strong Python skills, especially PySpark rather than only pandas
SQL proficiency, including complex joins, window functions, and analytical queries
Azure Cloud knowledge, including ADLS Gen2, ADF, Key Vault, and IAM concepts
Pipeline orchestration and deployment experience, including CI/CD and environment promotion
Experience with Azure DevOps
Strong understanding of ML lifecycle and MLOps best practices
Experience deploying models using MLflow or similar frameworks
Experience with ML and Generative AI workloads on Databricks
Experience with RAG, embeddings, or inference pipelines
Knowledge of Terraform, ARM, or Bicep
Knowledge of Databricks Asset Bundles
Knowledge of Airflow or ADF orchestration
Production monitoring and cost optimization experience
Knowledge of LangChain or similar frameworks
Experience with Azure AI services, including Azure Machine Learning and Azure Cognitive Services
Knowledge of Databricks Unity Catalog permissions, OBO tokens, and Service Principals
Ability to build and deploy Databricks Apps using Visual Studio Code
Ability to use FastAPI and React to build minimal single-page applications
Prompt engineering skills, including system and user prompts
Basic knowledge of token economics and prompt rule enforcement
Ability to use Claude Vision API and standard text processing
Understanding of Retrieval Augmented Generation, including AI Search indexes and hierarchical chunking
Knowledge of basic AI safety guardrails and caching mechanisms