Lead Data Engineer, Databricks

EXL

. Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture .

Posted 11/5/2026full-timeNew York City • New York • United StatesSenior💰 $130,000 - $160,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing enterprise-scale data platforms using Databricks Lakehouse architecture, along with strong leadership in data engineering practices and team mentorship. Proficient in developing scalable ETL/ELT pipelines and ensuring data governance and compliance across cloud-native solutions.

Highest-signal resume keywords

Databricks Lakehouse ArchitectureETL/ELT Data Pipeline DevelopmentSpark/PySpark Data EngineeringData Governance FrameworksCloud Data Platform Modernization

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

DatabricksSparkPySparkPythonSQLDelta LakeData Lake ArchitectureReal-Time StreamingData Quality FrameworksDimensional Modeling

Soft Skills

Team LeadershipMentoringCollaborationCommunicationProblem-Solving

Tools & Technologies

AzureAWSGCPGitTerraformDockerKubernetesAzure DevOpsJenkinsKafka

Certifications & Qualifications

Databricks Certified Data Engineer ProfessionalAzure Data Engineer AssociateAWS Data Analytics SpecialtyGoogle Professional Data Engineer CertificationTOGAF

Industry Keywords

Data GovernanceData TransformationData ModernizationData Protection StandardsOperational Excellence

Tech Stack

Tools & technologies

ApacheAWSAzureCloudDockerETLGoogle Cloud PlatformJenkinsKafkaKubernetesPySparkPythonScalaShell ScriptingSnowflakeSparkSQLTerraformUnityVault

About the role

Key responsibilities & impact

Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture
Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing
Define enterprise data models, ingestion frameworks, and transformation strategies
Lead implementation of Delta Lake, Unity Catalog, and data governance frameworks
Build and optimize scalable Spark and PySpark solutions handling large-volume datasets
Drive cloud data platform modernization and migration initiatives
Serve as technical lead for Databricks and Data Engineering programs
Establish engineering best practices, coding standards, and architecture guidelines
Conduct architecture reviews, code reviews, and solution assessments
Mentor Data Engineers, Technical Leads, and junior team members
Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders
Design cloud-native data solutions on Azure, AWS, or GCP
Implement secure, scalable, and resilient data architectures
Optimize data platform costs, performance, reliability, and availability
Lead CI/CD and Infrastructure-as-Code implementation for data workloads
Define monitoring, observability, and operational excellence practices
Establish data governance, lineage, cataloging, and security controls
Ensure compliance with organizational and regulatory requirements
Implement data quality frameworks and monitoring solutions
Collaborate with security and compliance teams to enforce data protection standards
Partner with business leaders to understand strategic data requirements
Translate business requirements into scalable technical solutions
Drive delivery planning, risk management, and resource coordination
Present architecture decisions and technical recommendations to leadership teams

Requirements

What you’ll need

12-15 years of overall IT experience
Minimum 6+ years of hands-on Databricks experience
7+ years of experience in Spark/PySpark-based data engineering
Experience designing enterprise-scale Data Lake and Lakehouse solutions
Strong experience leading large data transformation and modernization programs
Experience handling structured, semi-structured, and unstructured data processing
Experience supporting analytics, AI/ML, and reporting ecosystems
Proven experience managing technical teams and driving architectural decisions
Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field (preferred)
Databricks Certified Data Engineer Professional (preferred)
Azure Data Engineer Associate (preferred)
AWS Data Analytics Specialty (preferred)
Google Professional Data Engineer Certification (preferred)
TOGAF or Cloud Architecture certifications are a plus
Proficiency with Databricks Lakehouse Platform, Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables, and Databricks Asset Bundles
Proficiency with Apache Spark, PySpark, Spark SQL, Kafka, real-time streaming architectures, and data lake/lakehouse architectures
Strong experience with one or more of Azure, AWS, or GCP data platforms
Proficiency in Python and SQL; PySpark; Scala preferred; shell scripting
Experience with Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes
Knowledge of dimensional modeling, star and snowflake schema, Data Vault, master data management, and data governance frameworks