Data Engineer, Databricks

EXL

. Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture .

Posted 11/5/2026full-timeNew York City • New York • United StatesSeniorLead💰 $100,000 - $135,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing enterprise-scale data platforms using Databricks Lakehouse architecture, along with strong capabilities in leading data transformation initiatives and managing technical teams. Proficient in building scalable ETL/ELT pipelines and ensuring data governance and compliance across cloud platforms.

Highest-signal resume keywords

Databricks Lakehouse ArchitectureSpark/PySpark Data EngineeringCloud Platform Expertise (Azure, AWS, GCP)Data Governance FrameworksCI/CD and Infrastructure-as-Code Implementation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

DatabricksSparkPySparkSQLPythonShell ScriptingDimensional ModelingData Lake ArchitectureETL/ELT PipelinesData Transformation

Soft Skills

LeadershipMentoringCollaborationCommunicationRisk Management

Tools & Technologies

Azure DevOpsGitJenkinsTerraformDockerKubernetesKafkaDatabricks SQLDelta LakeUnity Catalog

Certifications & Qualifications

Bachelor's Degree in Computer ScienceMaster's Degree in EngineeringRelevant Professional Certifications

Industry Keywords

Data GovernanceData Quality FrameworksReal-Time Streaming ArchitecturesMaster Data ManagementData Lakehouse Solutions

Tech Stack

Tools & technologies

ApacheAWSAzureCloudDockerETLGoogle Cloud PlatformJenkinsKafkaKubernetesPySparkPythonScalaShell ScriptingSnowflakeSparkSQLTerraformUnityVault

About the role

Key responsibilities & impact

Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture
Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing
Define enterprise data models, ingestion frameworks, and transformation strategies
Lead implementation of Delta Lake, Unity Catalog, and data governance frameworks
Build and optimize scalable Spark and PySpark solutions for large-volume datasets
Drive cloud data platform modernization and migration initiatives
Serve as technical lead for Databricks and Data Engineering programs
Establish engineering best practices, coding standards, and architecture guidelines
Conduct architecture reviews, code reviews, and solution assessments
Mentor Data Engineers, Technical Leads, and junior team members
Collaborate with Enterprise Architects, Product Owners, Data Scientists, and business stakeholders
Design secure, scalable, and resilient cloud-native data solutions on Azure, AWS, or GCP
Optimize platform costs, performance, reliability, and availability
Lead CI/CD and Infrastructure-as-Code implementation for data workloads
Define monitoring, observability, and operational excellence practices
Establish data governance, lineage, cataloging, security controls, and data quality frameworks
Ensure compliance with organizational and regulatory requirements
Partner with business leaders to understand strategic data requirements
Translate business requirements into scalable technical solutions
Drive delivery planning, risk management, and resource coordination
Present architecture decisions and technical recommendations to leadership teams

Requirements

What you’ll need

12-15 years of overall IT experience
Minimum 6+ years of hands-on Databricks experience
7+ years of experience in Spark/PySpark-based data engineering
Experience designing enterprise-scale Data Lake and Lakehouse solutions
Strong experience leading large data transformation and modernization programs
Experience handling structured, semi-structured, and unstructured data processing
Experience supporting analytics, AI/ML, and reporting ecosystems
Proven experience managing technical teams and driving architectural decisions
Databricks Lakehouse Platform, Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables, and Databricks Asset Bundles
Apache Spark, PySpark, Spark SQL, Kafka, real-time streaming architectures, and data lake/lakehouse architectures
Strong experience in one or more of Azure, AWS, or GCP cloud platforms
Python, SQL, PySpark, and shell scripting; Scala preferred
Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes
Dimensional modeling, Star and Snowflake Schema, Data Vault, Master Data Management, and data governance frameworks
Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field is preferred
Relevant professional certifications are preferred
Location: US