Lead Data Engineer

Emory University

. Lead technical delivery of source system onboarding and data integration initiatives from source analysis and design through development, validation, deployment, and production support .

Posted 10/26/2026full-timeAtlanta • United StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and optimizing ETL and ELT pipelines using SQL, Python, and PySpark, while ensuring adherence to data governance and security standards. Proven ability to lead technical teams, mentor engineers, and manage complex data integration initiatives across multiple platforms.

Highest-signal resume keywords

ETL/ELT Pipeline DesignPython ProficiencySQL ExpertiseApache Spark (PySpark) ExperienceData Governance Knowledge

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

ETL DevelopmentELT DevelopmentData ModelingData TransformationIncremental Load StrategiesData Quality ChecksVersion Control (Git)Data Pipeline OrchestrationData IntegrationTechnical Documentation

Soft Skills

Analytical SkillsProblem-SolvingCommunication SkillsCollaboration SkillsMentoring

Tools & Technologies

Microsoft FabricAzure Synapse ServerlessDatabricksAirflowAzure Data Factory

Industry Keywords

Data GovernanceHIPAA ComplianceBiomedical InformaticsMetadata ManagementData Quality Frameworks

Tech Stack

Tools & technologies

AirflowApacheAzureCloudETLPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Lead technical delivery of source system onboarding and data integration initiatives from source analysis and design through development, validation, deployment, and production support
Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards
Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, and Microsoft Fabric or comparable cloud technologies
Analyze source schemas, business rules, and dependencies; define source-to-target mappings, incremental load strategies, and reconciliation requirements
Guide implementation of bronze, silver, and gold data layers, including data transformation, dimensional modeling, and curated datasets
Establish and review standards for coding, testing, documentation, version control, and deployment
Conduct code reviews and mentor data engineers
Coordinate engineering tasks and dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source system owners
Communicate delivery progress, risks, and resource needs
Implement data quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures
Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration
Apply data governance and security standards, including access controls, metadata, lineage, sensitive-data handling, and HIPAA requirements where applicable
Translate researcher and stakeholder requirements into technical specifications and explain design choices and data limitations
Create and maintain technical documentation, operational runbooks, source mappings, and support procedures
Coordinate production deployments and knowledge transfer
Support existing Unified Data Platform and Enterprise Business Intelligence integrations and services
Evaluate emerging technologies and develop proofs of concept
Lead technical teams, mentor analysts, develop reporting infrastructure, and perform related duties

Requirements

What you’ll need

Bachelor's degree in a related field and seven years of related experience, OR an equivalent combination of education, training, and experience
Strong experience designing and maintaining data pipelines and ETL/ELT processes across multiple source systems
Proficiency in Python, SQL, and Apache Spark (PySpark)
Experience with modern data platforms such as Azure Fabric (preferred), Azure Synapse Serverless, or Databricks
Ability to work with large, complex, and distributed datasets
Solid understanding of data modeling concepts and best practices
Familiarity with data governance, metadata management, and data quality frameworks
Ability to lead technical design discussions, review code, mentor engineers, and guide delivery across multiple workstreams
Experience with CI/CD practices and proficiency with version control systems like Git
Familiarity with data pipeline orchestration and monitoring tools like Airflow, Azure Data Factory, or similar frameworks
Understanding of biomedical informatics methodologies is a plus
Strong analytical, problem-solving, communication, and collaboration skills
Ability to prioritize engineering work, identify delivery risks, and adapt to changing business and platform requirements
Ability to follow HIPAA regulations and institutional data governance and security standards
Candidates must work from an approved U.S. location; Emory does not approve NJ, AK, HI, any U.S. Territories, or locations outside the United States as primary work locations

Benefits

Comp & perks

Remote work with only occasional visits to an Emory University location
Eastern (EST) time zone business hours may apply
Equal opportunity employer
Reasonable accommodations for qualified individuals with disabilities
Access to Emory University programs and facilities without discrimination