Lead Data Engineer

Capital Group

. Set the data engineering strategy and roadmap for CSGT, including Lakehouse architecture on Databricks and AWS .

Posted 9/29/2026full-timeUnited StatesSenior💰 $201,683 - $342,072 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in data engineering strategy, including Lakehouse architecture on Databricks and AWS, with strong capabilities in Python, SQL, and data governance. Proven ability to lead complex data initiatives, implement automated testing, and ensure data quality and security across platforms.

Highest-signal resume keywords

Data Engineering StrategyDatabricks on AWSPython and SQL ProficiencyData Quality and GovernanceApache Airflow Orchestration

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonSQLDatabricksPySparkDelta LakeDbtApache AirflowData ModelingCI/CDData Quality

Soft Skills

Technical LeadershipCommunicationInfluencing Without AuthorityMentoringProblem-Solving

Tools & Technologies

Databricks GenieTerraformDatadogDeequLakehouse Monitoring

Industry Keywords

Investment Management DataMulti-Asset Portfolio ConstructionAI ApplicationsSemantic MetadataGoverned Data Access

Tech Stack

Tools & technologies

AirflowApacheAWSPostgresPySparkPythonSQLTerraformUnity

About the role

Key responsibilities & impact

Set the data engineering strategy and roadmap for CSGT, including Lakehouse architecture on Databricks and AWS
Make and explain decisions on scalability, security, reliability, and cost; influence standards and practices across technology teams
Design and build ingestion, transformation, and serving pipelines with Databricks, PySpark, Delta Lake, dbt, and Airflow
Create reusable data engineering patterns and frameworks
Analyze structured and unstructured datasets and fit them into platform data domains and subject areas
Own complex, cross-team data initiatives from requirements through production support, including estimates, sequencing, dependencies, and cost
Lead AI-first engineering by translating business outcomes into specifications and directing AI agents to plan, build, test, and document changes
Build reusable agent workflows, skills, and tool integrations for profiling, source-to-target mapping, pipeline and test generation, schema-change analysis, and incident investigation
Define agent approval boundaries, activity review, and traceability
Curate Unity Catalog metadata, lineage, business definitions, semantic models, and access controls for AI applications
Connect governed data to Databricks Genie and other AI applications used by investment professionals
Define evaluation datasets and acceptance criteria for agents and AI-generated SQL and code
Define platform-wide testing strategy and approve quality metrics before release
Build data quality, reconciliation, freshness, observability, recovery, security, and policy controls into automated testing and CI/CD
Partner with investment professionals and product managers on product vision and business outcomes
Lead design and code reviews and direct day-to-day work of engineers on initiatives
Guide engineers through complex data and performance issues
Teach engineers to inspect and challenge AI-generated work
Share reusable patterns and context through internal and external forums
Help managers identify strengths and development needs

Requirements

What you’ll need

10+ years of experience in data or software engineering, including technical leadership of complex production data platforms delivered across multiple teams
Strong hands-on Python and SQL skills
Sound software design judgment
Deep understanding of distributed data processing, query performance, and automated testing
Production experience with Databricks on AWS, including PySpark, Delta Lake, Unity Catalog, Databricks Jobs, Databricks SQL, and Databricks Asset Bundles
Understanding of security, access, and cost implications of technical designs
Experience orchestrating production pipelines with Apache Airflow, including Astronomer
Experience building tested transformations with dbt, including reliable retries, backfills, and dependency management
Strong data modeling and governance experience, including dimensional and time-series models, slowly changing dimensions, bi-temporal history, data contracts, lineage, and semantic metadata
Experience implementing data quality, observability, and CI/CD for data platforms using tools such as Deequ, dbt tests, Lakehouse Monitoring, Datadog, Terraform, and Harness
Experience preparing governed data for AI through natural-language-to-SQL tools such as Databricks Genie, semantic metadata, or other governed data-access patterns
Proficiency using AI coding agents for specifications, context, testing, and source-control review
Ability to evaluate AI-generated output with representative test cases, regression tests, execution traces, and human review
Understanding of prompt injection, sensitive data handling, least-privilege access, approval boundaries, and audit trails
Ability to lead architecture discussions, influence without formal authority, develop other engineers, and explain technical choices and trade-offs clearly
Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience
Preferred: experience with investment management data or multi-asset portfolio construction
Preferred: experience building LLM applications or agent workflows calling tools and APIs
Preferred: familiarity with Model Context Protocol (MCP)
Preferred: experience with PostgreSQL, SQL Server, Lakebase, or modernizing legacy data platforms onto a Lakehouse

Benefits

Comp & perks

Competitive salary
Individual annual performance bonus
Capital’s annual profitability bonus
Retirement plan where Capital contributes 15% of eligible earnings
Generous time-away and health benefits from day one
Flexible work options
2-for-1 matching gifts for charitable contributions
Opportunity to secure annual grants for charitable organizations
On-demand professional development resources
Compensation and benefits plans, subject to plan guidelines, restrictions, and vesting requirements