Senior Data Engineer

Protective Life

. Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Apache Spark, and Delta Lake .

Posted 9/18/2026full-timeRemote • Alabama • United StatesSenior💰 $84,000 - $128,667 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and maintaining production data pipelines using Python, SQL, and Databricks, with a strong focus on data quality, governance, and orchestration. Proficient in implementing multi-layer lakehouse architectures and ensuring data observability and compliance.

Highest-signal resume keywords

Python ProgrammingSQL DevelopmentDatabricks ExperienceData Pipeline OrchestrationData Quality Assurance

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentDelta Lake ProcessingDimensional ModelingIncremental Load PatternsData Quality TestingDbt TransformationsGit-based Source ControlCI/CD ImplementationData Ingestion from APIsTroubleshooting Data Failures

Soft Skills

Technical CommunicationCollaboration with Stakeholders

Tools & Technologies

DatabricksApache SparkDagsterAzure DevOpsUnity CatalogAirflowAzure Data FactoryDltFivetranMeltano

Certifications & Qualifications

Databricks Certification

Industry Keywords

Cloud Data PlatformMedallion ArchitectureData GovernanceData ContractsData Observability

Tech Stack

Tools & technologies

AirflowApacheAzureCloudPythonSparkSQLUnityVault

About the role

Key responsibilities & impact

Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Apache Spark, and Delta Lake
Build Bronze-layer ingestion from APIs, relational databases, flat files, cloud storage, and SaaS platforms using dlt and Databricks-native ingestion
Develop Silver-layer transformations in dbt and Python over Delta Lake to cleanse, standardize, deduplicate, validate, conform, and enrich data
Create Gold-layer data products including dimensional models, slowly changing dimensions, fact and bridge tables, aggregates, and serving tables
Produce curated feature and training tables for ML engineering that are versioned and reproducible
Author and maintain data contracts using the Open Data Contract Standard
Implement data quality tests for uniqueness, not-null, referential integrity, accepted values, freshness, and business rules
Orchestrate ingestion and transformation in Dagster Cloud across development, branch, and production deployments
Apply governance through Unity Catalog and manage credentials through Azure Key Vault
Implement incremental and merge-based Delta Lake processing and tune Spark jobs, table layouts, and compute
Troubleshoot production failures, data-quality issues, source-system changes, late-arriving or duplicate data, backfills, and recovery
Participate in the on-call rotation and perform root-cause analysis
Build and maintain CI/CD for data assets in Azure DevOps
Instrument pipelines for freshness, volume, quality, latency, and cost observability with SLA/SLO alerting
Follow least-privilege access, secrets management, change management, and audit controls
Participate in code reviews and document architecture, runbooks, and data products
Collaborate with data architects, analysts, product owners, and business stakeholders to translate requirements into maintainable data solutions

Requirements

What you’ll need

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered
3+ years building and supporting production data pipelines in a cloud data platform environment
Strong hands-on Python and SQL
Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns
Practical understanding of medallion / multi-layer lakehouse design
Experience ingesting data from APIs, relational databases, files, or SaaS applications, including incremental and state-management problems
Working knowledge of dimensional modeling, ELT design patterns, and data quality practice
Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar
Git-based source control, pull request review, automated testing, and CI/CD; Azure DevOps or comparable
Experience troubleshooting production data failures, performance bottlenecks, and source-system changes
Experience with pipeline monitoring and alerting, including freshness or quality SLAs
Ability to explain technical designs and trade-offs to technical and non-technical partners
Databricks certification or equivalent demonstrated depth preferred
Unity Catalog experience preferred
dbt on Databricks or another Spark transformation framework preferred
Python-based modeling frameworks over Delta Lake, Type 2 history, surrogate keys, and merge strategies preferred
Experience with dlt, Airbyte, Meltano, or Fivetran preferred
Dagster experience with assets, asset checks, sensors, schedules, and branch deployments preferred

Benefits

Comp & perks

Comprehensive health, dental, and vision insurance
Mental health benefits
Employee assistance program
Paid time off
Paid parental leave
Short-term disability
Cultural observance day
Contributions to healthcare accounts
Pension plan
401(k) plan with Company matching
ProHealth Rewards wellness platform with cash rewards
Annual incentive based on individual and Company performance