Databricks Migration Engineer

A.C.Coy Company

. Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake) .

Posted 9/30/2026full-timeRemote • Virginia • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in leading data migration projects, specifically from SQL Server to Databricks Lakehouse, while implementing scalable ETL/ELT frameworks and optimizing data models for performance. Proficient in data governance, security models, and effective communication with diverse audiences.

Highest-signal resume keywords

Databricks LakehousePySparkData EngineeringCloud PlatformsDimensional Modeling

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

ETL FrameworksDelta LakeData WarehousingSQL OptimizationData Security ModelsIncremental Data LoadsMetadata-Driven IngestionZ-OrderingMaterialized ViewsData Quality Frameworks

Soft Skills

Excellent CommunicationDetail-OrientedOrganizational Skills

Tools & Technologies

Databricks WorkflowsPower BIAzure Active DirectoryMicrosoft Entra ID

Industry Keywords

Data System DevelopmentCross-Functional Data ProjectsCloud-Native DatabasesDistributed Compute PlatformsPosition of Public Trust Clearance

Tech Stack

Tools & technologies

ApacheAWSAzureCloudETLGoogle Cloud PlatformPySparkSparkSQLUnity

About the role

Key responsibilities & impact

Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake)
Translate relational data warehousing paradigms into scalable, distributed Lakehouse frameworks across Bronze, Silver, and Gold layers
Design reusable ETL/ELT frameworks using PySpark, Delta Live Tables, and Databricks Workflows
Architect and refine Gold Layer dimensional models and star schemas for Power BI performance
Optimize Databricks SQL Warehouses for high-concurrency, low-latency Power BI queries
Implement Z-Ordering, data skipping, liquid clustering, and materialized views
Define governance standards for cluster sizing, auto-scaling, and serverless SQL compute
Monitor Databricks Unit consumption and identify cost-saving opportunities
Establish Delta Lake partitioning and file size management practices
Design and implement Unity Catalog data security models
Enforce row-level and column-level security policies
Align Lakehouse security with Azure Active Directory/Microsoft Entra ID and RBAC standards
Serve as primary technical lead through pair-programming sessions, workshops, and code reviews
Create technical documentation, architecture diagrams, design patterns, and optimization playbooks
Build knowledge transfer processes for post-migration team self-sufficiency
Communicate with technical and non-technical audiences
Complete tasks in an organized, timely, and detail-oriented manner

Requirements

What you’ll need

Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field
5+ years of experience in Data Engineering, Data System Development or related roles
5+ years of experience with Cloud platforms (e.g. Azure, AWS, GCP)
1+ year leading complex, cross-functional data projects and technical teams
Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
Excellent communication skills
Must be a US Citizen and be able to obtain a Position of Public Trust Clearance
Must have resided in the US for the last 5 years and not have traveled outside the US for a combined total of 6 months or more in last 5 years

Benefits

Comp & perks

Contract-to-Hire position
Remote work arrangement
East Coast time zone candidates preferred