Senior Databricks Platform Engineer

ICF

. Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals .

Posted 10/13/2026full-timeRemote • Virginia • United StatesSenior💰 $98,614 - $202,850 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and optimizing Azure Databricks workspaces and data pipelines, with a strong focus on data governance, security, and performance troubleshooting. Proficient in implementing medallion architecture and utilizing big data technologies for effective data management and transformation.

Highest-signal resume keywords

Azure Databricks ExpertiseApache Spark ProficiencyPython ProgrammingData Governance ExperienceCI/CD Implementation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Azure DatabricksApache SparkPythonSQLDelta LakeETL/ELT Pipeline DesignData ModelingTerraformGitAgile Methodologies

Soft Skills

Stakeholder CommunicationCollaborationProblem-SolvingPerformance TroubleshootingDocumentation

Tools & Technologies

Azure Data FactoryAzure Data Lake StorageAzure Synapse AnalyticsGitHub ActionsDatabricks Asset Bundles

Certifications & Qualifications

Databricks Certification

Industry Keywords

Medallion ArchitectureChange Data CaptureData Quality ChecksDisaster RecoveryNIST Compliance

Tech Stack

Tools & technologies

AirflowApacheAWSAzureCloudDockerETLJavaScriptKafkaKubernetesNode.jsOraclePySparkPythonScalaSparkSQLTerraformUnity

About the role

Key responsibilities & impact

Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals
Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark
Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns
Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage
Develop frameworks and interfaces for governed training, inference, and reporting data across agency initiatives
Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools
Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services
Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and conduct root cause analysis
Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities
Participate in Agile planning, code reviews, demos, and retrospectives
Document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories
Explain architecture, trade-offs, risks, and operational impacts to technical and nontechnical stakeholders
Collaborate across data engineering, AI, cloud, security, application, and operations teams

Requirements

What you’ll need

BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience
More than 5 years of data engineering experience
At least 3 years specifically supporting Azure Databricks and related Azure data services
Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala
Experience designing and optimizing large-scale ETL/ELT pipelines
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies
Experience managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization
Experience with CI/CD or infrastructure as code using Git, Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable technologies
Experience with Agile delivery, data governance, data security, performance troubleshooting, and stakeholder communication
Ability to satisfy FDIC personnel eligibility and background-investigation requirements
U.S. Citizenship required
Must reside in the U.S., be authorized to work in the U.S., and perform all work in the U.S.
Preferred: experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse
Preferred: Databricks certification or comparable demonstrated experience
Preferred: streaming technologies including Kafka, Confluent, and Structured Streaming
Preferred: medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms
Preferred: Generative AI, NLP, RAG, vector stores, agents, or Databricks AI/ML workloads
Preferred: MLflow, model deployment pipelines, AWS SageMaker, AWS Bedrock, or cross-cloud architectures
Preferred: BCDR, high availability, disaster recovery, data retention, and enterprise security controls
Preferred: ATO processes, NIST or federal compliance controls, vulnerability remediation, SonarQube, Checkmarx, or Zero Trust
Preferred: Docker, Kubernetes, REST APIs, FastAPI, Node.js, or cloud-native application integration
Preferred: federal data modernization programs

Benefits

Comp & perks

Reasonable accommodations for disability, veteran, and sincerely held religious beliefs
Equal opportunity employment
Confidential accommodation support
Benefits information provided under the Transparency in Coverage Act