FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Databricks Engineer
Koniag Government Services. Design, develop, and deploy scalable data infrastructure .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and implementing scalable data infrastructure and pipelines using Databricks, PySpark, and SQL. Proficient in data warehousing concepts and automating deployments with Azure DevOps while ensuring data quality and security.
Highest-signal resume keywords
Databricks Ecosystem ExpertisePySpark ProficiencyData Warehousing ConceptsActive SECRET ClearanceAutomated Deployment Experience
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data EngineeringPySparkSQLRESTful API IntegrationsData Ingestion PipelinesETL/ELT ProcessesObject Oriented ProgrammingData Quality StandardsMedallion ArchitectureCustom Python Libraries
Soft Skills
Concise CommunicationMentoringCollaboration
Tools & Technologies
Azure DevOpsDatabricks Asset BundlesAzure Data Lake Storage Gen2PowerBITableauJupyter NotebooksGit
Industry Keywords
Data LakeData Vault 2.0Data Engineering InitiativesHTTP REST MethodsData Quality Monitoring
Tech Stack
Tools & technologiesAzureElasticSearchETLJavaJavaScriptOraclePySparkPythonSparkSQLTableauVault.NET
About the role
Key responsibilities & impact- Design, develop, and deploy scalable data infrastructure
- Drive key data engineering initiatives using the Databricks platform and modern data warehousing
- Build data pipelines supporting the Department of State’s “4Ps” data lake
- Work with project managers, team members, and business owners to gather requirements, communicate progress, and ensure data meets customer needs
- Architect and implement scalable, highly available data pipelines using Databricks and the Medallion Architecture
- Establish data quality standards and implement monitoring dashboards
- Write production-ready PySpark and SQL code for complex ETL/ELT processes
- Develop DLT and structured streaming pipelines
- Design and build data ingestion pipelines using RESTful API integrations
- Implement infrastructure and pipelines as code
- Automate deployments across Development, Staging, and Production using Databricks Asset Bundles and Azure DevOps
- Develop, package, and maintain custom Python libraries and modules
- Apply data warehousing concepts, including dimensional modeling and slowly changing dimensions
- Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs
- Share data securely with external BI tools such as PowerBI and SAP BusinessObjects
- Develop dashboards and capabilities natively within the Databricks environment
- Serve as a technical SME, conduct code reviews, establish best practices, and mentor staff members
Requirements
What you’ll need- Active SECRET clearance
- Knowledge of Object Oriented Programming (OOP) concepts
- Proficiency in one or more programming or scripting languages: Java, C#, Python, JavaScript, or PowerShell
- 8+ years of dedicated Data Engineering experience
- At least 3+ years functioning in a senior capacity focused on the Databricks ecosystem
- Expert-level proficiency in PySpark and complex SQL
- Proven ability to create, package, and manage custom Python libraries, including building Wheel files
- Deep understanding of HTTP REST methods such as GET, POST, and PUT
- Hands-on experience extracting, paginating, and processing data from external API sources
- Deep theoretical and practical understanding of modern Data Warehousing concepts
- Hands-on experience building out the Medallion Architecture
- Proven experience automating Databricks deployments
- Hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps, including creating YAML pipelines, managing service principals, and handling multi-workspace deployments
- Strong familiarity with Azure Data Lake Storage Gen2 and Azure Key Vault
- Proficiency with Git-based version control workflows
- Concise and comprehensive written and oral communication skills for technical understanding and customer usage support
- Desired: Experience with Tableau, PowerBI, Jupyter Notebooks, d3js, Elasticsearch, SOLR, and Lucene
- Desired: Experience working with HR application (PeopleSoft) data for reporting purposes
- Desired: 3+ years of .NET, Java, or similar OOP experience
- Desired: Experience writing Oracle PL/SQL queries, packages, and procedures
- Desired: Experience architecting Data Vault 2.0 solutions
Benefits
Comp & perks- Medical, dental, and vision insurance
- 401(k) retirement plan
- Paid time off
- Paid parental leave
- Life and disability insurance
- Flexible spending accounts
- Commuter benefits
- Tuition reimbursement