FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Data Engineer, Databricks
49 Solutions. Design, build and maintain enterprise data pipelines using Databricks .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates extensive experience in designing and maintaining enterprise data pipelines using Databricks, Apache Spark, and Python, with a strong focus on data quality and optimization. Proficient in implementing modern data engineering practices, including CI/CD and cloud-based data platforms, particularly in Azure environments.
Highest-signal resume keywords
Databricks ExperienceApache Spark/PySpark ProficiencyData Engineering ExpertiseETL/ELT Process DesignAzure Data Services Familiarity
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
DatabricksApache SparkPySparkPythonSQLETL/ELTData QualityBatch ProcessingStreaming ProcessingDelta Lake
Tools & Technologies
LakeflowLakeflow ConnectAuto LoaderGitCI/CDAzure DevOpsMicrosoft PurviewTerraformDatabricks Asset BundlesUnity Catalog
Certifications & Qualifications
Databricks Certified Data Engineer ProfessionalMicrosoft Azure Data Certifications
Industry Keywords
Government of CanadaData IntegrationData GovernanceMedallion ArchitectureOn-Premises Source Integration
Tech Stack
Tools & technologiesApacheAzureCloudETLOraclePySparkPythonSparkSQLTerraformUnity
About the role
Key responsibilities & impact- Design, build and maintain enterprise data pipelines using Databricks
- Build ingestion and transformation processes using Lakeflow, Lakeflow Connect, Jobs and Auto Loader
- Develop batch and streaming solutions with Apache Spark and PySpark
- Implement raw, cleaned and curated layers using Lakehouse and medallion patterns
- Develop and optimize Delta Lake tables and enterprise data products
- Connect Databricks with cloud, enterprise and on-premises sources
- Modernize legacy ETL/ELT pipelines and data warehouse workloads
- Implement data quality, validation, reconciliation, monitoring and Unity Catalog controls
- Develop reusable engineering patterns and participate in CI/CD and automated deployments
- Support testing, migration, production deployment, runbooks and client knowledge transfer
- Join 49 Solutions' prospective team for anticipated Government of Canada Databricks initiatives; placement or contract award is not guaranteed
Requirements
What you’ll need- Minimum of 10 years of professional experience working in the IT industry or performing IT-related roles
- Valid and active Government of Canada Reliability, Secret or Top Secret security clearance
- Significant experience in data engineering, ETL/ELT or data integration
- Demonstrated hands-on Databricks experience
- Strong Python, SQL, Apache Spark and/or PySpark experience
- Experience designing enterprise pipelines and cloud-based data platforms, preferably Azure
- Experience with data quality, troubleshooting and performance optimization
- Experience with Git and modern software development practices
- Databricks, Lakeflow, Delta Lake, Unity Catalog, Apache Spark/PySpark, Python, SQL, ETL/ELT, batch and streaming processing, medallion architecture, Azure data services, Git and CI/CD
- Lakeflow Connect and Auto Loader, Terraform or Databricks Asset Bundles, Azure DevOps or GitHub, Microsoft Purview, on-premises source integration, and migration from SAS, Synapse, DataStage, DB2, Oracle, SQL Server, Teradata or Netezza are assets
- Government of Canada delivery experience is an asset
- Bilingualism in English and French is an asset
- Databricks Certified Data Engineer Professional is a target certification; candidates without it may still be considered based on experience
- Must be willing to obtain a specific certification at own expense if required for an opportunity
- Databricks EDW/ETL migration training, Databricks Unity Catalog or data governance training, relevant Microsoft Azure data certifications, and additional advanced Databricks certifications are assets
- Willing and able to travel to Ottawa for onsite meetings approximately once per month when applicable
Benefits
Comp & perks- Candidates may remain in the candidate database and be considered for current and future opportunities
- AI use for recruitment and proposal activities is optional
- Candidates may opt out of AI processing and still apply, work with 49 Solutions, and be considered for opportunities