FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in architecting data platforms, including Lakehouse and EDW environments, while ensuring data governance, security, and compliance. Proficient in designing scalable data models and implementing CI/CD practices for data pipelines using tools like Databricks and Azure Data Factory.
Highest-signal resume keywords
Data Platform ArchitectureDatabricks TechnologiesSpark, PySpark, ScalaCI/CD for Data PipelinesData Governance and Security
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data ModelingDistributed Job OptimizationBatch and Streaming ProcessingData Warehouse DesignCloud SecurityObservability PracticesData Pipeline OptimizationData IntegrationAutomation with CI/CDDatabase Management
Soft Skills
TeamworkCommunicationProblem-SolvingMentorshipIntellectual Curiosity
Tools & Technologies
DatabricksDelta LakeAzure Data FactoryFivetranAirflowPrometheusGrafanaDatadogAzure DevOpsGitHub Actions
Certifications & Qualifications
Databricks CertificationCloud Architect CertificationData Certifications
Industry Keywords
LakehouseEDWData GovernanceIAM/RBACVPC/VNetMedallion PatternLambda PatternKappa PatternData ContractsData Lineage
Tech Stack
Tools & technologiesAirflowAWSAzureCloudETLGrafanaJenkinsMS SQL ServerOraclePostgresPrometheusPySparkScalaSparkSQLUnity
About the role
Key responsibilities & impact- Define overall data platform architecture for Lakehouse/EDW environments, including reference patterns, technology selection, and integration blueprints
- Design conceptual, logical, and physical data models for multi-tenant and vertical-specific data products
- Standardize logical data layers, including ingest/raw, staged/curated, and serving
- Establish data governance, metadata, cataloging, lineage, data contracts, and classification practices
- Define security and compliance controls, including access management, data masking, encryption, network segmentation, and audit policies
- Architect scalability, high availability, disaster recovery, capacity, and cost management strategies
- Lead selection and integration of Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory/Data Fabric, orchestration, and monitoring components
- Design and enforce CI/CD patterns for data artifacts, including testing, automated deployments, and rollback strategies
- Define batch and streaming ingestion patterns, compaction strategies, partitioning schemes, and storage layouts
- Specify observability practices including metrics, SLAs, dashboards, logging, tracing, and alerting
- Serve as technical authority and mentor for Data Engineering teams
- Perform architecture and code reviews for critical components
- Collaborate with Data Product Owners, Security, Infrastructure, BI, and ML stakeholders to translate requirements into solutions and roadmaps
- Design, develop, test, and deploy processing modules using Spark, PySpark/Scala, Spark SQL, and stored procedures
- Build and optimize data pipelines on Databricks and database platforms
- Implement DevOps practices, infrastructure-as-code, CI/CD pipelines, automated testing, and version control
- Troubleshoot complex data quality, performance, and availability issues and implement continuous improvements
Requirements
What you’ll need- Previous experience as architect or lead technical role on enterprise data platforms
- Hands-on experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming
- Strong expertise in Spark, PySpark and/or Scala, Spark SQL, and distributed job optimization
- Solid background in data warehouse and lakehouse design
- Practical familiarity with Medallion, Lambda, and Kappa patterns
- Experience integrating SaaS/ETL/connectors such as Fivetran
- Experience with orchestration platforms such as Airflow, Azure Data Factory, and Data Fabric
- Experience with relational and hybrid databases, including MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or equivalents
- Proficiency in CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar
- Experience packaging and deploying artifacts such as .whl files and containers
- Experience with batch and streaming processing, file compaction, partitioning strategies, and storage tuning
- Understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking
- Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, or native cloud monitoring
- Automation experience with CI/CD pipelines and trunk-based development is a plus
- Advanced PySpark proficiency is a plus
- Advanced Spark workflow optimization and orchestration proficiency is a plus
- Databricks, cloud architect, or data certifications are a plus
- Ability to identify, troubleshoot, and resolve complex data issues
- Strong teamwork and communication skills
- Intellectual curiosity and willingness to embrace new tools, technologies, and methodologies
Benefits
Comp & perks- Full-time employment
- Equal opportunity and inclusive work environment
- Diversity commitment