FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and developing scalable data ingestion and transformation pipelines using Databricks, PySpark, and SQL, while ensuring data quality and governance. Proficient in automation and optimization of data workflows, with strong collaboration skills to translate business requirements into technical solutions.
Highest-signal resume keywords
Python ProgrammingSQL ProficiencyPySpark ExperienceETL/ELT Pipeline DevelopmentDatabricks Knowledge
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data IngestionData TransformationData IntegrationAutomation FrameworksMetadata ManagementData Quality ChecksDimensional ModelingPerformance OptimizationData GovernanceVersion Control
Soft Skills
Analytical SkillsProblem-Solving SkillsCommunication SkillsCollaboration SkillsTroubleshooting Skills
Tools & Technologies
DatabricksApache SparkDelta LakeMicrosoft AzureAWSGitCI/CD FrameworksDevOps ToolingAPIsCloud Storage
Industry Keywords
Data EngineeringData ArchitectureLakehouse ArchitectureData WarehousingData QualityData ValidationStructured DataSemi-Structured DataAutomationData Pipelines
Tech Stack
Tools & technologiesApacheAWSAzureCloudETLPySparkPythonSparkSQLSSISUnity
About the role
Key responsibilities & impact- Design, develop, test, and maintain scalable data ingestion, transformation, and integration pipelines using Databricks, PySpark, Python, and SQL
- Build and enhance enterprise data pipelines following modern Lakehouse and Medallion Architecture principles
- Develop reusable, modular, and metadata-driven ETL/ELT frameworks
- Ingest and integrate structured and semi-structured data from databases, APIs, flat files, cloud storage, and enterprise applications
- Work with Delta Lake, Unity Catalog, Databricks Workflows, and related platform capabilities
- Support migration and modernization of legacy SQL Server, SSIS, and traditional data warehouse workloads onto Databricks
- Optimize data pipelines and transformations for performance, reliability, scalability, and cost efficiency
- Develop Python-based automation frameworks, utilities, and reusable engineering components
- Design metadata-driven and configuration-driven solutions that reduce manual coding and operational dependencies
- Leverage scripting, APIs, orchestration, and intelligent automation techniques to streamline engineering workflows
- Develop efficient data transformation logic using SQL, Python, and PySpark
- Implement joins, aggregations, business rules, enrichment logic, and incremental processing patterns
- Support dimensional modeling, historical data tracking, and efficient data processing patterns
- Collaborate with business analysts, data architects, senior engineers, architects, and technical teams to translate requirements into engineering solutions
- Implement automated data quality checks for completeness, accuracy, consistency, uniqueness, and referential integrity
- Develop reconciliation and validation routines across source systems, transformation layers, and downstream datasets
- Support metadata management, data lineage, cataloging, and data governance initiatives
- Build monitoring, logging, and alerting capabilities for pipelines and scheduled workloads
- Troubleshoot pipeline failures, data discrepancies, performance issues, and integration errors
- Participate in root-cause analysis and implement permanent corrective solutions
- Follow security, access control, data privacy, and auditability requirements
- Maintain engineering code using Git-based version control and collaborative development practices
- Support automated deployment pipelines using CI/CD frameworks and DevOps tooling
- Implement unit tests, integration tests, and automated validation
- Contribute to infrastructure automation and environment management
- Document technical designs, data mappings, transformations, deployment procedures, and operational runbooks
- Continuously improve engineering productivity through automation and reusable frameworks
Requirements
What you’ll need- Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical discipline, or equivalent practical experience
- 2+ years of hands-on experience in data engineering, software engineering, data integration, or related technical roles
- Very strong Python programming skills, including scripting, modular development, data processing, and automation
- Very strong SQL proficiency, including complex queries, joins, aggregations, window functions, transformations, and query performance optimization
- Strong hands-on PySpark experience, including distributed data processing, DataFrame operations, transformations, and performance optimization
- Experience developing ETL/ELT pipelines and working with relational databases or modern data platforms
- Working knowledge of Databricks, Apache Spark, Delta Lake, or comparable cloud-based data engineering technologies
- Understanding of modern data architecture concepts, including data lakes, data warehouses, lakehouses, and data pipelines
- Experience working with structured data, JSON, CSV, Parquet, APIs, and other common integration formats
- Familiarity with cloud platforms such as Microsoft Azure or AWS
- Basic understanding of data modeling, data quality, and data validation practices
- Experience using Git or comparable source control tools
- Strong automation and hyperautomation mindset, with demonstrated ability to simplify workflows, eliminate manual tasks, and build reusable engineering solutions
- Strong analytical, troubleshooting, and problem-solving skills
- Ability to work independently on assigned engineering tasks while collaborating effectively within a broader technical team
- Strong communication skills and willingness to learn new technologies
- Employment offers are contingent upon successful completion of a background screening
Benefits
Comp & perks- PTO
- Medical insurance
- Dental insurance
- Vision insurance
- Retirement savings
- Disability insurance
- Life insurance
- Reasonable accommodations throughout the application and employment process