FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Tech Stack
Tools & technologiesAirflowApacheAWSAzureCloudDockerKubernetesPySparkPythonSparkSQLTerraform
About the role
Key responsibilities & impact- Design, develop, and optimize scalable data workflows using Python, PySpark, and Airflow
- Implement real-time and batch data processing using Spark
- Enforce data quality, governance, and security best practices throughout the data lifecycle
- Ensure data availability, reliability, and performance through monitoring and automation
- Manage cloud infrastructure and optimize costs for data processing workloads
- Implement CI/CD pipelines for data workflows
- Build and optimize large-scale data processing pipelines using Apache Spark and PySpark
- Implement data partitioning, caching, and performance tuning for Spark workloads
- Work with structured and unstructured data formats for analytics and machine learning initiatives
- Design and maintain Airflow DAGs to automate complex data workflows
- Monitor, troubleshoot, and optimize job execution and dependencies
- Lead a team of data engineers and provide technical guidance and mentorship
- Foster coding standards, version control, and documentation best practices
- Engage directly with clients to understand business needs and communicate technical concepts
- Collaborate with engineering, product, and architecture peers to drive scalable, maintainable products
- Partner with Program Management to align delivery timelines and mitigate risks
Requirements
What you’ll need- 8+ years of experience in data engineering
- Expertise in Python, PySpark, Apache Spark, and Apache Airflow
- Strong programming skills in Python and SQL
- Ability to write efficient and maintainable code
- Deep understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning
- Experience with Airflow DAGs, scheduling, and dependency management
- Knowledge of Git, Docker, Kubernetes, Terraform, and DevOps best practices for CI/CD workflows
- Experience optimizing large-scale data processing
- Experience leading teams and working in Agile/Scrum environments
- Experience with common data warehouse and/or data lake platforms
- Knowledge of cloud technologies, including Azure, AWS, or Google Cloud
- Proven experience using generative AI tools for software development, data engineering, system design, testing, and troubleshooting
- Strong understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows
- Ability to identify opportunities to apply AI solutions to improve team productivity, code quality, and operational efficiency
- Experience in ML Engineering is a plus
- Strong communication and collaboration skills
- Ability to effectively work with remote teams
- Prospective candidates may be asked to consent to background checks
Benefits
Comp & perks- Flexible and accommodating work environment
- Paid time off
- Medical, dental, and vision insurance
- 401(k)
- Bonus pay eligibility
- Diversity, equity, and inclusion commitment
- Accessibility accommodations for applicants with disabilities
- Background checks conducted in accordance with local legislation
- Current employer not contacted without permission