FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building and optimizing high-volume data ingestion pipelines using Apache Spark and Python, with a focus on reliability, performance, and cost efficiency. Capable of diagnosing and resolving production issues while contributing to the technical direction of data processing platforms.
Highest-signal resume keywords
Apache SparkPythonData Ingestion PipelinesETLAirflow
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
DebuggingOptimizingBatch-ProcessingData ProcessingData Formats
Soft Skills
Independent WorkPractical ImprovementsFast-Moving Environment
Tools & Technologies
AirflowFlyteDatabricksDelta LakeScalaJava
Industry Keywords
Ingestion PipelinesData ScienceModel TrainingHigh-Throughput SystemsProduction Experience
Tech Stack
Tools & technologiesAirflowApacheETLJavaPythonScalaSpark
About the role
Key responsibilities & impact- Debug and unblock failing ingestion pipelines, stalled jobs and processing queues
- Investigate corrupt, malformed or unexpected data from varied sources
- Optimise Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency
- Design and implement resilient pipelines so bad data segments do not block wider workflows
- Improve monitoring, retries and failure recovery across multi-step ingestion workflows
- Prioritise and unblock datasets for annotation, data science and model-training teams
- Contribute to the technical direction of Wayve’s ingestion platform
- Build and operate high-volume pipelines processing real-world driving data for annotation, data science, model training and evaluation
- Orchestrate complex dependencies, queues, retries and priority data flows
- Improve performance, cost and operability of the Spark-based data platform
- Ingest varied data formats from partners, suppliers and third-party sources
Requirements
What you’ll need- Strong production experience with Apache Spark and Python
- Experience building, operating or debugging large-scale ingestion, ETL or distributed data-processing pipelines
- Comfortable diagnosing production failures involving corrupt data, stalled queues, retries and complex dependencies
- Understanding of optimising high-throughput systems for reliability, performance and cost
- Ability to learn an existing system quickly, work independently and deliver practical improvements in a fast-moving environment
- Experience with Airflow, Flyte, Databricks, Delta Lake, Scala or Java would be an advantage
- Eligibility to work in the country where applying
- Strong production experience with Apache Spark
- Largest processed or ingested data volume must be provided in the application
Benefits
Comp & perks- Meaningful equity, sharing in the ownership and long term success of Wayve
- Relocation support and visa sponsorship where applicable
- Hybrid working, core hours and the chance to work hands on in vehicle workshops and labs
- Learning and development budgets with support for training, conferences and growth
- Comprehensive benefits including health insurance, dental, enhanced maternity and paternity leave, retirement or pension where applicable, access to therapists, wellbeing partnerships and team socials