Software Engineer, Big Data

ZipRecruiter

. Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and via event streams for downstream applications .

Posted 11/3/2026full-timeRemote • United StatesJuniorMid-Level💰 $135,000 - $185,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing, building, and maintaining data pipelines and MLOps frameworks, with a strong focus on data-intensive systems and infrastructure. Proficient in leveraging tools and technologies for data processing, orchestration, and analysis to enhance ML development and product reliability.

Highest-signal resume keywords

Apache Spark ExperienceData Pipeline DevelopmentMLOps FrameworksAWS Services KnowledgeContainerized Services Deployment

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data StructuresAlgorithmsDistributed SystemsBatch ProcessingStreaming TechniquesETL PipelinesData DocumentationData AnalysisMetrics InstrumentationSchema Evolution

Tools & Technologies

Apache KafkaApache AirflowKubernetesDockerAWS GlueApache FlinkDelta LakeParquetS3EC2

Industry Keywords

Data LakeData Pipeline OrchestrationML DevelopmentData DiscoverabilityUser-Facing Product

Tech Stack

Tools & technologies

AirflowApacheAWSDistributed SystemsDockerEC2ETLKafkaKubernetesSpark

About the role

Key responsibilities & impact

Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and via event streams for downstream applications
Process large, evolving datasets using batch and streaming techniques while balancing speed, efficiency, and reliability
Design, build, deploy, and maintain MLOps frameworks and systems that accelerate ML development across teams and products
Improve the velocity, quality, observability, and reliability of ML development and ML-powered products
Design, build, deploy, and maintain data documentation frameworks and tooling
Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool
Build scalable, reliable, and efficient infrastructure, tools, and processes that enable engineering teams to deliver products and services
Build and maintain tools for data processing, dataset ingestion and discoverability, data pipeline orchestration, data documentation, data analysis, ML model pipelines, and experiments
Help developers leverage AI through an agentic AI data exploration and analysis tool and an LLM Gateway

Requirements

What you’ll need

2+ years of professional software development experience, including building and operating data pipelines that feed into a customer facing product
Hands-on Apache Spark experience: writing, tuning and debugging batch and/or streaming jobs at scale
Experience deploying and operating containerized services and batch jobs
Solid computer science fundamentals: data structures, algorithms, distributed systems, and testing
Professional software development experience, including building and operating data pipelines and infrastructure that feed into a customer facing product
Preferred: 4+ years of professional software development experience with a focus on data-intensive systems
Preferred: Working knowledge of AWS (S3, IAM, EC2, MSK or equivalents) and instrumenting jobs with metrics, logs and alerts
Preferred: Experience with a data lake stack: parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake
Preferred: Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming
Preferred: Experience with Apache Airflow, including authoring DAGs, backfills, dependency management and on-call for scheduled pipelines
Preferred: Experience with open table format internals: schema evolution, partitioning, and compaction
Preferred: Experience deploying services on Kubernetes with Docker
Preferred: Experience building and maintaining a data platform powering a user-facing product, including streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery
BS/MS/PhD in Computer Science, Mathematics, Physics or a related field, or equivalent practical experience (preferred)
Must be able to work in the United States, as indicated by the US base salary range and US-based positions language

Benefits

Comp & perks

Competitive compensation
Exceptional benefits package
Flexible Vacation & Paid Time Off
Employer-matched 401(k) plan
Equity, bonuses, commission, or other forms of compensation may also be provided depending on the position offered
Medical, financial, and other benefits