Senior Software Engineer, Big Data

ZipRecruiter

. Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and via event streams .

Posted 11/3/2026full-timeRemote • California • United StatesSenior💰 $190,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing, building, and maintaining data pipelines and MLOps frameworks, with a strong focus on improving the quality and reliability of ML development. Proficient in utilizing cloud technologies and data lake architectures to enhance data accessibility and collaboration.

Highest-signal resume keywords

Apache Spark ExperienceMLOps Framework DevelopmentAWS (S3, IAM, EC2, MSK)Data Lake Stack (Parquet, AWS Glue, Apache Iceberg)Kubernetes with Docker

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentBatch and Streaming ProcessingWorkflow Orchestration (Apache Airflow)Data Structures and AlgorithmsETL PipelinesStreaming Experience (Kafka, Apache Flink)Data Permissions ManagementData DiscoverySchema EvolutionCompaction

Tools & Technologies

Apache AirflowKubernetesDockerApache KafkaApache FlinkAWS GlueApache IcebergDelta LakeSift Agentic ToolMetrics and Logging Tools

Industry Keywords

Data LakeMLOpsData DocumentationData-Driven Decision MakingDistributed Systems

Tech Stack

Tools & technologies

AirflowApacheAWSDistributed SystemsDockerEC2ETLKafkaKubernetesSpark

About the role

Key responsibilities & impact

Design, build, deploy, and maintain data pipelines and systems that make web events available in the datalake and via event streams
Process large, evolving datasets using batch and streaming techniques while balancing speed, efficiency, and reliability
Design, build, deploy, and maintain MLOps frameworks and systems that accelerate ML development across teams and products
Improve the velocity, quality, observability, and reliability of ML development and ML-powered products
Design, build, deploy, and maintain data documentation frameworks and tooling
Improve the comprehensiveness and quality of data documentation for human and automated processes
Design, build, deploy, and maintain the Sift agentic data exploration and analysis tool
Democratize access to data, enable collaboration on insights, and accelerate data-driven decision making

Requirements

What you’ll need

4+ years of professional software development experience, including building and operating data pipelines that feed into a customer facing product
Hands-on Apache Spark experience: writing, tuning and debugging batch and/or streaming jobs at scale
Experience deploying and operating containerized services and batch jobs
Solid computer science fundamentals: data structures, algorithms, distributed systems, and testing
Professional software development experience, including building and operating data pipelines and infrastructure that feed into a customer facing product
Working knowledge of AWS (S3, IAM, EC2, MSK or equivalents) and instrumenting jobs with metrics, logs and alerts
Experience with a data lake stack: Parquet on object storage, a data catalog such as AWS Glue, and an open table format such as Apache Iceberg or Delta Lake
Streaming experience with Kafka producers and consumers, Apache Flink, and/or Spark Structured Streaming
Experience with a workflow orchestrator such as Apache Airflow: authoring DAGs, backfills, dependency management and on-call for scheduled pipelines
Experience with open table format internals: schema evolution, partitioning, and compaction
Experience deploying services on Kubernetes with Docker
Experience building and maintaining a data platform that powers a user-facing product, including streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery
BS/MS/PhD in Computer Science, Mathematics, Physics or a related field, or equivalent practical experience

Benefits

Comp & perks

Competitive compensation
Exceptional benefits package
Flexible Vacation & Paid Time Off
Employer-matched 401(k) plan
Potential equity, bonuses, commission, or other compensation as part of total compensation package
Medical, financial, and other benefits