Senior Software Engineer, Big Data

ZipRecruiter

. Design, build, deploy, and maintain data pipelines and systems that make web events available in the data lake and via event streams .

Posted 11/3/2026full-timeRemote • United StatesSenior💰 $190,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing, building, and maintaining data pipelines and MLOps frameworks, with a strong focus on data-intensive systems and infrastructure. Proficient in utilizing tools and technologies such as Apache Spark, AWS, and container orchestration to enhance machine learning development and data processing efficiency.

Highest-signal resume keywords

Apache Spark ExperienceData Pipeline DevelopmentMLOps FrameworksAWS Services KnowledgeContainerization with Kubernetes

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data StructuresAlgorithmsDistributed SystemsBatch ProcessingStreaming TechniquesETL PipelinesApache AirflowKafkaApache FlinkData Lake Stack

Tools & Technologies

AWS S3AWS IAMAWS EC2DockerKubernetesApache IcebergDelta LakeMetrics InstrumentationData Documentation FrameworksSift Tool

Certifications & Qualifications

BS in Computer ScienceMS in Computer SciencePhD in Computer Science

Industry Keywords

Data EngineeringMachine LearningData ExplorationData DiscoveryData Permissions Management

Tech Stack

Tools & technologies

AirflowApacheAWSDistributed SystemsDockerEC2ETLKafkaKubernetesSpark

About the role

Key responsibilities & impact

Design, build, deploy, and maintain data pipelines and systems that make web events available in the data lake and via event streams
Process large, evolving datasets using batch and streaming techniques while balancing speed, efficiency, and reliability
Design, build, deploy, and maintain MLOps frameworks and systems that accelerate machine learning development
Improve the velocity, quality, observability, and reliability of ML development and ML-powered products
Design, build, deploy, and maintain data documentation frameworks and tooling
Design, build, deploy, and maintain Sift, an agentic data exploration and analysis tool
Build scalable, reliable, and efficient infrastructure, tools, and processes for engineering teams

Requirements

What you’ll need

4+ years of professional software development experience, including building and operating data pipelines that feed into a customer facing product
Hands-on Apache Spark experience: writing, tuning and debugging batch and/or streaming jobs at scale
Experience deploying and operating containerized services and batch jobs
Solid computer science fundamentals: data structures, algorithms, distributed systems, and testing
Professional software development experience, including building and operating data pipelines and infrastructure that feed into a customer facing product
Preferred: 4+ years of professional software development experience with a focus on data-intensive systems
Preferred: Working knowledge of AWS (S3, IAM, EC2, MSK or equivalents) and instrumenting jobs with metrics, logs and alerts
Preferred: Experience with a data lake stack, including Parquet on object storage, AWS Glue or similar data catalog, and Apache Iceberg or Delta Lake
Preferred: Streaming experience with Kafka, Apache Flink, and/or Spark Structured Streaming
Preferred: Experience with Apache Airflow workflow orchestration, DAGs, backfills, dependency management, and on-call for scheduled pipelines
Preferred: Experience with open table format internals, including schema evolution, partitioning, and compaction
Preferred: Experience deploying services on Kubernetes with Docker
Preferred: Experience building and maintaining a data platform powering a user-facing product, including streaming and batch data ingestion, ETL pipelines, data permissions management, and data discovery
Preferred: BS/MS/PhD in Computer Science, Mathematics, Physics, or a related field, or equivalent practical experience

Benefits

Comp & perks

Competitive compensation
Exceptional benefits package
Flexible Vacation & Paid Time Off
Employer-matched 401(k) plan
Potential equity, bonuses, commission, or other forms of compensation as part of total compensation package
Medical, financial, and other benefits