Staff Data Platform Engineer

Reality Defender (YC W22)

. Own the end-to-end data platform for AI research, including continuous ingestion, preprocessing, governance, and export of image, video, audio, and text data at high volume .

Posted 10/16/2026full-timeRemote • New York • United StatesLead💰 $210,000 - $260,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and managing high-volume data platforms, with a focus on distributed processing, data governance, and synthetic data generation. Proficient in collaborating with ML researchers to enhance model iteration and ensuring pipeline reliability and observability.

Highest-signal resume keywords

Data Platform OwnershipDistributed Processing (Spark, Ray)Databricks Production ExperiencePython and SQL ProficiencyInfrastructure Automation (Terraform, CI/CD)

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data IngestionData GovernanceDataset VersioningData Quality ValidationSynthetic Data GenerationMetadata ManagementPerformance TuningOperational TroubleshootingOrchestration (Airflow, Dagster)Cloud Object Storage (S3, GCS, Azure Blob)

Soft Skills

JudgmentMentoringCollaboration

Tools & Technologies

DatabricksUnity CatalogDelta LakeAWSTerraformCI/CDDatabricks Workflows

Industry Keywords

ML ResearchUnstructured MediaSecurityFraudTrust & Safety

Tech Stack

Tools & technologies

AirflowAWSAzureCloudPythonRaySparkSQLTerraformUnity

About the role

Key responsibilities & impact

Own the end-to-end data platform for AI research, including continuous ingestion, preprocessing, governance, and export of image, video, audio, and text data at high volume
Complete and scale the Databricks lakehouse and distributed processing stack
Make architecture and build-vs-buy decisions as data volume grows
Build reusable infrastructure for in-house synthetic data generation across modalities
Implement dataset versioning, lineage, metadata, quality checks, and stable train/evaluation splits
Own pipeline reliability, observability, and processing cost per dataset
Partner with ML researchers to create self-serve tools that speed model iteration
Mentor engineers and improve code review, testing, documentation, and runbooks
Work with Security and compliance stakeholders on data handling, licensing, and access controls
Shape the long-term data platform roadmap with engineering and research leadership
Report to the Director of Data
Within 12 months, own a mature self-serve platform, scale synthetic data generation, deliver reproducible evaluation datasets, and lead high-leverage roadmap initiatives

Requirements

What you’ll need

8+ years building data platforms or data infrastructure, including staff- or lead-level ownership of production pipelines that generate, ingest, or deliver data at high volume
Deep hands-on experience with distributed processing (Spark, Ray, or similar) and lakehouse platforms, ideally Databricks, on top of cloud object storage (S3, GCS, or Azure Blob)
Production experience with Databricks, Unity Catalog, Delta Lake, and Spark on AWS, including performance tuning, access controls, and operational troubleshooting
Experience working with large unstructured media such as video, audio, or images, including large files, decoding and transcoding, and metadata at scale
Track record of building reproducible ML datasets: versioning, lineage, data quality validation, and orchestration (Airflow, Dagster, Databricks Workflows, or similar)
Strong Python and SQL
Judgment to own ambiguous platform problems end-to-end on a lean team
Infrastructure and deployment automation using Terraform and CI/CD, including AWS storage, IAM, compute, Databricks resources, and distributed processing environments
Experience building synthetic data generation pipelines or running generative models at scale (nice to have)
Background supporting ML research teams with training and evaluation data, especially for multimodal models (nice to have)
Experience in security, fraud, trust & safety, or another adversarial domain, or at a Series A or B startup (nice to have)
Authorization to work in the U.S. is requested in the application
Willingness to relocate if not currently located in or near NYC, depending on work arrangement

Benefits

Comp & perks

Healthcare plans with 100% premium coverage for employees and partial coverage available for dependents
Dental and Vision plans with 100% premium coverage for employees and their dependents
Short/Long-term disability and life insurance plans with 100% premium coverage for employees
FSA/HSA and 401k programs
Equity compensation
20 days of PTO per year
12 weeks of Parental Leave
Learning and Development budget
Monthly wellness benefits
Annual company-sponsored offsite
Daily in-office lunch through UberEats for NYC HQ employees
Commuter benefits for NYC HQ employees
Remote Fridays for NYC HQ employees
Happy Hours and other local events for NYC HQ employees
Eligibility for annual performance bonus