Principal Engineer – Data Retrieval

NVIDIA

. Architect and scale data acquisition infrastructure powering NVIDIA’s foundation models .

Posted 10/26/2026full-timeRemote • United StatesLead💰 $272,000 - $431,250 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in architecting and scaling data acquisition infrastructure, with a strong focus on web crawling, distributed systems, and high-throughput data processing. Proven ability to lead cross-team collaborations and mentor engineers while ensuring production readiness and compliance with data governance standards.

Highest-signal resume keywords

Large-Scale Distributed Systems DesignWeb Crawling InfrastructureHigh-Throughput Data ProcessingTechnical Communication and LeadershipData Governance and Compliance

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Software EngineeringSystem DesignData Pipeline DevelopmentDistributed SchedulingPerformance AnalysisData ProcessingCrawl Platform ArchitectureData Quality AssuranceProvenance InfrastructureBatch and Streaming Pipelines

Soft Skills

Technical CommunicationLeadershipMentoringCross-Team CollaborationProblem-Solving

Industry Keywords

Data AcquisitionCrawl Frontier StorageData GovernancePublisher ConsentDataset LicensingMachine Learning Data PlatformsOperational ReliabilityResource EfficiencyQuality-Aware Crawl RankingMultimodal Datasets

Tech Stack

Tools & technologies

Distributed Systems

About the role

Key responsibilities & impact

Architect and scale data acquisition infrastructure powering NVIDIA’s foundation models
Build systems that discover valuable sources, crawl the open web at scale, and deliver reliable, high-quality data to training pipelines
Own architecture of a unified platform covering source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion
Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and quality-, diversity-, and freshness-based scheduling
Build a continuous discovery loop that extracts links and feeds new candidates into the crawl frontier
Develop a shared crawl execution layer that fetches reliably and politely at scale, including dynamic websites
Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets
Define stable interfaces for source connectors
Partner with parsing and curation teams to preserve valuable code, math, and structured content and improve accepted data yield
Implement provenance and policy enforcement with legal, privacy, security, and data governance partners
Translate publisher preferences, opt-outs, licensing terms, and use restrictions into auditable downstream controls
Own production readiness, including capacity planning, observability, fault tolerance, recovery, and cost efficiency
Instrument the pipeline end to end and set service objectives for throughput, freshness, reliability, and training cutoffs
Evaluate internal crawls and external datasets through comparable pilots with research and product teams
Deliver reproducible data releases tied to source coverage and processing quality in controlled training experiments
Coordinate cross-team dependencies and balance immediate data delivery with long-term platform development
Lead architecture reviews and critical implementation work
Mentor engineers and help build the team

Requirements

What you’ll need

BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience
15+ years of relevant experience in software engineering
Experience designing and operating large-scale distributed systems
Experience setting technical direction and delivering complex infrastructure across multiple teams
Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure
Experience with distributed scheduling, stateful services, storage, and high-throughput data processing
Strong hands-on engineering and system design skills
Ability to move between architecture, implementation, performance analysis, and production debugging
Experience building batch or streaming pipelines
Knowledge of consistency, retries, duplicate processing, backpressure, and recovery
Track record of improving throughput, latency, resource efficiency, or operational reliability
Experience delivering data platforms for research, ML, search, or other demanding technical users
Ability to translate evolving requirements into clear interfaces and measurable engineering outcomes
Strong technical communication and leadership through influence across engineering, product, research, and governance teams
Experience architecting a web-wide crawl platform, scaling crawl frontier storage, or consolidating multiple acquisition systems into shared infrastructure
Direct experience supplying foundation-model pretraining data, including targeted discovery, quality-aware crawl ranking, and acquisition of code or technical content
Experience implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale
Background with browser-based or agentic data collection, programmable extraction, and sandboxed environments for tool-use data or post-training evaluation
Experience onboarding licensed or multimodal datasets, building reusable partner integrations, or delivering streaming feeds with low latency for retrieval and grounding

Benefits

Comp & perks

Equity
Benefits 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score