Data Engineer

• Architect and scale data acquisition infrastructure powering NVIDIA’s foundation models, initially for code and STEM data for Nemotron.
• Design and own a unified platform for source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion.
• Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and quality-, diversity-, and freshness-based scheduling.
• Build a continuous discovery loop that extracts links and feeds new candidates into the crawl frontier.
• Develop a shared crawl execution layer that fetches reliably and politely at scale, including dynamic websites.
• Build extensible pipelines for open-web, licensed, and partner content, including text, PDFs, code repositories, and multimodal assets.
• Define stable source-connector interfaces.
• Partner with parsing and curation teams to preserve code, math, and structured content and improve accepted data yield.
• Implement provenance and policy enforcement with legal, privacy, security, and data governance partners.
• Translate publisher preferences, opt-outs, licensing terms, and use restrictions into auditable downstream controls.
• Own production readiness, capacity planning, observability, fault tolerance, recovery, and cost efficiency.
• Instrument the pipeline end to end and set service objectives for throughput, freshness, reliability, and training cutoffs.
• Work with research and product teams to evaluate internal crawls and external datasets through comparable pilots.
• Deliver reproducible data releases connecting source coverage and processing quality to controlled training experiments.
• Coordinate cross-team dependencies and tradeoffs between immediate data delivery and long-term platform development.
• Lead architecture reviews, critical implementation work, engineer mentoring, and team building.

Explore Principal Engineer – Data Retrieval at NVIDIA. Review requirements and application details.

Remote Data Engineer jobs in Switzerland

Principal Engineer – Data Retrieval

Senior Data Engineer, Report Builder, Integration QA

Data Architect

Senior Scientific Data Engineer

Junior Data Engineer, AI

Senior Data Engineer