Senior Machine Learning Engineer, AI Safety Evaluations

NVIDIA

. Design and maintain safety benchmarks for large language and multimodal models .

Posted 11/5/2026full-timeRemote • California • United StatesSenior💰 $184,000 - $356,500 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing safety benchmarks and metrics for machine learning models, with strong proficiency in Python and experience in automated evaluation methods. Capable of translating complex evaluation scenarios into actionable test cases and communicating findings effectively to diverse stakeholders.

Highest-signal resume keywords

Machine Learning EngineeringPython ProgrammingBenchmark DevelopmentAutomated Red-TeamingModel Evaluation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Safety Benchmark DesignTest Case DevelopmentStatistical ComparisonData Pipeline ConstructionExperiment TrackingRegression TestingDataset Quality AssessmentConfidence IntervalsHuman-in-the-Loop EvaluationAdversarial Testing

Soft Skills

Clear CommunicationCollaboration with ResearchersProblem-Solving

Certifications & Qualifications

Bachelor’s Degree

Industry Keywords

Large Language ModelsMultimodal ModelsEvaluation FrameworksPublic BenchmarksStatistical Practices

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Design and maintain safety benchmarks for large language and multimodal models
Translate risk scenarios into test cases, datasets, and scoring rubrics
Define metrics and protocols assessing safety alongside model usefulness
Use baselines, uncertainty estimates, and failure analysis to distinguish meaningful improvements from noise
Build reusable Python pipelines with versioned datasets, model configurations, and experiment tracking
Develop automated red-teaming tools to discover failure modes and expand evaluation coverage
Incorporate human review to validate automated scoring and provide additional judgment
Investigate model failures with researchers and document actionable findings
Develop regression tests to assess whether model changes resolve identified issues
Partner with platform engineers to store and aggregate evaluation results
Communicate findings, limitations, and emerging risks to researchers and decision-makers

Requirements

What you’ll need

Bachelor’s degree or equivalent experience
6+ years in machine learning engineering or evaluation
Experience developing benchmarks, metrics, and statistically sound comparisons for ML systems
Ability to translate ambiguous evaluation questions into testable experiments
Strong Python skills
Experience building maintainable shared tools, data pipelines, and experiment-tracking workflows
Ability to assess dataset quality and identify limitations in metrics and scoring methods
Ability to support conclusions with reproducible evidence
Ability to clearly explain findings and tradeoffs to researchers, engineering partners, and executives
Experience evaluating LLMs, multimodal models, or agentic systems through adversarial testing or automated red-teaming
Contributions to public benchmarks or evaluation frameworks
Publication track record and interest in continuing to publish at top tier venues
Experience validating model-based evaluators against human judgments or designing human-in-the-loop evaluation workflows
Strong model evaluation skills and commitment to sound statistical practices, including confidence intervals

Benefits

Comp & perks

Equity
Benefits 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score