Member of Technical Staff, Evals

Magic

. Develop new benchmarks and evaluation methodologies .

Posted 11/11/2026full-timeRemote • California • United StatesLead💰 $200,000 - $550,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing model evaluations and benchmarks, with a strong focus on improving evaluation correctness, reproducibility, and reliability. Proficient in analyzing model performance and interpreting experimental results within machine learning systems.

Highest-signal resume keywords

Model Evaluation DevelopmentBenchmark CreationMachine Learning SystemsData AnalysisTechnical Project Ownership

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Model EvaluationBenchmarkingData AblationDataset Quality AssessmentMetrics DefinitionExperimental Result InterpretationEvaluation MethodologiesModel Performance AnalysisOpen-Source Implementation AuditingEvaluation Tool Development

Soft Skills

Skepticism Toward ResultsComfort with AmbiguityCommitment to Rigor

Industry Keywords

Machine LearningAI EvaluationsModel TrainingModel DeploymentEvaluation CorrectnessReproducibilityReliability

About the role

Key responsibilities & impact

Develop new benchmarks and evaluation methodologies
Run data ablations
Assess dataset quality
Investigate whether measurements accurately reflect model performance
Design and validate evaluations and benchmarks across model training and deployment
Investigate model behavior and identify gaps in existing evaluations
Identify opportunities to improve training data
Improve evaluation correctness, reproducibility, and reliability
Audit and improve public benchmarks, research papers, and open-source implementations
Define metrics that accurately reflect model quality
Build tools that make evaluation results accessible and useful across Magic

Requirements

What you’ll need

Experience developing model evaluations, benchmarks, or ML experiments
Strong programming skills and experience working with machine learning systems
Track record of owning technical projects end-to-end
Skepticism toward results that cannot be reproduced, validated, or explained
Experience analyzing model performance and interpreting experimental results
Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
Commitment to raising the standard of rigor in AI evaluations

Benefits

Comp & perks

Equity is a significant part of total compensation, in addition to salary
401(k) plan with 6% salary matching
Generous health, dental, and vision insurance for you and your dependents
Unlimited paid time off
Visa sponsorship and relocation support for candidates moving to San Francisco
A small, fast-moving, highly collaborative team working on frontier AI systems