FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and managing evaluations for machine learning models, with a strong focus on reproducibility, contamination checks, and debugging. Capable of collaborating across teams to enhance evaluation frameworks and contribute to open-source projects.
Highest-signal resume keywords
Evaluation DesignBenchmark DevelopmentSoftware Engineering SkillsDebugging Inference FailuresModel Training Experience
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Evaluation ConstructionContamination ChecksDebugging ScoresManaging EvaluationsDesigning EvaluationsBuilding BenchmarksUnderstanding Judge BiasVariance AnalysisRunning EvaluationsDeep Systems Expertise
Tools & Technologies
Lm-eval-harnessLightevalOLMESHELMInspect
Industry Keywords
CybersecurityAgentic WorkflowsEvaluation SuiteOpen-source EvaluationsFailure-mode Reviews
Tech Stack
Tools & technologiesCyber Security
About the role
Key responsibilities & impact- Maintain, optimize, and ensure reproducibility of the internal evaluation suite and infrastructure at scale
- Design held-out evaluations for agentic workflows, computer use, cybersecurity, and other priority capabilities
- Document evaluation construction and perform contamination checks
- Build evaluations targeted at small edge models
- Read generated outputs to debug unexpected scores
- Convert customer and open-source feedback into environments for data, training, and evaluation
- Partner with other teams to design evaluations for their models
- Release open-source evaluations, write papers, and publish failure-mode reviews of popular evaluations
- Drive post-training decisions through reliable internal evaluation evidence
- Co-own the evaluation suite and support other teams' evaluation needs
Requirements
What you’ll need- Strong ability to design accurate, general-purpose evaluations that separate good models from bad ones and are hard to game
- At least one independently built benchmark, evaluation paper, or substantial contribution to lm-eval-harness, lighteval, OLMES, HELM, or Inspect
- Understanding of contamination, saturation, judge bias, and variance
- Hands-on experience running and managing evaluations in a team
- Habit of debugging unexpected scores by reading individual generated answers
- Strong software engineering skills for owning evaluation infrastructure and debugging inference and infrastructure failures at scale
- Helpful: model training experience
- Helpful: publications or research background
- Helpful: deep systems expertise
- Helpful: familiarity with small edge models
Benefits
Comp & perks- Competitive base salary plus equity
- Unlimited PTO
- Company-wide Refill Days
- Long weekends when the whole company rests together
- Comprehensive health coverage
- Paid leave for pregnancy, parenting, medical needs, and caring for family
- Real compute to run evals at scale
- Opportunity to open-source and publish work