Principal Research Scientist

. Own the evaluation methodology and set standards for evaluating AI agents .

Posted 10/27/2026full-timeRemote • United StatesLead💰 $190,000 - $235,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in evaluation design and methodology for AI agents, with a strong background in machine learning and natural language processing. Capable of leading research agendas, mentoring teams, and translating complex evaluation practices into actionable product decisions.

Highest-signal resume keywords

Evaluation DesignMachine Learning SystemsLLM-Based AgentsStatistical JudgmentResearch Agenda Ownership

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Evaluation MethodologyDataset CurationMachine LearningNatural Language ProcessingExperimental DesignStatistical AnalysisAgent EvaluationUser SimulationAutomated ChecksScoring Approaches

Soft Skills

Clear WritingMentoringCommunicationJudgmentCollaboration

Tools & Technologies

AI BenchmarkingOpen-Source Evaluation ToolsBusiness Software Workflows

Certifications & Qualifications

Graduate Degree in Machine LearningPhD in Computer Science

Industry Keywords

Multi-Agentic SystemsLong-Horizon WorkflowsTest-Set ContaminationPublications at ACLEMNLPNeurIPSICMLICLR

About the role

Key responsibilities & impact

Own the evaluation methodology and set standards for evaluating AI agents
Work with engineers and product managers to build and improve evaluations in G2’s current offerings
Scope features, turn methods into production-ready components, and iterate based on customer and business needs
Design realistic environments and task sets reflecting business work
Develop accurate, reproducible and difficult-to-game scoring approaches using automated checks, LLM-based judges and human review
Measure consistency and reliability of evaluations
Audit tasks and graders, quantify noise, and establish when agent differences are real
Present and defend evaluation findings to buyers, customers, partners and the research community
Mentor and guide researchers and engineers on evaluation design and measurement
Review work and document standards for repeatable evaluations
Set and pursue a research agenda across agent evaluation, user simulation, LLM-as-judge reliability, voice and multimodal agents, long-horizon workflows and test-set contamination
Follow new benchmarks, papers and evaluation practices and translate them into product and roadmap decisions
Write methodology documentation, papers and public posts where appropriate
Partner with data and go-to-market teams on presentation and use of results

Requirements

What you’ll need

Experience working with evaluation design or dataset curation in multi-agentic systems
5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents
2+ years experience managing small to medium teams preferred
A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience
Track record of original ideas in machine learning, shown through first-author publications or projects led end-to-end
Ability to own a research agenda and carry long-running projects independently
Track record of shipping ML work in a product setting with engineers and product managers
Experience setting technical direction, mentoring researchers or engineers, and reviewing methodology
Familiarity with AI benchmarking and benchmark limitations
Sound statistical judgment, including experimental design, variance, significance and agreement between human and automated raters
Good judgment about practical trade-offs
Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences
Experience in agent evaluation, user simulation, or human and AI-based judging is helpful
Experience with conversational, voice, retrieval-based or tool-using AI systems is helpful
Contributions to open-source evaluation tools or public leaderboards are helpful
Familiarity with business software workflows is helpful
Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues are helpful

Benefits

Comp & perks

Equity
Flexible work
Ample parental leave
Unlimited PTO