FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in quality evaluation and measurement, with a focus on defining metrics, managing golden datasets, and ensuring inter-rater reliability. Proven ability to lead product lifecycle management and present methodologies to clients.
Highest-signal resume keywords
Quality Evaluation OwnershipGolden Dataset ManagementRubric DesignInter-Rater ReliabilityProduct Lifecycle Management
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Quality Metrics DefinitionFirst-Pass-Right MeasurementRegression DetectionHuman Quality-Control MechanismEval Loop DevelopmentClient RFP Response EvidenceMetric OwnershipSubjective Domain Measurement
Soft Skills
Client PresentationMethodology Defense
Tools & Technologies
Adevals.ai
Industry Keywords
Quality ControlMeasurement ProblemProduct RoadmapUser-Centric Design
About the role
Key responsibilities & impact- Own the charter for Evals & Quality, including the problem, success metrics, and scope
- Define and improve core quality metrics such as first-pass-right, coverage, and regression detection
- Own rubrics and calibration for automated judges and human raters
- Design and run the human quality-control mechanism
- Own curation, coverage, and versioning of the golden dataset
- Build reusable eval loops for other teams
- Own adevals.ai as a product, including its design quality
- Provide evidence for client RFP responses and creative enablement packages
- Partner with Agent Architects on client-specific quality bars using configuration and rubrics
- Run the full product lifecycle from discovery through post-launch audit
- Review every launch 14 days after launch against its stated hypothesis
Requirements
What you’ll need- Experience owning a quality, evals or ML measurement problem as a product, with its own roadmap and users
- Experience shipping measurement in a subjective domain
- Direct experience with golden datasets, rubric design, judge calibration, and inter-rater reliability
- Experience deriving a product decision from evidenced customer need in a domain you had to learn
- Experience owning a metric that changed decisions outside your own team
- Experience presenting or defending methodology to clients/buyers
Benefits
Comp & perks- 25 days PTO plus public holidays, although we operate a Flexible Time Off scheme
- Health insurance / private medical cover
- Monthly stipend towards wellness, fitness, and learning and development
- Remote - work from anywhere in your home country
- Enhanced parental leave policies, whether you become a parent through birth, adoption or surrogacy
- Access to our Pencil office in The Shard, London for our UK employees
- Flexible working hours