Program Manager, Human Evaluations – Recommendations
Netflix
Posted: 10/20/2026
Onboarding Program Manager
The Sequoia Project
Posted: 10/20/2026
Senior Program Manager
The JPI Group
Posted: 10/20/2026
Principal PM – Project/Program Management
Analog Devices
Posted: 10/20/2026PMP
Digital Program Manager – Release Management
Caterpillar Inc.
Posted: 10/20/2026AzureERPSDLC
Senior Physician Relations Program Manager
Medtronic
Posted: 10/20/2026
Program Manager, Client Services – Clinical Trial Support
Valeris
Posted: 10/20/2026PMP
Senior Program Manager
Affirm
Posted: 10/20/2026
Program Manager Lead – Strategic Programs
USAA
Posted: 10/20/2026
Customer Experience Program Manager – Product Support
Nametag
Posted: 10/20/2026
Program Manager 3
Mission Technologies, a division of HII
Posted: 10/20/2026
Senior Analyst, Program Cost Controls
RTX
Posted: 10/20/2026
Compliance Program Manager – Regulatory Compliance
OpenAI
Posted: 10/20/2026
AI Operations Program Manager
Gusto
Posted: 10/20/2026
Senior Manager – Program Management, Optimization
American Express
Posted: 10/20/2026
Senior Program Manager – Power Systems, Electric Propulsion
Honeywell Aerospace Technologies
Posted: 10/20/2026Strategic Defense Program Manager
Honeywell Aerospace Technologies
Posted: 10/20/2026PMP
Professional Services Program Manager
Palo Alto Networks
Posted: 10/20/2026CloudCyber SecurityPMP
Director, Technical Program Management
Capital One
Posted: 10/20/2026AWSCloudDistributed SystemsPMP
KPI Program Manager
Lamb Weston
Posted: 10/20/2026Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in managing human evaluation and data operations initiatives, with a strong focus on operational efficiency, evaluation design, and cross-functional collaboration. Proven ability to implement process improvements and manage vendor relationships effectively.
Highest-signal resume keywords
Human Evaluation ManagementOperational Efficiency ImprovementProgram ManagementEvaluation Design ExpertiseVendor Performance Management
About the role
Key responsibilities & impact- Lead end-to-end execution of human evaluation and data operations initiatives from intake and scoping to delivery
- Partner with Product Management to translate recommendation strategy and roadmap priorities into evaluation designs, success metrics, and timelines
- Manage human annotation vendors, including capacity planning, SLAs, quality performance, and escalations
- Drive operational efficiency across evaluation programs, including cost per task, throughput, turnaround time, and speed to IRR
- Develop clear, scalable guidelines and scoring rubrics for consistent rater judgment
- Oversee rater onboarding, calibration, and QA workflows
- Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality
- Anticipate and resolve bottlenecks and blockers
- Coordinate product, ML, data science, and vendor partners for aligned execution of evaluations at scale
- Act as a thought partner to Product Management and help build shared language around recommendation quality
Requirements
What you’ll need- 4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in ML environments
- Track record of implementing process improvements or quality control systems for data collection needs
- Prior experience managing human annotation vendors, raters, or data labeling teams, including vendor performance and cost management
- Experience improving operational efficiency of annotation programs, including throughput, turnaround, and cost
- Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols
- Working fluency in film and TV content, including genres, formats, and talent
- Proven ability in end-to-end management of complex, cross-functional programs
- Strong Program Management skills and clear accountability for successful delivery
- Experience with human labeling platforms
- Excellent written and verbal communication skills
- Ability to synthesize feedback into clear recommendations and process improvements
- Familiarity with rater subjectivity and bias in taste-based judgments and evaluation design
- Strong organizational skills and executional focus
- Experience with personalization, recommender systems, search, or ranking evaluation is nice to have
- Familiarity with offline human evaluation, online experimentation, and engagement metrics is nice to have
- Experience evaluating at the list or page level is nice to have
- Experience partnering with Product Managers and Data Scientists on metric definition is nice to have
Benefits
Comp & perks- Health Plans
- Mental Health support
- 401(k) Retirement Plan with employer match
- Stock Option Program
- Disability Programs
- Health Savings and Flexible Spending Accounts
- Family-forming benefits
- Life and Serious Injury Benefits
- Paid leave of absence programs
- Flexible time off for full-time salaried employees
ATS Keywords
✓ Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Data CollectionAnnotation OperationsProcess ImprovementQuality Control SystemsEvaluation GuidelinesScoring RubricsHuman Labeling PlatformsCross-Functional Program ManagementFeedback SynthesisEngagement Metrics
Soft Skills
Excellent Communication SkillsStrong Organizational SkillsExecutional Focus
Industry Keywords
Machine LearningFilm and TV ContentPersonalizationRecommender SystemsRanking EvaluationRater SubjectivityBias in Judgments