Senior Product Manager, AI Evaluation – Quality

Circana

. Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana’s AI agents .

Posted 10/28/2026full-timeRemote • United StatesSenior💰 $150,000 - $159,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in evaluation strategies and methodologies for AI agents, with a strong focus on statistical evaluation and performance metrics. Capable of designing structured tests and communicating technical findings effectively to diverse audiences.

Highest-signal resume keywords

Product Evaluation ExperienceStatistical EvaluationAI Agent EvaluationAnalytical ReasoningCommunication Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Evaluation StrategiesTesting MethodologiesPerformance MetricsData ScienceQuality AssuranceExperimentationLLM-as-a-Judge ScoringBenchmark DevelopmentTest Scenario DesignProduction Trace Analysis

Soft Skills

Attention to DetailAdaptabilityClear Communication

Certifications & Qualifications

Bachelor’s Degree in Quantitative Discipline

Industry Keywords

AI AgentsLiquid AIAgentic WorkflowEvaluation FrameworksPerformance Monitoring

About the role

Key responsibilities & impact

Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana’s AI agents
Build evaluation frameworks for AI agent and agentic workflow capabilities
Partner with the Liquid AI Product Leader, engineering, and product verticals to validate agent performance before release
Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods
Evaluate agent quality, accuracy, consistency, reliability, safety, and client value
Identify failure patterns and translate findings into product improvement recommendations
Establish ongoing monitoring of production traces to detect post-launch performance changes
Document evaluation results and communicate findings to technical and business audiences
Promote consistent evaluation standards across Liquid AI product verticals

Requirements

What you’ll need

4–6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field
Understanding of statistical evaluation, testing methodologies, and performance metrics
Familiarity with evaluating AI agents or LLM-based systems, including LLM-as-a-judge scoring and production trace analysis
Ability to design structured tests for complex or ambiguous use cases
Strong analytical reasoning and attention to detail
Comfortable operating in an early-stage, fast-moving environment with evolving requirements
Ability to communicate technical findings clearly to varied audiences
Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience

Benefits

Comp & perks

Flexible and accommodating work environment
Paid time off
Medical, dental, and vision insurance
401(k)
Bonus pay eligibility
Reasonable accommodation support for applicants with disabilities
Background check conducted in accordance with local legislation
AI-assisted resume screening