FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in product management for AI products, including defining product vision, success metrics, and quality standards. Proficient in leading cross-functional teams, conducting hypothesis-driven testing, and managing production quality and performance.
Highest-signal resume keywords
Product Management ExperienceAI/ML Evaluation FrameworksAgile Software DevelopmentTechnical ArchitectureData Analysis and Metrics Optimization
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Product StrategyUser Story DevelopmentA/B TestingLLM Design and CalibrationSQLAnalytics ToolsTechnical WritingIncident ManagementQuality AssuranceHypothesis-Driven Development
Soft Skills
Strong CommunicationCollaborationInfluence SkillsCustomer ResearchMentoring
Tools & Technologies
Agile ScrumDevOpsCI/CDAPIsTableauRESTEDISOAPProduct TelemetryQuality Scorecards
Certifications & Qualifications
Certified Scrum MasterCertified Scrum Product OwnerAgile Certified Practitioner
Industry Keywords
AI ProductsConversational AIHuman-in-the-Loop ValidationProduction QualityCustomer ExperienceTechnical DebtMulti-Model EvaluationRed-TeamingBias EvaluationCarrier-Grade Standards
Tech Stack
Tools & technologiesSOAPSQLTableau
About the role
Key responsibilities & impact- Own product strategy, roadmap, lifecycle, and measurable outcomes for one or more AI products or reusable AI capabilities
- Define product vision, success metrics, quality and evaluation metrics, and investment priorities
- Analyze market data, customer insights, product telemetry, evaluation results, and production data
- Conduct hypothesis-driven testing and experiments, including paper prototypes, A/B testing, and offline evaluation
- Lead customer discovery, validation, user testing, prototyping, and MVP development
- Write detailed product requirements, user stories, acceptance criteria, and measurable AI quality criteria
- Own launch-readiness validation and cross-functional readiness reviews before General Availability
- Collaborate with Technical Architects, engineering teams, PMs, Product Owners, DevOps, Architecture, and Engineering to deliver scalable products
- Coordinate with go-to-market and commercialization teams on launch strategies
- Resolve technical and business challenges and make product trade-off decisions
- Prioritize backlogs, technical debt, production defects, and long-term technical investments
- Support sales, marketing, and stakeholder teams with product and technical knowledge
- Own production product quality and performance, including critical-defect response and stakeholder communications
- Drive root-cause analysis and resolution of production incidents and AI behavior issues
- Build scenario libraries, simulated-user test suites, and end-to-end agent tests
- Design and calibrate LLM-as-judge evaluations, rubrics, prompts, and automated release gates
- Conduct human-in-the-loop validation and maintain golden datasets
- Define carrier-grade AI standards for accuracy, latency, task success, safety, and customer experience
- Build and maintain multi-model evaluation pipelines and quality scorecards
- Manage quality feedback loops with model providers and engineering partners
- Monitor production AI regressions through alerting, dashboards, and on-call protocols
- Build relationships across Network, IT, Business, Marketing, Care, Legal, Privacy, Compliance, Finance, and other teams
- Communicate roadmaps, launches, and quality results to internal audiences and partners
- Mentor junior product managers and quality engineers
- Maintain professional knowledge through workshops, publications, and professional societies
Requirements
What you’ll need- Bachelor's Degree plus 7 years of related work experience OR advanced degree with 5 years of related experience; acceptable areas include Computer Science, Engineering, IT, Data Science, or equivalent experience
- At least 18 years of age
- Legally authorized to work in the United States
- 6+ years of experience in hands-on technical roles writing production code, solution engineering, or technical architecture in a large-scale company, application, or product environment
- 7+ years of relevant Product Management experience in an agile software product development environment
- At least 3 years focused on AI/ML, conversational or agentic AI, or evaluation frameworks for production systems
- Hands-on experience designing LLM and conversational AI evaluation frameworks, including golden datasets, evaluation harnesses, human-in-the-loop validation, prompt regression testing, and red-teaming protocols
- Experience testing AI agents performing multi-step actions, including task completion and tool-call accuracy metrics, trajectory review, simulated-user and scenario-based testing, and failure analysis
- Experience designing and calibrating LLM-as-judge systems, including rubric and judge-prompt design, human-rater agreement, bias and drift controls, and automated release gates
- Understanding of carrier-grade production quality, SLAs, incident management, on-call rotations, and production change governance
- Strong business, customer research, product vision, feature definition, technical architecture, development, and execution skills
- Strong analytical skills using product data to drive decisions
- Advanced customer experience knowledge, including beta programs, reference customer pipelines, and feedback-to-product workflows
- Deep understanding of platform technologies including security, performance, optimization, and API integration; advanced knowledge of the assigned product's full technology stack
- Proven success directing matrixed resources and delivering software across multiple teams using Agile Scrum methodologies and backlog/project management tools
- Experience with external partners, vendors, ecosystem players, model providers, and engineering partners
- Experience applying design thinking, lean startup, rapid prototyping, and hypothesis-driven development
- Ability to develop Initiatives, Features, and User Stories consumable by DevOps teams
- Experience delivering large and complex business/technology initiatives as a Product Manager or lead technology role
- Strong communication, requirements elicitation, technical writing, collaboration, and influence skills
- Knowledge of integration patterns, APIs, and protocols such as REST, EDI, and SOAP
- Strong understanding of AI technologies, LLMs, conversational and agentic AI, and ML platforms
- Proficiency with analytics tools, SQL, Tableau or equivalent analytical solutions, and metrics-based product optimization
- Extensive knowledge of DevOps, CI/CT/CD, Agile environments, and automated evaluation gates in release pipelines
- Preferred: multi-model and multi-vendor quality benchmarking and management
- Preferred: AI safety, red-teaming, fairness and bias evaluation, and responsible AI practices
- Preferred certifications: Certified Scrum Master, Certified Scrum Product Owner, Agile Certified Practitioner
Benefits
Comp & perks- Annual stock grant
- Employee stock purchase plan
- 401(k)
- Free, year-round money coaches
- Medical, dental, and vision insurance
- Flexible spending account
- Paid time off
- Up to 12 paid holidays
- Paid parental and family leave
- Family building benefits
- Back-up care
- Enhanced family support
- Childcare subsidy
- Tuition assistance
- College coaching
- Short- and long-term disability
- Voluntary AD&D coverage
- Voluntary accident coverage
- Voluntary life insurance
- Voluntary disability insurance
- Voluntary long-term care insurance
- Mobile service and home internet discounts
- Pet insurance
- Commuter and transit programs
- Annual bonus or periodic sales incentive or bonus
- Career growth and development opportunities
