Senior AI Product Manager – Evals & Annotations

Arize AI

. Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle .

Posted 11/10/2026full-timeRemote • United StatesSenior💰 $200,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in product management for technically complex developer-facing products, with a strong focus on LLM evaluations, AI development workflows, and backend systems. Capable of driving product strategy, collaborating with cross-functional teams, and delivering intuitive user experiences.

Highest-signal resume keywords

Product Management ExperienceLLM Evaluations UnderstandingBackend Systems KnowledgeTechnical Prototyping SkillsStrong Communication Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Product StrategyEvaluation Lifecycle ManagementAI Development WorkflowsBackend Systems ArchitecturePrototypingDebuggingTechnical ImplementationHuman Feedback IntegrationExperimentationTechnical Discussions

Soft Skills

OwnershipInitiativeCollaborationLow-Ego ApproachAbility to Navigate Ambiguity

Tools & Technologies

Developer ToolsAI InfrastructureML PlatformsEvaluation ToolsAnnotation Workflows

Industry Keywords

AI ApplicationsAutonomous AgentsManaged AgentsTechnical DepthCustomer Conversations

About the role

Key responsibilities & impact

Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle
Set product strategy and roadmap, balancing customer needs, platform investments, UX improvements, and emerging AI and agent-development opportunities
Develop strong workflow opinions, independently prototype solutions, and partner with Design on intuitive, polished experiences
Shape the future of evaluations for autonomous and managed agents, including opportunities within the broader Dynatrace ecosystem
Work with Engineering on evaluation execution, backend systems, orchestration, and data infrastructure
Contribute to technical discussions, implementation approaches, and tradeoffs
Have multiple customer conversations each week to understand evaluation practices, uncover problems, and validate product direction
Collaborate with cofounders and cross-functional teams on product direction, design exploration, and prioritization
Make prioritization decisions and drive products from problem definition through launch, adoption, and iteration

Requirements

What you’ll need

Significant product management experience independently owning technically complex developer-facing products; exact years flexible for candidates with exceptional technical depth, product judgment, and ownership
Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and iteration on AI applications and agents
Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products
Solid understanding of backend systems and software architecture
Strong technical depth and hands-on engineering ability
Ability to debug issues independently and navigate code when needed
Ability to put up small fixes or PRs
Ability to build prototypes and technical proofs of concept
Ability to propose technical implementations and reason through tradeoffs
Ability to hold credible technical discussions with engineers
Excellent product and UX instincts; ability to define and prototype core product experiences
Strong ownership, initiative, and bias toward action
Ability to navigate ambiguity and make tradeoffs
Collaborative, low-ego approach to working with founders, engineers, and designers
Experience working closely with technical customers
Strong communication skills and ability to move between strategy and execution

Benefits

Comp & perks

Competitive equity package
Medical insurance
Dental insurance
Vision insurance
401(k) plan
Unlimited paid time off
Generous parental leave plan
Mental and wellness support
WFH monthly stipend for co-working spaces
Optional in-person work in New York City and the San Francisco Bay Area
Regular chats with industry experts, researchers, and ethicists
Culturally conscious events such as LGBTQ trivia during pride month
Active Lady Arizers subgroup