Senior Technical Product Manager – Agent Evaluation, Observability

American Express

. Own the roadmap for shared agent evaluation and observability capabilities from user discovery and requirements through launch, adoption and improvement.

Posted 11/4/2026full-timeUnited StatesSenior💰 $123,000 - $215,250 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in product management and evaluation, with a strong focus on defining metrics, interpreting evidence, and ensuring compliance with privacy and data governance standards. Proven ability to lead cross-functional teams in delivering AI-enabled systems and technical products that drive measurable user outcomes.

Highest-signal resume keywords

Product ManagementEvaluation and ObservabilityAI or Agent-Enabled SystemsCross-Functional LeadershipTechnical Troubleshooting

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Product DevelopmentAnalyticsQuality ToolingMetrics DefinitionAutomated TestingHuman ReviewScenario DesignModel-Assisted ScoringTechnical PlatformsAPI Interactions

Soft Skills

Clear CommunicationRoadmap PrioritizationCross-Functional CollaborationProblem-SolvingCustomer Outcome Focus

Tools & Technologies

Telemetry StandardsEvaluation FrameworksMCPFunction CallingTracesLogs

Industry Keywords

Financial ServicesRegulated IndustryData GovernancePrivacy ComplianceUser Adoption

About the role

Key responsibilities & impact

Own the roadmap for shared agent evaluation and observability capabilities from user discovery and requirements through launch, adoption and improvement.
Partner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics and acceptance criteria.
Define requirements for automated evaluation, human review and tools that help teams diagnose failures.
Define product requirements for traces, events, diagnostic views and review tools connecting customer intent, tool or API calls and observed outcomes where data access permits.
Work with engineering, analytics and enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation and standardize evidence.
Identify where limited access to partner data prevents a complete assessment.
Partner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access and required approvals.
Define measures including task completion, correct tool use, response quality, failure and recovery rates, latency and cost.
Define how testing and monitoring work across controlled test environments, end-to-end partner integrations and production, using findings to prioritize improvements.
Measure adoption and usefulness of evaluation workflows, including review effort, coverage and time to identify problems.
Translate measurement gaps into a prioritized backlog, clear requirements and actionable acceptance criteria.
Recommend when to extend existing tools, integrate a solution or build a missing capability.
Build privacy and data governance into evaluation and telemetry requirements, including permitted data uses, access, minimization and retention.
Communicate uncertainty, release risks and tradeoffs clearly.

Requirements

What you’ll need

6-8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.
Experience owning evaluation, testing, observability, analytics or related technical products through launch, adoption and iteration, with measurable user or business outcomes.
Practical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.
Ability to define metrics, interpret evidence and identify sampling, coverage, or measurement limitations.
Familiarity with traces, logs, APIs and technical troubleshooting.
Strong product discovery, roadmap and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.
Cross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints and privacy or risk requirements.
Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring or production failure analysis.
Familiarity with MCP, function calling, telemetry standards or evaluation frameworks; ability to use code or queries to investigate evidence.
Experience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.
Employment eligibility to work with American Express in the United States is required; the company will not pursue visa sponsorship.
Bachelor's degree or equivalent experience is preferred.

Benefits

Comp & perks

Competitive base salaries
Bonus incentives
6% Company Match on retirement savings plan
Free financial coaching and financial well-being support
Comprehensive medical, dental, vision, life insurance, and disability benefits
Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need
20+ weeks paid parental leave for all parents, regardless of gender, offered for pregnancy, adoption or surrogacy
Free access to global on-site wellness centers staffed with nurses and doctors (depending on location)
Free and confidential counseling support through our Healthy Minds program
Career development and training opportunities