CENTRL Inc

Technical Product Manager

• Own the accuracy, reliability, and cost-efficiency of the AI behind CentrlX
• Build and own the evaluation foundation, including golden datasets, grading rubrics, LLM-as-judge pipelines calibrated against human labels, and regression suites
• Define measurable quality standards for document digitization, extraction, retrieval, groundedness, summaries, responses, evaluations, and multi-step agent runs
• Evaluate agent behavior, including tool selection, retrieval quality, step sequencing, and finished deliverable quality
• Convert client failures into permanent evaluation cases
• Benchmark models across providers on accuracy, latency, and cost
• Own model migrations and determine model deployment by workflow
• Track AI spend and optimize cost through routing, model tiering, caching, and context strategy
• Define logging and tracing requirements for prompts, retrieved context, outputs, tool calls, token counts, and latency
• Build in-app feedback capture with Product and Design
• Build, label, curate, and maintain evaluation datasets with holdouts and rotation
• Serve as the point of contact for AI quality escalations; triage, reproduce, identify root causes, and close the loop
• Write user stories and acceptance criteria and implement prompt, configuration, and model changes
• Publish regular quality and cost readouts
• Work with domain practitioners to encode industry judgment into evaluation rubrics

Technical Product Manager, Data Science