AI Product Lead – AI Experience, Quality

Flodesk

. Manage member-facing AI behavior across prompt-to-create, agentic editing, segmentation, scheduling, analytics and recommendations .

Posted 9/23/2026full-timeRemote • California • United StatesSenior💰 $150,000 - $200,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in managing AI behavior and evaluation practices, with a strong focus on prompt iteration, member data analysis, and cross-functional collaboration. Capable of translating member needs into actionable product requirements and quality definitions.

Highest-signal resume keywords

AI Evaluation PracticePrompt IterationSystems ThinkingClear CommunicationProduct Judgment

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Prompt CreationBehavioral ScenariosQuality MonitoringData AnalysisEvaluation ToolsJSONStructured OutputsTestable HypothesesFailure TaxonomiesAgent Behavior

Soft Skills

Clear CommunicationExperimental Rigor

Tools & Technologies

LangfuseBraintrustEvaluation Platforms

Industry Keywords

AI Product BehaviorConversation DesignComputational LinguisticsContent-Generation ToolsMarketing Tools

About the role

Key responsibilities & impact

Manage member-facing AI behavior across prompt-to-create, agentic editing, segmentation, scheduling, analytics and recommendations
Create, test, version and document prompts and agent instructions
Define required context, member data, tools and product state, translating them into requirements and acceptance criteria
Own AI evaluation practice, including datasets, behavioral scenarios, scoring rubrics, regression coverage and human review
Run recurring evaluation and quality monitoring; set baselines and release gates for prompt, model and agent changes
Review traces and AI interactions to diagnose failures across prompts, context, models, tools, product structure and implementation
Fix owned issues and provide actionable changes for engineering
Turn member research, support feedback, analytics, production failures and domain expertise into priorities, experiments and regression cases
Own day-to-day product direction and learning agenda for the AI experience
Partner with product, design, marketing, copy and engineering

Requirements

What you’ll need

Shaped or meaningfully improved real LLM or agentic products, with evidence of improved user outcomes or product quality
Practical knowledge of prompts, context, tools, agent behavior and evaluation in production experiences
Hands-on experience with prompt iteration, evaluation tools and trace review
Comfortable working with JSON, schemas and structured outputs
Strong product judgment and ability to turn ambiguous member needs into clear behavior, quality definitions and testable hypotheses
Experimental rigor, including representative test sets and failure taxonomies
Systems thinking and ability to create reusable patterns
Clear communication across technical and non-technical disciplines
Ability to make tradeoffs involving member value, quality, latency, reliability and cost
Not expected to build or deploy production code, but must operate the prompt and evaluation system independently
Bonus: experience with AI product behavior, conversation design, computational linguistics or prompt systems
Bonus: experience building content-generation, creative or marketing tools
Bonus: experience with Langfuse, Braintrust or comparable evaluation and observability platforms
Bonus: experience working on products for small businesses, creators or marketers

Benefits

Comp & perks

Fully paid health insurance for individual coverage
16 weeks paid parental leave for non-birthing parents
22 weeks paid maternity leave for birthing parents
Unlimited flexible time off
401(k) match (US employees only)
$1,000 annual stipend for learning and development