Technical Product Manager – Pedagogy, AI Quality

Workshop Venture Partners

. Define quality standards and targets for high-quality reading and writing scaffolds based on research in literacy, special education, and multilingual learner instruction .

Posted 11/10/2026full-timeRemote • United StatesMid-LevelSenior💰 $130,000 - $150,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in defining quality standards for literacy instruction and AI evaluation, with a strong focus on scaffolding for diverse learners. Proficient in collaborating with engineering teams to enhance product features and usability while maintaining high educational standards.

Highest-signal resume keywords

Pedagogical Expertise in Literacy InstructionHands-On Experience with AI EvaluationsAI Product FluencyCollaboration with EngineersStrong Communication Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Error AnalysisPrompt Writing and TestingDataset MaintenanceRubric DesignModel ComparisonsRegression TestingProduct Specification WritingAI Metrics DefinitionScaffolding DesignUser Research

Soft Skills

Strong InitiativeSharp JudgmentClear Writing

Tools & Technologies

LangSmithBraintrustArizeLangfuseAnnotation Platforms

Industry Keywords

Special EducationMultilingual Learner EducationEdTechAccessibility StandardsAssistive Technology

About the role

Key responsibilities & impact

Define quality standards and targets for high-quality reading and writing scaffolds based on research in literacy, special education, and multilingual learner instruction
Build and maintain datasets, rubrics, and evaluation suites
Conduct regular error analysis and trace review using human labeling, LLM-as-judge, and rule-based evaluations
Write, test, and refine prompts
Run model comparisons and regression tests
Review educator signals, support-ticket data, product analytics, and conduct user research with students and teachers
Maintain a prioritized backlog of AI quality and scaffold UX issues
Write clear product specifications
Partner with engineering to scope, build, and verify fixes, product updates, and new features
Own the quality and usability of Medley’s scaffold suite over time
Prioritize product improvements and new features to support teachers and improve student outcomes

Requirements

What you’ll need

Pedagogical expertise, especially in scaffolding literacy instruction for struggling readers, special education, and/or multilingual learner education
Ideally from classroom teaching, intervention, instructional coaching or curriculum design
Hands-on experience with AI evals, including human labeling, LLM-as-judge rubric design, AI metrics targets definition, and error analysis of LLM outputs
AI product fluency; daily use of LLMs and experience designing complex AI systems for products
Experience defining new features and collaborating with engineers to deliver them
Ability to clearly communicate problems and solutions and work through technical tradeoffs and UX questions
Strong initiative, sharp judgment, and clear writing
Graduate degree and/or significant direct experience in special education, reading/literacy, ESL/ELD or bilingual education is nice to have
Familiarity with AI eval and observability tools such as LangSmith, Braintrust, Arize, Langfuse, or annotation platforms is nice to have
Proficiency in a language other than English is nice to have
Prior work at an edtech company, or on content quality, assessment design or psychometrics is nice to have
Familiarity with accessibility standards such as WCAG and assistive technology is nice to have
Must address work authorization sponsorship question for the United States

Benefits

Comp & perks

Equity
Work directly with founders and engineering leadership
Meaningful technical and design ownership
Opportunity to shape a product improving access to learning for students nationwide