FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing and implementing evaluation systems for GenAI solutions, with a strong focus on LLMs and traditional ML models. Proficient in developing automated evaluation pipelines, statistical validation, and observability frameworks to ensure quality and compliance in AI applications.
Highest-signal resume keywords
LLM-Based Applications ExperienceEvaluation Frameworks ExpertisePython Programming SkillsStatistical Validation KnowledgeCloud Platforms Experience
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Evaluation MethodologiesAutomated Evaluation PipelinesStatistical ValidationModel Evaluation MetricsPrompt EngineeringData Annotation WorkflowsRegression TestingQuality GatesSampling TechniquesExperimental Design
Soft Skills
Clear Communication SkillsTeam CollaborationPresentation Skills
Tools & Technologies
LangSmithArize PhoenixWhyLabsMLflowAzureAWSGCPRagasDeepEvalLangfuse
Industry Keywords
GenAI SolutionsModel DriftHallucination RiskQuality CriteriaResponsible AI StandardsAI Risk and GovernancePerformance CompetenciesAgentic SystemsNLPQA Automation
Tech Stack
Tools & technologiesAWSAzureCloudGoogle Cloud PlatformPython
About the role
Key responsibilities & impact- Design, build, and operate evaluation systems for GenAI solutions
- Establish standardized, automated evaluation frameworks for LLMs and traditional ML models
- Deploy observability solutions for model drift, hallucination risk, latency, throughput, and token consumption
- Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications
- Build reusable evaluation frameworks, templates, and libraries
- Translate business requirements into measurable quality criteria and acceptance thresholds
- Develop automated evaluation pipelines integrated into CI/CD with regression testing and quality gates
- Implement offline and online evaluation, including benchmarks, golden datasets, production monitoring, sampling, drift, and quality alerts
- Support A/B and shadow testing for model, prompt, and architecture changes
- Design and maintain LLM-as-a-judge evaluators with rubric design, bias mitigation, and human-label calibration
- Establish human annotation workflows and labeling guidelines
- Build golden datasets, adversarial and edge-case suites, and synthetic test-data generation approaches
- Apply data-handling practices including de-identification, synthetic data, and access controls
- Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing
- Produce evaluation evidence and reporting for security, privacy, compliance, and model-risk reviews
- Contribute to responsible AI standards and release-readiness criteria
- Embed with product pods to onboard teams to evaluation practices and coach them toward ownership
- Build dashboards and reporting for quality and cost/latency trends
- Evaluate and recommend evaluation tooling and platforms; document standards and best practices
- Perform other duties as assigned
- Travel as required
Requirements
What you’ll need- Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred
- 4+ years in software, ML, or data engineering
- At least 1-2 years working hands-on with LLM-based applications
- Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs
- Strong understanding of statistical validation and model evaluation metrics
- Strong Python skills and software engineering fundamentals, including testing, version control, CI/CD, and code review
- Practical experience evaluating LLM/RAG systems, including retrieval metrics such as recall, precision, MRR/nDCG and generation metrics
- Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation
- Working knowledge of statistics and experimental design, including sampling, confidence intervals, significance testing, and inter-rater agreement
- Familiarity with at least one eval or observability tool, such as Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent
- Experience with cloud platforms such as Azure, AWS, or GCP and their managed LLM services
- Clear communication skills and ability to work with engineers, product owners, SMEs, and risk stakeholders
- Excellent oral and written communication skills, including presentation skills
- Ability to work in a team environment
- Ability to meet or exceed Performance Competencies
- Preferred: experience evaluating agentic systems, background in ML evaluation/search/relevance/NLP/QA automation, LLMOps/monitoring/observability experience, familiarity with AI risk and governance frameworks, data annotation workflows, and enterprise controls
Benefits
Comp & perks- Work-life balance
- Reasonable accommodations when applicable and appropriate
- Equal Opportunity Employer
- Drug-Free Workplace