FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates extensive experience in designing and implementing LLM-based evaluation frameworks, with a strong focus on assessing agent response quality and performance. Proficient in collaborating with stakeholders to maintain golden datasets and instrumenting AI agents using Langfuse.
Highest-signal resume keywords
LLM-Based Evaluation FrameworksLangfuse InstrumentationAgent Response Quality AssessmentCollaboration With Business AnalystsEnd-To-End Agent Evaluation
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
LLM EvaluationAgent Performance MonitoringResponse Quality TestingTool Usage AssessmentReasoning Workflow Analysis
Tools & Technologies
Langfuse
Industry Keywords
Golden DatasetsAI AgentsAgent Execution TracingEvaluation Scores CaptureTroubleshooting Failures
About the role
Key responsibilities & impact- Evaluate LLM-powered agents as the primary focus of the role
- Design, implement, and run LLM-as-a-Judge evaluation frameworks
- Assess agent response quality, accuracy, relevance, and reliability
- Collaborate with Business Analysts and stakeholders to create, validate, and maintain golden datasets
- Instrument AI agents with Langfuse
- Trace agent execution, capture evaluation scores, monitor performance, and troubleshoot failures
- Test agent behavior, tool usage, reasoning workflows, and response quality
Requirements
What you’ll need- 14+ years of experience
- Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability
- Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets for agent evaluation
- Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures
- Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality
- Actual implementation experience with agent evaluation frameworks; theoretical knowledge or general GenAI/LLM development alone is insufficient