FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in defining evaluation metrics for AI systems, curating golden datasets, and ensuring compliance and privacy safeguards. Proficient in monitoring AI components in production and collaborating with software engineers to implement solutions.
Highest-signal resume keywords
AI System EvaluationGolden Dataset CurationMonitoring AI/LLM ComponentsLangfuse ObservabilityMulti-Agent Systems Development
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Evaluation Metrics DefinitionData AnalysisSensitive-Data DetectionEnd-to-End AuditabilityTool-Calling OrchestrationAI EngineeringAgentic Systems DevelopmentAutomation Opportunities IdentificationCollaboration with Software EngineersGeneral Technical Capability
Tools & Technologies
LangfuseLLM Observability Tools
Industry Keywords
AIBankingCompliance SafeguardsProduction MonitoringHistorical Data Analysis
About the role
Key responsibilities & impact- Define and implement evaluation metrics tailored to each AI use case
- Curate and maintain golden datasets comparing system behaviour with ideal results
- Analyse historical data to determine what system evaluations should check
- Monitor AI/LLM components in production and identify regressions
- Build and manage agentic systems, including LLM agents and multi-agent systems with tool-calling and orchestration
- Design and maintain privacy and compliance safeguards, including sensitive-data detection and redaction
- Ensure AI systems are auditable end to end
- Build and maintain observability for agentic components using Langfuse
- Partner with software engineers to take projects into production
- Help prioritise automation opportunities across Banking
- Help other teams adopt solutions developed by the team
- Contribute to end-to-end systems within the Banking AI Foundation & Ops Efficiency team
Requirements
What you’ll need- Experience relevant to data science, AI engineering, AI system evaluation, or agentic systems (specific minimum years not stated)
- Ability to define and implement evaluation metrics for AI systems
- Ability to curate and maintain golden datasets
- Ability to analyse historical data to determine evaluation requirements
- Experience monitoring AI/LLM-based components in production
- Experience building single-purpose LLM agents and multi-agent systems with tool-calling and orchestration
- Knowledge of privacy and compliance safeguards, including sensitive-data detection and redaction
- Experience with end-to-end system auditability
- Experience with Langfuse or an LLM observability tool
- Ability to collaborate with software engineers to take projects to production
- General technical and coding capability, assessed during the recruitment process
Benefits
Comp & perks- Equal treatment and non-discrimination in recruitment
- Supportive and transparent hiring experience
- Opportunity to work with a multicultural European team
- Opportunity to contribute to systems supporting over one million small businesses and 15,000 accountants across Europe