Evaluation Engineer

Freshfields Bruckhaus Deringer

. Build and run tests that assess the quality and reliability of the firm’s AI applications .

Posted 10/16/2026full-timeNew York City • New York • United StatesMid-LevelSenior💰 $100,000 - $200,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in evaluating AI applications through automated testing, data analysis, and performance assessment, while maintaining a strong understanding of software development practices and large language models.

Highest-signal resume keywords

Python ProgrammingAutomated TestingCI/CD IntegrationAPIsAnalytical Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Java ProgrammingC# ProgrammingSoftware Development LifecycleStatistical MeasurementTest Dataset PreparationFunctional TestingIntegration TestingEvaluation of AI ApplicationsError HandlingRegression Suites Maintenance

Soft Skills

Attention to DetailCuriosityAbility to LearnClear DocumentationSeeking Guidance

Tools & Technologies

JiraJenkinsGitCoding AgentsAutomated Graders

Industry Keywords

AI ApplicationsLarge Language ModelsEvaluation DatasetsModel DefectsPerformance Assessment

Tech Stack

Tools & technologies

JavaJenkinsPython

About the role

Key responsibilities & impact

Build and run tests that assess the quality and reliability of the firm’s AI applications
Help maintain evaluation datasets, investigate failures and provide evidence of AI system performance on legal and business tasks
Implement automated evaluation cases and supporting scripts using established frameworks
Prepare and maintain test datasets, source documents and reference examples with legal specialists
Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation
Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling
Review outputs and execution traces to identify and reproduce failures and document supporting evidence
Distinguish data, retrieval, model and application defects
Compare results across model, prompt and application versions and report changes with uncertainty and limitations
Support calibration of automated graders by comparing scores with human reviews
Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases

Requirements

What you’ll need

Minimum five years’ experience developing software demonstrated through employment, placements or substantial projects using programming languages such as Python, Java or C#
Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes
Ability to use coding agents effectively for development, testing and analysis, with awareness of prompting, context management, validation, security, responsible use of generated code and when to seek human review
Working knowledge of APIs, Git, automated testing, software development lifecycle practices and CI/CD tools such as Jira and Jenkins
Curiosity and ability to learn in a rapidly evolving AI landscape and apply relevant learning pragmatically with appropriate guidance
Foundational understanding of statistics and measurement, including sampling, precision and recall
Strong analytical skills, attention to detail and ability to document issues clearly and seek technical or legal guidance when needed
Practical experience evaluating AI applications is a plus but is not required; relevant academic, internship or project experience will also be considered