Associate Director, AI Product Engineer – Financial Institution Operations & Technology

Berkeley Research Group (BRG)

. Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use .

Posted 11/3/2026full-timeRemote • United StatesSenior💰 $180,000 - $250,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in AI evaluation, including test design, scoring, and failure investigation, while ensuring output accuracy and compliance with sensitive data controls. Strong ability to communicate findings and mentor colleagues in a collaborative environment.

Highest-signal resume keywords

AI Evaluation OwnershipPython ProgrammingAutomated Test Software DevelopmentFinancial Services ExperienceStrong Communication Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Test DesignScoringFailure InvestigationData Source ConnectionError Rate InterpretationOutput Accuracy MeasurementAutomated Scoring VerificationSample SelectionControl ImplementationRepeatable Results Documentation

Soft Skills

TeamworkMentoringCommunicationClient InteractionCoaching

Industry Keywords

Financial ServicesRegulated EnvironmentAI ProductsSensitive Data ControlsConsulting Experience

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use
Build reusable software that runs tests, scores results, and compares versions
Work with engineers to reproduce failures, identify causes, and verify fixes
Contribute product code to improve accuracy and control
Set pass criteria and scoring rules with business and risk teams
Maintain representative test cases, including incomplete records and rare, high-impact errors
Measure factual accuracy, missing information, incorrect additions, and support from cited sources
Check automated scoring against expert review
Compare product versions on the same tests and report results by task and error severity
Verify extracted facts against original records, including amounts, dates, totals, and citation support
Test when the product should answer, request clarification, or refer work to a person
Test for invented facts, unsupported recommendations, unfair outcomes, sensitive-information disclosure, and safeguard circumvention
Protect client data, restrict access and permitted actions, and use only approved services
Document repeatable test results, unresolved failures, and limitations
Recommend release, restricted use, or further development and support independent review and client approvals
Sample live outputs, track error rates and user corrections, and trigger investigation or suspension when quality falls below agreed standards
Explain findings, coach colleagues, develop reusable tests and controls, and support proposals, technical estimates, and recruiting

Requirements

What you’ll need

Minimum 5 years of working experience, including a minimum of 1 year of experience in AI evaluation
Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation
Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions
Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring
Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review
Experience in financial services or another regulated environment, with ability to translate business requirements into practical tests and controls
Strong communication and teamwork skills, ability to mentor colleagues, and willingness to travel to client sites as needed
Preferred: banking experience in lending, payments, servicing, collections, or default operations
Preferred: experience independently evaluating AI products, checking automated scoring against expert judgment, or testing security and fairness in financial institutions
Preferred: experience evaluating document review, summaries, information search, or automated workflows, including comparisons across AI services and conventional software
Preferred: consulting or product delivery experience, including client discovery, implementation planning, and user adoption
Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future

Benefits

Comp & perks

Equal opportunity employment
Reasonable accommodation for essential job functions
Travel opportunities to client sites as needed