Research Engineer, Benchmarks – Evaluation

Ramp

. Build benchmarks from real finance workflows, including expense policy review, invoice coding, month-end close, and spreadsheet Q&A .

Posted 11/3/2026full-timeUnited StatesMid-LevelSenior💰 $204,400 - $352,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building LLM benchmarks and evaluations, with strong proficiency in Python and software engineering. Possesses a solid understanding of statistical judgment and experience in financial workflows, contributing to effective benchmarking and evaluation frameworks.

Highest-signal resume keywords

LLM Benchmark DevelopmentPython ProgrammingStatistical JudgmentSQL ProficiencyExperience with RL Environments

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

BenchmarkingEvaluation FrameworksStatistical AnalysisReproducible CodeMulti-Step ReasoningFinancial JudgmentReward FunctionsVariance AnalysisConfidence IntervalsInter-Rater Agreement

Soft Skills

Attention to DetailAnalytical ThinkingDecision-Making

Tools & Technologies

SQLLm-evaluation-harnessHELMLarge Datasets

Industry Keywords

Finance WorkBenchmarking TeamPost-TrainingAgentic TasksExpense Policy Review

Tech Stack

Tools & technologies

PythonSQL

About the role

Key responsibilities & impact

Build benchmarks from real finance workflows, including expense policy review, invoice coding, month-end close, and spreadsheet Q&A
Turn agent traces, customer edge cases, and expert labels into versioned benchmark suites
Develop benchmarks for long-horizon agents operating across tools and systems, including tool use, multi-step reasoning, and financial judgment
Own the benchmarking effort and decide what to measure
Run benchmark suites across models and agent versions
Report benchmark results to inform team decisions
Design rubrics, programmatic checks, and LLM-as-judge graders
Calibrate graders against human agreement and identify where each approach breaks
Turn evaluations into reward functions and RL environments
Detect reward hacking and contamination before systems ship
Publish external benchmarks and write-ups on model performance in real finance work

Requirements

What you’ll need

Hands-on experience building LLM benchmarks or evals that people used to make real decisions
Strong Python and software engineering; reproducible and fast evaluation code
Statistical judgment covering variance, confidence intervals, sample size, inter-rater agreement, and distinguishing noise from meaningful deltas
Working knowledge of post-training, including SFT and RL with verifiable rewards
Comfort with SQL and large datasets
Taste for what is worth measuring
Industry experience on an eval, benchmarking or post-training team
Experience building RL environments or verifiers for agentic tasks
Published benchmark or evaluation research
Experience building or contributing to eval frameworks such as verifiers, lm-evaluation-harness, or HELM
Familiarity with accounting, finance, or other expert-domain work
Public work such as a benchmark, eval suite, or failure-analysis write-up
Willingness to work out of the NYC or San Francisco office 2–3 days per week

Benefits

Comp & perks

Flexible PTO
Centralized home-office equipment ordering
Health and wellness stipend
Budget for intra-office travel
Weekly coffee stipend
100% medical, dental & vision insurance coverage for you, with partial coverage for dependents
One Medical annual membership
401(k), including employer match on contributions made while employed by Ramp
Fertility HRA (up to $10,000 per year)
Parental leave: up to 16 weeks (birthing + bonding) or 8 weeks (bonding only) at 100% pay
Pet insurance
In-office perks: lunch, snacks, drinks, and more
Relocation expense coverage to NYC or SF (if needed)
Group medical, dental, and vision coverage through Sun Life
Life, AD&D, and disability coverage
Fertility drug coverage (up to $4,000 lifetime)
Group Retirement Plan with employer match (RRSP + DPSP)
Employee Assistance Program and virtual care through Lumino Health
Private medical insurance through Freedom Elite
Virtual GP and at-home care via eMed x Livi
Workplace pension through Penfold, with salary sacrifice option