Member of Technical Staff, Benchmarking

GenMD

. Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use .

Posted 11/3/2026full-timeRemote • United StatesLead💰 $150,000 - $210,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and maintaining clinical benchmarks and evaluation systems for healthcare data, with a strong focus on LLMs and data engineering. Proficient in collaborating with clinical experts to translate benchmark results into impactful research and products.

Highest-signal resume keywords

LLM EvaluationPython ProgrammingData EngineeringClinical Data ExperienceBenchmark Design

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

LLM TrainingEvaluation Pipeline DevelopmentSystematic Failure AnalysisAutomated GradersRubric CreationData AnalysisSQLBigQueryReward ModelingBiomedical Informatics

Tools & Technologies

PyTorchHugging FaceDashboardsReportsLongitudinal Patient Histories

Industry Keywords

Healthcare DataNLP PublicationsModel Performance GapsClinical BenchmarksMultimodal Records

Tech Stack

Tools & technologies

BigQueryPythonPyTorchSQL

About the role

Key responsibilities & impact

Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use
Build end-to-end evaluation pipelines, including runs, scoring, dashboards and reports comparing models and training runs at scale
Create rubrics, automated graders and LLM-as-a-judge systems, and calibrate them against clinician review
Run systematic failure analysis, categorize failure modes, measure their frequency and feed results into dataset design and post-training
Work with clinical experts and partner AI teams to turn benchmark results into research papers and products
Build benchmarks and evaluation systems for real-world healthcare data and use findings to create data that closes model performance gaps

Requirements

What you’ll need

Hands-on experience evaluating or training LLMs (PyTorch, Hugging Face or similar)
Strong Python and data engineering skills
Comfortable with large tabular and text datasets
SQL/BigQuery is a plus
Ability to design evaluations that measure meaningful outcomes and resist shortcuts
B.Sc. or higher in CS, ML, NLP, biomedical informatics or a related field
About 4 hours of daily overlap with Pacific Time (roughly 9 AM to 1 PM PT)
LLM/NLP publications preferred
Experience with clinical data or medical benchmarks is nice to have
Experience with RL from verifiable rewards, reward modeling or post-training data is nice to have

Benefits

Comp & perks

Equity
Well funded, revenue-generating company
Remote work arrangement