Staff Applied Scientist

Robots & Pencils

. Lead the design and build of an evaluation framework for an enterprise planner-based agent .

Posted 11/12/2026full-timeRemote • United StatesLead💰 $200,000 - $300,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in LLM and agent evaluation methods, experimental design, and statistical inference, with a strong ability to build production-quality research tooling and communicate technical findings effectively to diverse stakeholders.

Highest-signal resume keywords

LLM Evaluation MethodsExperimental DesignStatistical InferenceMulti-Objective OptimizationProduction-Quality Research Tooling

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Python ProgrammingStatistical TestingUser SimulationMulti-Step Agent ArchitecturesCost-Aware Machine LearningExperimental BenchmarkingScenario Schema DefinitionGrading SpecificationsData Pipeline DevelopmentPerformance Metrics Calculation

Soft Skills

CommunicationMentoringClient EngagementTechnical Translation

Tools & Technologies

LLM GatewayAPI-First DashboardContainerized ExecutionSandboxed ExecutionObservability Tools

Industry Keywords

Production Cost-Quality TradeoffRegulated Domain ExperienceOpen-Source ContributionsPublications

Tech Stack

Tools & technologies

PythonSwitching

About the role

Key responsibilities & impact

Lead the design and build of an evaluation framework for an enterprise planner-based agent
Benchmark planner models, executor LLMs and inference optimization levers with controlled, statistically sound experiments
Design the end-to-end evaluation workflow covering scenarios, user simulation, configuration, execution, grading, comparison and insights
Define scenario schemas, grading specifications and validators with the client team
Build an LLM-based user simulator for realistic multi-turn conversations and validate it against production logs
Build isolated, parallel experiment execution with configuration files and harness adapters using the client's LLM gateway
Package and implement optimization levers such as cascade-on-failure, cache-aware model switching and prompt caching
Tag model calls and send traces to attribute tokens, cache usage, latency and cost
Grade runs, compute success/cost/latency/failure metrics, and compare experiments using confidence intervals, paired methods and non-inferiority tests
Project dollar savings at production volume and build an evidence-linked automated insight engine
Build a locally hosted API-first dashboard with one-command deployment
Run the staged experiment campaign and analyze generalization by scenario category
Deliver final recommendations, runbooks and integration guides supporting go/no-go decisions
Partner with client engineers and leadership, communicate findings to technical and business stakeholders, write reports and mentor engineers

Requirements

What you’ll need

Experience with LLM and agent evaluation methods, including programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation
Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production
Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis
Experience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and projected dollar savings
Familiarity with LLM routing, model cascades, model selection and LLM inference internals and cost levers
Experience with observability and tracing for LLM systems, including attribution of tokens, cache usage, latency and cost per request
Ability to build production-quality research tooling using Python services, containerized and sandboxed execution, and data pipelines
Prior customer-facing or consulting experience scoping and delivering research engagements
Ability to explain technical findings to non-technical audiences and translate them into business impact
Helpful extras include publications/open-source contributions, dashboard or web UI experience, LLM gateway/tracing stack experience, regulated-domain experience, and production cost-quality tradeoff experience