Member of Technical Staff, AI Evaluation

P-1 AI

. Study how engineers and customers use Archie in the field, identify where Archie succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations .

Posted 10/6/2026full-timeSan Mateo • California • United StatesLeadWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in evaluating AI systems within physical engineering domains, leveraging strong product judgment and collaboration with diverse stakeholders to enhance system capabilities and align evaluations with real-world engineering workflows.

Highest-signal resume keywords

Physical Engineering BackgroundAI Evaluation ExperienceEvaluation Methodology DevelopmentStrong Product JudgmentCustomer Collaboration

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Power SystemsPower ElectronicsMechanical EngineeringAerospace EngineeringRoboticsAgentic AI Systems UnderstandingEvaluation Task DesignAutomated Judges DevelopmentHuman Evaluation ProcessesFailure Mode Analysis

Soft Skills

CommunicationCollaborationProblem-SolvingAdaptabilityMotivation

Industry Keywords

Engineering WorkflowsAI SystemsEvaluation MethodologiesCustomer EngagementSubject Matter Expertise

About the role

Key responsibilities & impact

Study how engineers and customers use Archie in the field, identify where Archie succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations
Recreate real-world engineering tasks and failure modes in environments that can be repeatedly used by development teams
Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes
Assess whether automated judges align with expert human judgment and improve them when they do not
Work across engineering, product, customers, subject matter experts, and external partners to ensure evaluations reflect real engineering workflows rather than abstract benchmarks
Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users

Requirements

What you’ll need

Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related field
Strong enough understanding of agentic AI systems to design and reason about evaluations
Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment
Comfortable working directly with customers, engineers, subject matter experts, partners, and internal technical teams to extract what matters from ambiguous workflows
Strong product judgment, with the ability to turn observations about how a system is being used or failing into concrete evaluation work
Motivated by the mission of building superintelligence for engineering and excited by the challenge of making AI systems genuinely capable in the physical world

Benefits

Comp & perks

Competitive salary
Meaningful equity ownership
Healthcare
Dental
Vision
401(k) match
Unlimited PTO
Relocation support available