Machine Learning Engineering Lead

Cisco

. Set the multi-quarter strategy for measuring agentic AI quality .

Posted 10/24/2026full-timeMilpitas • California • United StatesSenior💰 $216,300 - $280,800 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing evaluation systems for machine learning and LLM products, with a strong focus on operational ownership and quality assurance. Proven ability to mentor teams, influence technical direction, and ensure trustworthy results through rigorous evaluation methodologies.

Highest-signal resume keywords

Machine Learning Production OwnershipPython ProgrammingDeep Learning Frameworks (PyTorch, TensorFlow)Evaluation System DesignLLM-as-Judge Calibration

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Machine LearningLLM-Based SystemsEvaluation SystemsProduction MonitoringSynthetic Data GenerationAutomated Evaluation IntegrationDrift DetectionAsynchronous ScoringTest Environment ReproducibilityRoot-Cause Analysis

Soft Skills

MentoringInfluencing Technical DirectionCollaboration

Tools & Technologies

CI/CD PipelinesAutomated ChecksHuman Expert CalibrationExperiment SnapshotsDataset Versioning

Industry Keywords

Agentic AI QualityMulti-Model EvaluationQuality AssuranceCost-Aware InferenceObservability

Tech Stack

Tools & technologies

PythonPyTorchTensorflow

About the role

Key responsibilities & impact

Set the multi-quarter strategy for measuring agentic AI quality
Own evaluation from architecture through production operation
Build layered evaluation systems using automated checks, calibrated LLM judges and expert human review
Extend evaluation systems to live production traffic and simulation
Design deterministic PR checks, pre-release model-based evaluation and human expert calibration
Build and calibrate multi-model LLM-as-judge panels
Run the human-in-the-loop flywheel with subject-matter experts and turn annotations into training signals
Score production traffic online, alert on regressions and drift, and convert traces into offline test cases
Lead simulation, synthetic data and adversarial testing
Close the loop from evaluation failure to issue clustering, candidate fixes and shadow/A/B validation
Ensure trustworthy results through experiment snapshots, dataset versioning, coverage reporting and cost tracking
Drive common standards, data contracts and judge criteria across partner teams
Mentor engineers, set evaluation engineering standards and represent the domain to senior leadership

Requirements

What you’ll need

BA/BS with 8+ years of relevant experience, MS with 6+ years, or PhD with 3+ years in Computer Science, Machine Learning or a related quantitative field
Experience taking machine learning or LLM-based systems into production and owning them operationally
Experience with Python and modern deep learning frameworks such as PyTorch or TensorFlow
Experience designing evaluation, measurement or quality systems for ML or LLM products
Experience setting technical direction across multiple teams through influence rather than reporting lines
Practical experience building and calibrating LLM-as-judge systems and validating them against human experts
Experience evaluating agentic systems, including multi-step reasoning, tool selection, trajectory quality and root-cause correctness
Experience with production monitoring at scale, asynchronous live-traffic scoring, drift detection and alerting
Experience building reproducible test environments or synthetic data generation
Experience integrating automated evaluation into CI/CD build pipelines and gating releases
Familiarity with networking, observability or infrastructure domains
Cost-aware inference, including routing, model selection and optimization
Publications at top ML venues, open-source contributions or public technical writing in the evaluation space

Benefits

Comp & perks

Medical, dental and vision insurance
401(k) plan with a Cisco matching contribution
Paid parental leave
Short- and long-term disability coverage
Basic life insurance
Potential Cisco restricted stock unit grants, vesting following continued employment
10 paid holidays per full calendar year
1 floating holiday for non-exempt employees
Paid birthday day off
Paid year-end holiday shutdown
4 paid days off for personal wellness
16 days of paid vacation per full calendar year for non-exempt employees
Flexible vacation time off program with no defined limit for eligible exempt employees
80 hours of sick time off provided on hire date and each January 1st thereafter
Up to 80 hours of unused sick time carried forward annually
Additional paid time away for critical or emergency family issues
Optional 10 paid volunteer days per full calendar year
Annual bonuses for non-sales roles, subject to Cisco policies
Performance-based incentive pay for sales-plan employees