Software Engineer L5 – AI Observability, Agent Evaluation
Netflix
Posted: 10/21/2026AWSAzureCloudDistributed SystemsGoogle Cloud PlatformGrafanaJavaMicroservicesPrometheus+3 more
Software Engineer
Booz Allen Hamilton
Posted: 10/21/2026CloudDrupalJavaJavaScriptNode.jsPHPPythonReactC+++1 more
Senior Developer, Euna Grants
Euna Solutions
Posted: 10/21/2026ASP.NETReactSCSSTypeScript.NET
Lead Subject Matter Expert – Electrical, Rolling Stock Engineering
DB E.C.O. North America
Posted: 10/21/2026
XR Developer
Marshall Economic Development Corporation
Posted: 10/21/2026AzureCloudJavaScriptUnityUnreal EngineC++
Software Engineer
Boeing
Posted: 10/21/2026AngularDockerJavaScriptKubernetesPythonTypeScript
Regional Electrical Project Engineering Specialist
Alcoa
Posted: 10/21/2026
Software Engineer
H&R Block
Posted: 10/21/2026AzureCloudDistributed SystemsJavaSQL.NET
Power Developer
Intersect Power
Posted: 10/21/2026Power Developer
Intersect Power
Posted: 10/21/2026
Audit Senior Associate, AEC – Arc, Engineering, Construction
Baker Tilly US
Posted: 10/21/2026Head of Agent Engineering
Baker Tilly US
Posted: 10/21/2026PythonTypeScript
Mainframe Developer – Cards
Regions Bank
Posted: 10/21/2026Cyber Security
Engineering Intern
Ascend Performance, LLC
Posted: 10/21/2026
Waste & Groundwater Programs Engineering Intern
Duke Energy Corporation
Posted: 10/21/2026
Software Engineer
Granicus
Posted: 10/21/2026
Senior Developer Marketing Manager – Lifecycle & Growth
1Password
Posted: 10/21/2026Open SourceSQL
Secure Facilities Technical Engineering – Mission Critical & Secure Facilities
AECOM
Posted: 10/21/2026
Lead Developer
Endava
Posted: 10/21/2026AWSCloudDockerGrafanaJavaScriptJenkinsJestKubernetesMicroservices+4 more
Software Engineer
Thomson Reuters
Posted: 10/21/2026AWSAzureCloudDistributed SystemsGoogle Cloud PlatformMicroservicesNoSQLPostgresPython+3 more
Manager II, Engineering – Kubernetes Platform
CloudDistributed SystemsKubernetes
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building observability frameworks and evaluation tools for ML/LLM systems, with strong coding skills in Python and experience in production environments. Capable of integrating vendor platforms and collaborating cross-functionally to enhance model performance and data quality.
Highest-signal resume keywords
Python ProgrammingML/LLM EvaluationObservability FrameworksVendor IntegrationDistributed Systems
Tech Stack
Tools & technologiesAbout the role
Key responsibilities & impact- Build observability framework and platform capabilities providing metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration
- Build primitives for teams to monitor model performance, data quality, drift, and degradation on their own systems
- Build and extend evaluation frameworks for LLM and agentic systems supporting response quality, grounding and hallucination, task success, tool-use and trajectory correctness, LLM-as-a-judge, and human-in-the-loop scoring
- Lead build-vs-buy evaluations for observability and evaluation tooling
- Own SDKs, connectors, and APIs integrating vendor platforms into a consistent interface
- Build reusable libraries, SDKs, and templates that make observability and evaluation the default for new systems
- Provide dashboarding, alerting, and SLO/SLI building blocks and out-of-the-box templates
- Partner with engineering, product, machine learning, and data teams to turn needs into reusable platform capabilities
- Enable ML/AI practitioners to monitor model quality in production, evaluate LLM and agentic systems, and adopt vendor tooling through a self-serve platform
Requirements
What you’ll need- Experience in software, AI/ML, or platform engineering, with hands-on time in production observability, monitoring, or ML/LLM evaluation
- Proven track record of designing standards, libraries, SDKs, or frameworks that other teams build on and adopt
- Strong coding skills in Python and at least one of Java, Go, or Scala, with experience building production services
- Practical experience operating ML models in production (online serving and/or batch), including drift, model performance, and data quality
- Hands-on experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, Jaeger/Tempo, or similar
- Solid understanding of distributed systems, microservices, and at least one major cloud platform (AWS, GCP, or Azure)
- Ability to work cross-functionally with ML, data, infra, and product teams, and to communicate clearly about system behavior, quality, and risk
- Experience with vendor integration and VPC deployment
- AI-Native Engineering Mindset who uses AI tools as a core part of their own workflow to accelerate design, development, testing, and code review
- Nice to have: Hands-on experience with ML/LLM observability and evaluation tools such as Arize, Braintrust, LangFuse, Weights & Biases, Galileo, Vertex AI Model Monitoring, or SageMaker Model Monitor
- Nice to have: Experience building or shipping LLM/GenAI applications and evaluating them, including prompt/result logging, evaluation metrics, LLM-as-a-judge, and human-in-the-loop review
- Nice to have: Experience evaluating agentic systems, including tool use, multi-step reasoning, and trajectory/task-success measurement
Benefits
Comp & perks- Health Plans
- Mental Health support
- 401(k) Retirement Plan with employer match
- Stock Option Program
- Disability Programs
- Health Savings and Flexible Spending Accounts
- Family-forming benefits
- Life and Serious Injury Benefits
- Paid leave of absence programs
- Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off
- Full-time salaried employees are immediately entitled to flexible time off
ATS Keywords
✓ Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Software EngineeringAI/ML EngineeringProduction ObservabilityMonitoringModel PerformanceData QualitySDK DevelopmentFramework DesignCloud PlatformsMicroservices
Soft Skills
Cross-Functional CollaborationClear Communication
Tools & Technologies
PrometheusGrafanaDatadogOpenTelemetryELKOpenSearchJaegerTempoWeights & BiasesSageMaker
Industry Keywords
ObservabilityModel DriftHuman-in-the-LoopAgentic SystemsLLM-as-a-Judge