Agent Felet – Multi Agent Systems, Horizon AI

STN Incorporated

. Build the evaluation harness for agent behaviour, including scenario suites from real incidents, CI regression gating, and metrics that detect confidently wrong agents .

Posted 10/21/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and deploying LLM-based agentic systems, with a focus on evaluation harnesses, multi-agent orchestration, and operational runbook encoding. Strong systems engineering skills in Python are essential for effective task decomposition and governance.

Highest-signal resume keywords

LLM-Based Agentic SystemsMulti-Agent OrchestrationSystems Engineering FundamentalsTool and Function CallingEvaluation and Metrics Development

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonLLM TracingOnline EvaluationTask DecompositionContext ManagementOperational RunbooksScenario SuitesMetrics DevelopmentEscalation GovernanceAgent Architecture

Soft Skills

Strong Opinions About Evaluation

Industry Keywords

Autonomous SystemsHuman-in-the-Loop DesignOperational Data RetrievalSRE Background

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Build the evaluation harness for agent behaviour, including scenario suites from real incidents, CI regression gating, and metrics that detect confidently wrong agents
Define the autonomy SLO with the Chief Architect and own evidence for moving decisions from recommendation to action
Own agent architecture, including orchestration, tool dispatch, task decomposition, escalation, and read/write governance boundaries
Improve agent reasoning explanations so recommendations can be acted upon
Encode operational runbooks into agent-executable reasoning

Requirements

What you’ll need

Built and shipped LLM-based agentic systems in production, including malformed tool output, loops, stuck states, and context exhaustion
Strong and specific opinions about evaluation
Experience with tool and function calling at scale
Experience with LLM tracing and online evaluation
Strong systems engineering fundamentals in Python
Experience with multi-agent orchestration, handoff, and context management
Nice to have: operational or SRE background
Nice to have: human-in-the-loop system design
Nice to have: retrieval over operational data
Nice to have: experience defining what an autonomous system is allowed to do