Staff AI Engineer

Cobalt

. Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery .

Posted 10/26/2026full-timeRemote • United StatesLead💰 $200,000 - $240,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates extensive experience in software engineering with a focus on building and operating LLM-based agents, ensuring agent reliability, and establishing engineering standards. Capable of leading technical direction across teams while mentoring engineers and collaborating with cross-functional partners.

Highest-signal resume keywords

LLM-Based Agent DevelopmentTechnical LeadershipAgent Reliability ImprovementProduction ObservabilitySecurity Risk Management

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Software EngineeringAgent ArchitecturesBackend Systems FundamentalsDistributed SystemsAutomated Evaluation HarnessesRegression TestingIncident InvestigationConcurrencyFault ToleranceAPIs

Soft Skills

MentoringCollaborationTechnical Communication

Tools & Technologies

Managed Cloud Model PlatformsDirect Provider APIsDistributed TracingStructured LogsMetrics

Industry Keywords

Agent OrchestrationContext ManagementMemory ManagementEvaluation PracticesCost TrackingPrompt InjectionSensitive Data ExposureTool CallingStructured OutputsMulti-Provider Gateways

Tech Stack

Tools & technologies

CloudDistributed Systems

About the role

Key responsibilities & impact

Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery
Design, build and operate shared foundations for evaluations, observability, tracing, execution controls and model access with routing and failover
Establish evaluation practices for task completion, accuracy, false positives, security and cost per successful task
Build representative test cases with domain experts and turn production failures into regression tests
Improve resilience to tool failures, timeouts, rate limits and partial execution with limits on retries, runtime and spend
Build scoped permissions, customer data isolation, audit trails and safe tool/code execution with security and platform teams
Establish engineering standards for CI evaluations, prompt/model versioning, regression testing, cost and latency budgets, and staged rollouts
Lead model selection and cost optimization by benchmarking models against real workloads
Partner with teams on production readiness, incident investigation and agent reliability
Help engineering teams use AI coding agents across planning, implementation, review and testing
Mentor engineers and technical leads through design reviews, pairing, internal talks and written guidance
Create a technical roadmap with product, research and security leaders

Requirements

What you’ll need

Significant software engineering experience, typically 8–10+ years, with a track record of technical leadership across teams
Hands-on experience building and operating LLM-based agents in production
Track record of improving agent reliability through evaluations, non-deterministic failure investigation and measurable success-rate improvements
Strong knowledge of agent architectures, including tool use, orchestration, planning, context and memory management
Experience running LLM workloads through managed cloud model platforms or direct provider APIs, including rate limits, failure handling and cost tracking
Strong backend and distributed systems fundamentals, including APIs, asynchronous workflows, concurrency and fault tolerance
Experience with production observability, incident investigation and safe rollout of system changes
Understanding of security risks in agent systems, including prompt injection, unsafe tool execution and sensitive-data exposure
Evidence of impact beyond individual code, such as shared platforms, improved standards or mentoring engineers
Experience with agent orchestration, tool calling and structured outputs
Experience with Model Context Protocol (MCP), APIs and permission-controlled tool execution
Experience with automated evaluation harnesses, domain-specific test datasets and regression testing in CI
Experience with distributed tracing, structured logs and metrics for multi-step agent workflows
Experience with managed cloud model platforms, direct provider APIs and multi-provider gateways

Benefits

Comp & perks

Competitive compensation and an attractive equity plan
401(k) program (US) or pension (EU)
Medical, dental, vision and life insurance (US) or statutory healthcare (EU)
Wellness stipend
Work-from-home equipment and Wi-Fi stipend
Learning and development stipend
Flexible, generous paid time off
Paid parental leave
Ongoing mentorship opportunities