Platform Reliability – Horizon AI

STN Incorporated

. Build the caching layer to prevent every agent tool call from directly hitting external systems .

Posted 10/21/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building reliable distributed systems with a focus on caching, event streaming, and observability. Proficient in Python and containerized environments, with experience in Kubernetes and deployment practices such as canary releases.

Highest-signal resume keywords

Distributed Systems ReliabilityPython ProgrammingKubernetes OrchestrationObservability MetricsEvent Streaming

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

CachingCircuit BreakingEvent StreamingObservabilityIdempotencyBackpressureTimeoutsHealth ChecksService-Level MetricsCross-Service Tracing

Soft Skills

JudgmentCollaboration

Tools & Technologies

KubernetesContainerized EnvironmentsInfrastructure-As-CodeCanary ReleasesTraffic Shifting

Industry Keywords

Durable Workflow OrchestrationBlue/Green DeploymentPerformance Profiling

Tech Stack

Tools & technologies

Distributed SystemsKubernetesPython

About the role

Key responsibilities & impact

Build the caching layer to prevent every agent tool call from directly hitting external systems
Implement event streaming so services are less dependent on synchronous HTTP chains
Harden agent dispatch with health checks, circuit breakers, retry semantics, and observability distinguishing failure from degradation
Build platform observability through service-level metrics, cross-service tracing, and actionable alerting
Contribute to orchestration migration and infrastructure-as-code
Improve deployment practices with canary releases, traffic shifting, and reliable rollback
Work with the Principal Engineer
Report to the Chief AI Officer, later the Senior Engineering Manager

Requirements

What you’ll need

Production experience making distributed systems reliable, including caching, queuing, circuit breaking, backpressure, timeouts, and idempotency
Comfort in Python and containerized environments
Real exposure to Kubernetes or equivalent orchestration
Experience with event streaming and its associated failure modes
Depth in observability, including metrics and distributed tracing
Judgment to alert on symptoms rather than causes
Record of shipping reliability improvements with measurable before-and-after results
Nice to have: durable workflow orchestration
Nice to have: blue/green and canary deployment at scale
Nice to have: performance profiling of Python services