Senior Software Engineer, AI Gateway

. Design and implement core systems powering the AI Gateway, including the model router, MCP gateway, and control plane .

Posted 10/15/2026full-timeRemote • United StatesSenior💰 $160,900 - $260,700 per yearWebsite

Tools & technologies

CloudDockerKubernetesMicroservicesRustGo

Key responsibilities & impact

Design and implement core systems powering the AI Gateway, including the model router, MCP gateway, and control plane
Build infrastructure for dynamic model selection and routing, auto-failover, cost-based routing, and policy enforcement
Own secure credential storage, session summarization, caching, and rate limiting
Develop APIs for OpenAI-compatible interfaces and the Model Context Protocol
Build infrastructure for evaluation, telemetry, replay, and backtesting for agents and LLM workflows
Lead architectural decisions and mentor engineers as the team scales
Tune performance through caching, session summarization, and rate limiting under production load
Contribute to roadmap planning, technical strategy, and cross-functional alignment
Build unified abstractions over model and tool providers
Improve credential and identity handling for tool access
Provide real-time and historical visibility into agent activity with enforceable policies
Maintain low latency and high reliability as traffic scales

What you’ll need

6+ years of backend engineering experience with production-grade systems
Deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design
Experience building secure, high-throughput services such as gateways, proxies, load balancers, or policy engines
Fluency in Go and/or Rust; both preferred
Familiarity with AI/ML platforms or model serving infrastructure
Strong product mindset focused on developer-facing tools
Ownership mentality with a bias for shipping, learning, and iterating
Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience
Prior experience with OpenAI, Anthropic, or similar LLM APIs
Familiarity with RAG architectures, vector databases, or agent frameworks such as LangChain, AutoGPT, or CrewAI
Experience with policy engines such as OPA, observability frameworks such as OpenTelemetry, or API gateways
Understanding of OAuth2.1, secret management, and cloud IAM systems
Experience with Kubernetes, Docker, and microservices architecture
Candidates must be legally authorized to work in the country where they are applying
Visa sponsorship considered case by case based on business needs

Comp & perks

Remote-first by design; work from home, with offices in Seattle and Paris for connection and collaboration
Flexible schedule
Generous PTO
Designated quarterly Whaleness Days
Designated end-of-year Whaleness break
Home office support
Technology stipend equivalent to US$100 net per month
Annual learning and development stipend for conferences, courses, certifications, and continued learning
16 weeks of paid parental leave after six months of employment
Equity for all full-time employees
Medical benefits
Retirement benefits
Paid holidays
Docker swag