Senior Software Engineer, ML API – AI Inference

Cerebras

. Design, implement, test, and operate features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging inference capabilities .

Posted 10/27/2026full-timeSunnyvale • California • United StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing production-level software services, particularly in the context of machine learning and inference APIs. Proficient in Python and familiar with various programming languages and frameworks, with a strong focus on collaboration, technical delivery, and mentorship.

Highest-signal resume keywords

Python ProgrammingProduction API DevelopmentDistributed Systems ExperienceContainer Orchestration (Kubernetes)Machine Learning Inference

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

C++ ProgrammingGo ProgrammingRust ProgrammingAPI ValidationError HandlingREST ProtocolGRPC ProtocolPerformance BenchmarkingDebugging WorkflowsModel Validation

Soft Skills

Strong CommunicationCollaboration SkillsMentorshipTechnical LeadershipProblem-Solving

Tools & Technologies

PyTorchHugging FaceKubernetesCI/CDLinuxCerebrasVLLMTritonTensorRT-LLMSDKs

Industry Keywords

Machine LearningInference APIsTokenizationChat TemplatesStreaming ProtocolsMulti-Model ServingQuantizationReduced-Precision InferenceOpenAI-Compatible InterfacesSelf-Service ML Workflows

Tech Stack

Tools & technologies

CloudDistributed SystemsGRPCKubernetesLinuxNode.jsPythonPyTorchRustC++Go

About the role

Key responsibilities & impact

Design, implement, test, and operate features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging inference capabilities
Write design documents, evaluate technical tradeoffs, define milestones, and coordinate implementation with engineers and partner teams
Implement consistent request and response semantics across GPU prefill, Cerebras decode, and other inference backends
Maintain compatibility with adopted inference interfaces and evolve APIs through validation, versioning, and deprecation
Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features
Extend inference services and integrations with vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components
Implement routing, state-transfer integration, error handling, retries, cancellation, and request lifecycle management for disaggregated inference
Improve time to first token, streaming latency, throughput, and resource efficiency
Build conformance tests, model-validation suites, numerical comparisons, integration tests, and performance benchmarks
Improve metrics, tracing, logging, alerts, and diagnostic tools; investigate incidents and resolve root causes
Improve configuration, documentation, examples, and debugging workflows
Conduct design and code reviews, mentor engineers, share domain knowledge, and remove delivery roadblocks
Collaborate with model enablement, compiler, runtime, cloud infrastructure, product, customer-facing teams, and customers

Requirements

What you’ll need

5+ years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components
Strong programming ability in Python and proficiency in Go, C++, Rust, or another language used to build production services
Experience designing and delivering features spanning multiple components
Experience building and evolving production APIs, including validation, error handling, compatibility, and testing
Familiarity with REST, gRPC, or streaming protocols
Experience integrating software across service, framework, runtime, or infrastructure boundaries
Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD
Ability to diagnose correctness, reliability, and performance issues across distributed components
Strong communication and collaboration skills, including leading technical delivery for a small project and supporting engineers through reviews or mentorship
Ability to turn ambiguous requirements into maintainable software
Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience
Preferred: production ML or LLM inference APIs, OpenAI-compatible interfaces, vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, ML systems, tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, constrained decoding, disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, inference request scheduling, API conformance and model-quality testing, multi-model or multi-tenant serving, Go/C++/Rust performance-sensitive services, GPU inference, AMD Instinct, ROCm, CUDA, collective communication, multi-node model serving, reduced-precision inference, quantization, SDKs, developer tools, configuration systems, self-service ML workflows, customers, or open-source communities

Benefits

Comp & perks

Build a breakthrough AI platform beyond the constraints of the GPU
Publish and open source cutting-edge AI research
Work on one of the fastest AI supercomputers in the world
Job stability with startup vitality
Simple, non-corporate work culture that respects individual beliefs
Equal and diverse work environment
Continuous learning, growth and support