FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building and running production LLM-based systems, with a strong focus on agent runtime, evaluation, and performance measurement. Capable of clear communication with both technical and non-technical stakeholders while ensuring data security and compliance.
Highest-signal resume keywords
LLM-Based Systems DevelopmentAgent Runtime EngineeringPerformance MeasurementGCP Production ServicesData Security Management
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Software EngineeringProduction EngineeringTesting and DebuggingAPIs DevelopmentMachine LearningEvaluation Data AnalysisContext DesignPrompt EngineeringVersion ManagementAgent Behavior Monitoring
Soft Skills
Clear CommunicationCollaboration with Operations TeamsMentoring Engineers
Tools & Technologies
GCPAI Graded EvaluationsTest HarnessesRegression TestsRetrieval-Augmented Generation (RAG)
Certifications & Qualifications
Bachelor's Degree in Computer ScienceBachelor's Degree in Engineering
Industry Keywords
HealthcareFinanceInsuranceSensitive Data HandlingGreenfield Development
Tech Stack
Tools & technologiesGoogle Cloud Platform
About the role
Key responsibilities & impact- Build and run production LLM-based agents and assistants
- Build agent runtime and orchestration, including tool calls, context handling, handoffs, failure handling, retries, timeouts, and sandboxing
- Build evaluation sets, test harnesses, regression tests, and AI-graded evaluations
- Work across multiple model providers, manage version upgrades, and plan provider fallbacks
- Trace agent behavior, monitor production performance, and attribute AI spend to agents and use cases
- Connect agents to company data using retrieval (RAG), context design, and prompt engineering
- Work with operations teams to determine which workflow steps agents can perform reliably and when to escalate to people
- Protect patient data (PHI), control agent access, and defend against prompt injection
- Build on GCP and partner with Data Platform teams
- Contribute to code and design reviews and support engineers newer to AI
- Explain AI trade-offs to product, operations, and clinical stakeholders
- Perform other duties as assigned
Requirements
What you’ll need- Bachelor's degree in Computer Science, Engineering, or a related technical field
- 5 to 15+ years of software engineering experience
- Hands-on experience building and running LLM-based systems in production that real users relied on
- Senior Staff candidates typically bring 2+ years of production LLM experience
- Hands-on depth in at least one core area: agent runtime, evaluation, retrieval, or observability and cost
- Staff and Senior Staff candidates need depth in agent runtime or evaluation
- Senior Staff candidates need a second deep area plus working knowledge across the rest
- Experience measuring AI system performance with evaluation data
- Strong production engineering across testing, debugging, services, APIs, deployment, and on-call
- Experience handling sensitive data with a security-first mindset
- Comfort working in a greenfield space
- Clear communication with technical and non-technical partners
- Healthcare, finance, or insurance experience preferred
- GCP production services experience preferred
- Experience migrating systems between model versions or providers preferred
- Machine learning experience beyond LLMs preferred
- Experience building internal platforms used by other engineering teams preferred
- For Senior Staff, experience leading a small team's technical direction and growing a single pod into multiple teams preferred
Benefits
Comp & perks- Medical, Dental, and Vision plans
- Flexible Spending/Health Savings Accounts
- Flexible PTO
- 401(k) + Company Match
- Life Insurance
- Pet insurance