FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Senior Software Engineer, AI Evaluation Infrastructure
Riva Scientific. Build and operate production backend systems .
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in building and operating production backend systems, with strong proficiency in Python and experience in LLM evaluation. Capable of making pragmatic design decisions and effectively collaborating with frontend technologies like TypeScript and React.
Highest-signal resume keywords
Production Backend Systems DevelopmentStrong Python ProgrammingDjango Framework ExperiencePostgreSQL Database ManagementLLM Evaluation Infrastructure
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
PythonDjangoPostgreSQLCeleryLLM EvaluationBenchmark Task ConstructionAgent Trial ScoringResult AggregationSystem Design DecisionsTechnical Writing
Soft Skills
OwnershipCollaborationProblem-SolvingAdaptabilityClear Communication
Tools & Technologies
HarborTerminal-BenchSWE-benchInspect
Industry Keywords
Production SystemsTask Quality AssessmentUser-Facing Edge CasesDesign Rationale Documentation
Tech Stack
Tools & technologiesDjangoPostgresPythonReactTypeScript
About the role
Key responsibilities & impact- Build and operate production backend systems
- Own services from design through on-call
- Develop and maintain AI/LLM evaluation infrastructure
- Build benchmark tasks, score agent trials, and aggregate and compare results
- Assess task quality, verifiers, answer leakage, and test validity
- Make pragmatic system design decisions and extend existing systems where appropriate
- Implement migrations, tests, and handling for user-facing edge cases
- Collaborate with TypeScript and React frontend changes when needed
- Document design rationale in pull requests and design notes
Requirements
What you’ll need- Typically 5+ years building and operating production backend systems
- Ownership of at least one service from design through on-call
- Strong Python
- Experience with Django or a similar framework
- Experience with PostgreSQL
- Experience with a task queue such as Celery
- Hands-on experience in LLM evaluation beyond calling a model API
- Understanding of benchmark task construction, agent trial scoring, and result aggregation/comparison
- Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar harness is a strong plus
- Ability to identify ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason
- Ability to make pragmatic design tradeoffs and explain what was not built
- Ability to take ownership of ambiguous requests and complete work including migrations, tests, and edge cases
- Ability to quickly adopt team conventions
- Comfort with a TypeScript and React frontend when changes are visible there
- Clear technical writing in pull request descriptions and design notes
Benefits
Comp & perks- Remote work
- Full-time employment