Software Engineer, AI Quality – Evaluation

Bask Health

. Design, build, and maintain software infrastructure supporting AI quality and evaluation across Bask Health .

Posted 11/12/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in backend service development, API design, and automated testing frameworks, with a strong focus on AI evaluation and quality assurance. Proficient in implementing CI/CD workflows and managing production infrastructure while ensuring secure handling of evaluation data.

Highest-signal resume keywords

Backend Service DevelopmentAPI Design and IntegrationAutomated Testing FrameworksContinuous Integration and DeploymentAI Evaluation and Quality Assurance

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

TypeScriptJavaScriptPythonAPI DevelopmentAutomated TestingRelational DatabasesAsynchronous ProcessingLogging and MonitoringProduction DiagnosticsDistributed Systems

Soft Skills

CollaborationProblem-SolvingTechnical Design OwnershipEffective CommunicationProject Management

Tools & Technologies

GitAWS LambdaSQSEventBridgeOpenAI APIsLangSmithDeepEvalRagasPromptfooServerless Infrastructure

Industry Keywords

AI Quality EngineeringMachine Learning TestingHealthcare Data PrivacyMulti-Tenant SaaS ArchitectureEvaluation Tools

Tech Stack

Tools & technologies

AWSCloudDistributed SystemsJavaScriptNext.jsNode.jsPythonReactTypeScript

About the role

Key responsibilities & impact

Design, build, and maintain software infrastructure supporting AI quality and evaluation across Bask Health
Develop backend services that execute and manage automated AI evaluations
Build reusable testing tools for large language model applications
Create APIs for submitting evaluation jobs and retrieving results
Develop systems for evaluation datasets, test cases, and expected outcomes
Build automated workflows comparing prompts, models, and application configurations
Implement AI regression testing infrastructure and CI/CD evaluation checks
Record, organize, store, and compare evaluation results over time
Build tools, dashboards, interfaces, and libraries for reviewing AI outcomes and simplifying testing
Support deterministic checks, model-based evaluations, tool-calling behavior, structured outputs, and retrieval-augmented generation applications
Build asynchronous processing, scheduling, retries, timeouts, and error handling for evaluation workloads
Develop monitoring and logging; optimize speed, reliability, and infrastructure costs
Collaborate with AI evaluation engineers, machine learning engineers, and backend engineers to translate testing requirements into reusable tools
Implement access controls, secure handling of evaluation data, tenant boundaries, and privacy requirements
Investigate infrastructure failures and resolve reliability issues
Improve developer documentation and internal AI testing tools
Participate in code reviews, technical planning, and architecture discussions
Own AI quality engineering projects from design through deployment, monitoring, and continuous improvement

Requirements

What you’ll need

3+ years of professional software engineering experience, preferably in backend, platform, infrastructure, or developer tooling roles
Experience building and maintaining production backend services
Strong programming skills in TypeScript, JavaScript, Python, or comparable languages
Experience designing and developing APIs and service integrations
Familiarity with automated testing frameworks and software quality practices
Understanding of continuous integration and deployment workflows
Experience building reusable software components or internal engineering tools
Familiarity with large language models and modern AI application development
Understanding of the challenges involved in testing nondeterministic AI systems
Experience working with relational databases and structured application data
Familiarity with asynchronous processing and background job systems
Understanding of distributed systems fundamentals and service reliability
Experience developing automated testing pipelines
Familiarity with cloud-based application deployment
Understanding of authentication, authorization, and secure service communication
Experience implementing logging, monitoring, and production diagnostics
Ability to troubleshoot backend services and infrastructure issues
Familiarity with configuration management and software release processes
Experience writing maintainable, well-tested production code
Understanding of API error handling, retries, and failure recovery
Experience working with Git and collaborative engineering workflows
Ability to translate technical requirements into reusable engineering systems
Ability to collaborate effectively with software engineers, AI engineers, and machine learning specialists
Ability to independently own engineering projects from technical design through deployment
Nice to have: Experience building infrastructure for AI evaluation or machine learning testing
Nice to have: Familiarity with OpenAI, Anthropic, or comparable model APIs
Nice to have: Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools
Nice to have: Familiarity with LLM-as-a-judge evaluation approaches
Nice to have: Experience building evaluation execution services or automated benchmark systems
Nice to have: Familiarity with AI regression testing and prompt comparison workflows
Nice to have: Experience developing internal developer platforms or SDKs
Nice to have: Experience building testing infrastructure for distributed applications
Nice to have: Familiarity with TypeScript and Node.js backend development
Nice to have: Experience with React or Next.js for internal tooling interfaces
Nice to have: Familiarity with tRPC, Prisma, Drizzle, or comparable development tools
Nice to have: Experience working with AWS Lambda, SQS, EventBridge, or related cloud services
Nice to have: Familiarity with serverless infrastructure and infrastructure-as-code tools
Nice to have: Experience with SST or comparable application deployment frameworks
Nice to have: Experience building scalable asynchronous job processing systems
Nice to have: Familiarity with embeddings, vector databases, and retrieval-augmented generation
Nice to have: Experience implementing model observability or AI request tracing
Nice to have: Understanding of multi-tenant SaaS architecture and data isolation
Nice to have: Familiarity with HIPAA and healthcare data privacy considerations
Nice to have: Experience using AI-assisted development tools such as Cursor or Claude Code