AI Evaluation Engineer

Bask Health

. Design and implement evaluation workflows for AI-powered applications across Bask Health.

Posted 11/12/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing and implementing automated tests for AI applications, with strong programming skills in Python and TypeScript. Proficient in evaluating AI-generated content and ensuring compliance with application standards and security protocols.

Highest-signal resume keywords

Automated Testing DevelopmentPython ProgrammingAI Evaluation MethodologiesAPI TestingRegression Testing

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Test AutomationSoftware Testing MethodologiesTest Case DesignDebuggingStructured Data ValidationContinuous IntegrationPrompt EngineeringEvaluation Dataset DevelopmentSchema Compliance TestingQuality Engineering

Soft Skills

Analytical ThinkingAttention to DetailProblem-SolvingCollaborationCommunication

Tools & Technologies

GitAWSOpenAI APIsLangSmithDeepEvalPlaywrightJestVitestPytestNode.js

Industry Keywords

AI-Powered ApplicationsLarge Language ModelsHealthcare Data PrivacyHIPAA ComplianceHuman Evaluation Workflows

Tech Stack

Tools & technologies

AWSCloudJavaScriptJestNext.jsNode.jsPythonReactTypeScript

About the role

Key responsibilities & impact

Design and implement evaluation workflows for AI-powered applications across Bask Health.
Build automated tests for large language model outputs and AI application behavior.
Develop evaluation datasets representing realistic user requests and product workflows.
Create test scenarios for patient-facing, merchant-facing, and internal AI features.
Implement checks for response accuracy, relevance, completeness, and consistency.
Evaluate AI-generated content for unsupported claims and misleading information.
Test structured model outputs for schema compliance and application compatibility.
Build evaluations for AI tool calling and authorized action execution.
Validate that AI assistants respect application permissions and tenant boundaries.
Test retrieval-augmented generation systems for grounding and information relevance.
Develop regression tests identifying unexpected changes in AI behavior.
Integrate evaluation checks into engineering development and release workflows.
Create automated comparison tests for prompts, model versions, and configuration changes.
Develop scripts and utilities simplifying AI evaluation execution.
Build reporting tools summarizing evaluation results and failure patterns.
Investigate failed evaluations and help engineers reproduce underlying issues.
Work with product managers to define expected AI behavior and acceptance criteria.
Collaborate with applied AI engineers to improve prompts and application logic.
Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
Develop evaluation cases for prompt injection and unauthorized tool-use attempts.
Support human review workflows for outputs requiring subjective assessment.
Track recurring AI failures and help prioritize reliability improvements.
Monitor production evaluation signals and investigate quality regressions.
Document test coverage, evaluation procedures, and known limitations.
Improve evaluation efficiency, repeatability, and maintainability.
Participate in code reviews, engineering planning, and technical discussions.
Support controlled testing and rollout of new AI-powered features.
Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.

Requirements

What you’ll need

3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
Experience developing automated tests for production software applications.
Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
Familiarity with large language models and modern AI-powered applications.
Understanding of software testing methodologies and test case design.
Experience building automated testing scripts, frameworks, or validation tools.
Familiarity with API testing and backend application integrations.
Understanding of prompt engineering and common AI model failure modes.
Ability to evaluate AI-generated responses against expected requirements.
Experience working with structured data and validating application outputs.
Familiarity with regression testing and software release processes.
Experience debugging application failures and reproducing technical issues.
Understanding of authentication, authorization, and secure application behavior.
Familiarity with databases and application data models.
Experience working with Git and collaborative software development workflows.
Understanding of continuous integration and automated testing pipelines.
Familiarity with logging, monitoring, and production troubleshooting.
Ability to create clear, reproducible test cases.
Strong analytical thinking and attention to detail.
Ability to distinguish application defects from model behavior limitations.
Experience documenting technical issues and communicating findings.
Familiarity with cloud-based application environments.
Ability to collaborate effectively with engineers, product managers, and other stakeholders.
Strong problem-solving skills and a quality-focused engineering mindset.
Ability to independently own evaluation and test automation projects.
Nice to Have: Experience evaluating large language models or generative AI applications.
Nice to Have: Familiarity with OpenAI, Anthropic, or comparable model APIs.
Nice to Have: Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
Nice to Have: Familiarity with LLM-as-a-judge evaluation techniques.
Nice to Have: Experience developing AI evaluation datasets and test benchmarks.
Nice to Have: Understanding of human evaluation and response labeling workflows.
Nice to Have: Experience testing retrieval-augmented generation systems.
Nice to Have: Familiarity with embeddings, semantic search, and vector databases.
Nice to Have: Experience evaluating AI agents and tool-calling behavior.
Nice to Have: Familiarity with prompt injection testing and AI security evaluations.
Nice to Have: Experience implementing AI regression tests in CI/CD pipelines.
Nice to Have: Experience building automated evaluation reports or dashboards.
Nice to Have: Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.
Nice to Have: Experience testing TypeScript or Node.js applications.
Nice to Have: Familiarity with React or Next.js application testing.
Nice to Have: Experience working with AWS or serverless infrastructure.
Nice to Have: Experience evaluating AI features in healthcare or other sensitive environments.
Nice to Have: Understanding of HIPAA and healthcare data privacy considerations.
Nice to Have: Experience developing human-in-the-loop quality review processes.
Nice to Have: Experience using AI-assisted development tools such as Cursor or Claude Code.