Machine Learning Engineer, Evaluation, Reliability

Bask Health

. Build and maintain evaluation systems for AI-powered applications across Bask Health.

Posted 11/12/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing and maintaining evaluation systems for AI applications, with a strong focus on automated testing frameworks, model evaluation metrics, and collaboration with cross-functional teams. Proficient in analyzing model performance and implementing quality assurance measures to enhance AI reliability.

Highest-signal resume keywords

Machine Learning EngineeringAutomated Testing FrameworksPython ProgrammingModel Evaluation MetricsData Processing and Evaluation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Machine Learning EngineeringAutomated TestingPython ProgrammingTypeScript ProgrammingModel Evaluation MetricsData ProcessingRegression TestingAPI IntegrationCloud-Based DevelopmentProduction Code Quality

Soft Skills

Analytical ThinkingAttention to DetailTechnical CommunicationCollaborationProject Ownership

Tools & Technologies

GitCI/CD ToolsRelational DatabasesMonitoring ToolsDashboards

Industry Keywords

AI ApplicationsLarge Language ModelsGenerative AIEvaluation MethodologiesQuality Assurance

Tech Stack

Tools & technologies

CloudPythonTypeScript

About the role

Key responsibilities & impact

Build and maintain evaluation systems for AI-powered applications across Bask Health.
Develop automated testing frameworks for large language models and AI application workflows.
Create and maintain representative datasets for evaluating model behavior.
Implement benchmarks measuring response accuracy, relevance, completeness, and consistency.
Develop evaluation pipelines for prompts, retrieval systems, and model configurations.
Test AI assistants and agents against realistic user scenarios and workflow requirements.
Build automated regression tests to detect performance changes between releases.
Measure hallucination rates, unsupported claims, and other model quality failures.
Evaluate retrieval-augmented generation systems for document relevance and answer grounding.
Develop tests for structured model outputs and schema compliance.
Evaluate tool-calling accuracy and agent action selection.
Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
Implement evaluation checks for sensitive patient and operational information.
Build dashboards and reports communicating AI quality metrics to engineering teams.
Analyze evaluation results and identify recurring failure patterns.
Investigate production incidents involving inaccurate or unreliable AI outputs.
Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.
Support model comparisons and controlled experiments.
Evaluate tradeoffs between model quality, latency, reliability, and inference cost.
Develop automated quality checks for CI/CD and AI feature release workflows.
Implement monitoring and alerting for production AI performance.
Build tools for reviewing, labeling, and analyzing model responses.
Help maintain evaluation datasets as products and user behavior evolve.
Document evaluation methodologies, test cases, and known model limitations.
Work with product teams to define measurable acceptance criteria for AI features.
Collaborate with senior engineers on evaluation standards and reliability improvements.
Participate in code reviews, technical discussions, and engineering planning.
Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.

Requirements

What you’ll need

3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.
Experience building and maintaining production software or machine learning applications.
Strong programming skills in Python, TypeScript, or comparable languages.
Familiarity with large language models and modern generative AI applications.
Experience testing or evaluating machine learning models or AI-powered software.
Understanding of machine learning evaluation metrics and experimental methodologies.
Experience developing automated tests and validation pipelines.
Familiarity with prompt engineering and model output analysis.
Ability to analyze structured and unstructured datasets.
Experience writing scripts or services for data processing and evaluation.
Understanding of model accuracy, precision, recall, and other performance measures where applicable.
Familiarity with API integrations and backend application development.
Experience working with relational databases or analytical data systems.
Ability to identify patterns in model failures and translate findings into engineering improvements.
Understanding of regression testing and software quality assurance.
Familiarity with logging, monitoring, and production troubleshooting.
Experience using Git and collaborative software development workflows.
Understanding of the limitations and failure modes of large language models.
Ability to design representative test cases and evaluation scenarios.
Familiarity with cloud-based development and deployment environments.
Experience writing maintainable, well-tested production code.
Strong analytical thinking and attention to detail.
Ability to communicate technical evaluation results clearly.
Experience collaborating with engineers, product managers, and cross-functional teams.
Ability to independently own assigned technical projects.