Senior Machine Learning Engineer, Evaluation, Reliability

Bask Health

. Design and build evaluation infrastructure for Bask Health’s AI and machine learning applications .

Posted 11/12/2026full-timeRemote • United StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in developing and evaluating AI and machine learning systems, with a strong focus on automated testing frameworks, performance measurement, and production monitoring. Proficient in Python programming and familiar with large language models and healthcare-related AI workflows.

Highest-signal resume keywords

Machine Learning EngineeringAutomated Testing FrameworksPython ProgrammingAI Evaluation MethodologiesProduction Monitoring Systems

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Machine Learning EngineeringAutomated Testing FrameworksPython ProgrammingAI Evaluation MethodologiesBenchmark DatasetsData PipelinesRegression TestingModel EvaluationCloud InfrastructureMonitoring and Logging

Soft Skills

Strong Analytical SkillsAttention to DetailStrong Communication SkillsCollaboration with StakeholdersMentoring Engineers

Tools & Technologies

AWSMLflowWeights & BiasesLangSmithDeepEvalRagasCursorClaude CodeObservability InfrastructureDistributed Processing

Industry Keywords

Healthcare ApplicationsHIPAAHuman-in-the-Loop EvaluationRetrieval-Augmented GenerationModel Drift Detection

Tech Stack

Tools & technologies

AWSCloudPython

About the role

Key responsibilities & impact

Design and build evaluation infrastructure for Bask Health’s AI and machine learning applications
Develop automated testing systems for LLM applications and intelligent workflows
Establish measurable quality standards and evaluation pipelines assessing accuracy, relevance, completeness, consistency, and reliability
Build and maintain representative benchmark datasets and regression testing systems
Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems
Measure retrieval quality, grounding accuracy, and unsupported model outputs
Build tools to identify hallucinations, incorrect reasoning, and unreliable responses
Develop testing approaches for AI agents, tool calling, and multi-step workflows
Implement automated checks for structured outputs, API interactions, and tool execution
Build production monitoring systems, dashboards, and reporting tools for model quality, latency, errors, and operational reliability
Investigate production AI failures and identify underlying technical causes
Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations
Establish testing procedures for model upgrades and provider changes
Develop benchmarking tools comparing models by quality, performance, and cost
Implement evaluation processes for sensitive healthcare-related AI workflows
Work with product and clinical stakeholders to define acceptance criteria and escalation requirements
Support human review and feedback collection for AI outputs requiring additional validation
Design reproducible experiments and build reusable evaluation libraries, services, and internal tools
Integrate AI evaluation checks into development and deployment workflows
Improve observability through structured logging, tracing, and production diagnostics
Mentor engineers on AI testing methodologies and reliability practices
Contribute to AI infrastructure, testing, and monitoring architecture discussions
Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance

Requirements

What you’ll need

5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline
Demonstrated experience developing and operating machine learning or AI systems in production
Strong programming skills in Python and experience building production software services
Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications
Strong understanding of machine learning evaluation methodologies and performance measurement
Experience developing reproducible experiments and benchmark datasets
Familiarity with large language models and modern LLM application architectures
Understanding of prompt engineering, model inference, and AI application integration patterns
Experience evaluating model outputs for quality, accuracy, and reliability
Familiarity with retrieval-augmented generation and retrieval evaluation methods
Experience with structured datasets, data pipelines, and analytical workflows
Strong understanding of software testing, regression testing, and continuous integration
Experience building or maintaining backend services and APIs
Familiarity with cloud infrastructure and production deployment environments
Experience with monitoring, logging, tracing, and production troubleshooting
Understanding of model latency, throughput, and operational performance metrics
Ability to investigate complex system failures and identify root causes
Experience designing metrics and reporting systems for technical performance
Familiarity with version control, code review, and collaborative software development
Understanding of data quality and its impact on model evaluation
Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost
Strong analytical skills and attention to detail
Experience collaborating with engineers, product managers, and technical stakeholders
Ability to independently own complex engineering initiatives
Strong communication skills and the ability to explain evaluation results clearly
Nice-to-have qualifications include experience with LLM evaluation frameworks, LangSmith, DeepEval, Ragas, LLM-as-a-judge methods, hallucination detection, agent evaluation, tool-calling evaluation, retrieval metrics, vector databases, embeddings, semantic search, MLflow, Weights & Biases, observability infrastructure, model drift detection, CI/CD evaluation checks, distributed processing, AWS, AI developer tools, healthcare applications, HIPAA, human-in-the-loop evaluation, Cursor, and Claude Code