AI/ML Evaluation Reviewer – Healthcare AI

mpathic

. Independently assess technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, proposed monitoring, and safeguards .

Posted 10/20/2026contractRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in evaluating and validating AI systems in healthcare, with a strong focus on model performance, statistical analysis, and methodological rigor. Capable of providing clear, evidence-based recommendations while collaborating effectively with multidisciplinary teams.

Highest-signal resume keywords

Machine Learning EvaluationStatistical AnalysisClinical AI Systems AssessmentAnalytical Writing SkillsModel Validation Methodologies

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Model Performance EvaluationStatistical Uncertainty AnalysisRobustness TestingAdversarial EvaluationHuman-in-the-Loop SystemsData Quality AssessmentPerformance Metrics SelectionChange-Management PracticesGenerative AI EvaluationClinical Informatics

Soft Skills

Analytical ThinkingIndependent WorkCollaborationCommunication Skills

Tools & Technologies

TRIPOD+AI FrameworkSTARD-AI FrameworkCONSORT-AI FrameworkDECIDE-AI FrameworkModel Cards

Industry Keywords

Healthcare AI ApplicationsPredictive ModelsClinical Decision SupportDigital HealthFDA Guidance for AI Devices

About the role

Key responsibilities & impact

Independently assess technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, proposed monitoring, and safeguards
Evaluate model performance and validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty
Assess data quality and representativeness, including data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings
Evaluate technical reliability, robustness, reproducibility, known failure modes, and performance under atypical or out-of-distribution inputs
Review proposed evaluation plans, performance benchmarks, and success criteria for pilot deployment
Assess technical safeguards, escalation mechanisms, and human-in-the-loop processes
Evaluate monitoring and change-management plans, including performance degradation detection, model behavior monitoring, updates, intervention, and rollback criteria
Collaborate with clinicians and other technical experts on cross-disciplinary concerns
Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework
Provide clear, evidence-based recommendations and identify additional evidence or changes needed
Participate in reviewer calibration and quality assurance activities
Maintain independence and confidentiality while following conflict-of-interest, confidentiality, and data-handling requirements

Requirements

What you’ll need

At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems
Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error
Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset
Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models
Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions
Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences
Ability to work independently while contributing to multidisciplinary expert review and calibration processes
Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements
Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments
Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards
Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards
Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis
Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices
Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments
Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device
Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations
Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment

Benefits

Comp & perks

Contract, per-review opportunity
Reviewer orientation to the evaluation framework and project-specific requirements prior to the first assignment