ML Platform Engineer – Evaluation Systems

Bask Health

. Design, build, and maintain machine learning evaluation infrastructure across Bask Health .

Posted 11/12/2026full-timeRemote • United StatesMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and maintaining machine learning evaluation infrastructure, including automated pipelines, job orchestration, and performance monitoring. Proficient in developing scalable systems for model evaluations and integrating with cloud-based resources while ensuring secure access and compliance.

Highest-signal resume keywords

Machine Learning Evaluation InfrastructurePython ProgrammingAPI DevelopmentCloud-Based InfrastructureDistributed Systems

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Machine Learning EvaluationJob OrchestrationAutomated PipelinesModel VersioningAsynchronous ProcessingRelational DatabasesLogging and MonitoringError RecoveryProduction Code QualityData Pipelines

Soft Skills

Analytical ThinkingProblem-SolvingCollaborationProject Ownership

Tools & Technologies

AWS LambdaDockerKubernetesMLflowWeights & BiasesAirflowDagsterTemporalSST

Industry Keywords

Healthcare Data PrivacyHIPAA ComplianceMachine Learning WorkflowsExperiment ReproducibilityModel Performance Monitoring

Tech Stack

Tools & technologies

AirflowAWSCloudDistributed SystemsDockerKubernetesPythonTypeScript

About the role

Key responsibilities & impact

Design, build, and maintain machine learning evaluation infrastructure across Bask Health
Develop platform services that execute and manage automated model evaluation workloads
Build scalable systems for large batches of model and AI application evaluations
Implement job orchestration, scheduling, and execution management for evaluation pipelines
Develop infrastructure for storing and versioning evaluation datasets
Build systems for tracking model configurations, prompts, parameters, and evaluation results
Implement reproducible experiment workflows for consistent model comparisons
Develop automated pipelines for benchmarking machine learning models
Build APIs and services for submitting and monitoring evaluation jobs
Create infrastructure for comparing evaluation results across model versions and experiments
Support evaluation workflows for large language models, predictive models, and retrieval systems
Develop asynchronous processing systems for computationally intensive evaluation tasks
Implement monitoring for job performance, failures, latency, and resource usage
Build logging and tracing capabilities for investigating evaluation problems
Develop systems for evaluation artifacts, reports, and historical results
Integrate evaluation infrastructure with model registries and ML development workflows
Implement automated quality checks for controlled model releases
Build retry, timeout, and failure recovery mechanisms
Optimize evaluation performance, infrastructure utilization, and operational costs
Support distributed evaluation workloads across cloud-based compute resources
Collaborate with ML engineers and applied scientists on infrastructure requirements and reproducible experiments
Develop tools simplifying evaluation configuration and execution
Implement secure access controls for evaluation datasets and results
Ensure tenant boundaries and approved data-handling practices are respected
Investigate infrastructure incidents and improve platform reliability
Document infrastructure, deployment procedures, and operational standards
Own ML evaluation platform projects from technical design through deployment, monitoring, and continuous improvement

Requirements

What you’ll need

3+ years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related technical field
Experience building and maintaining production backend services or infrastructure
Strong programming skills in Python, TypeScript, or comparable languages
Familiarity with machine learning development and evaluation workflows
Experience developing APIs and backend service integrations
Understanding of distributed systems and asynchronous processing
Experience working with cloud-based infrastructure and production deployments
Familiarity with automated testing and continuous integration practices
Understanding of machine learning model versioning and experiment reproducibility
Experience working with relational databases and structured datasets
Familiarity with data pipelines and batch processing workflows
Experience designing reliable background job systems
Understanding of infrastructure scalability, performance, and fault tolerance
Familiarity with logging, monitoring, and production observability
Experience implementing retries, timeouts, and error recovery
Understanding of authentication, authorization, and secure service communication
Familiarity with configuration management and deployment automation
Experience writing maintainable, well-tested production code
Ability to troubleshoot complex backend or infrastructure issues
Understanding of software release processes and operational reliability
Experience using Git and collaborative engineering workflows
Ability to translate machine learning requirements into scalable infrastructure
Strong analytical thinking and problem-solving skills
Ability to collaborate effectively with machine learning engineers, applied scientists, and software engineers
Ability to independently own engineering projects from design through deployment
Nice to have: Experience building infrastructure for machine learning evaluation or experimentation
Nice to have: Familiarity with MLflow, Weights & Biases, or comparable experiment tracking platforms
Nice to have: Experience with model registries and artifact management systems
Nice to have: Familiarity with large language model evaluation frameworks
Nice to have: Experience with LangSmith, DeepEval, Ragas, or comparable AI evaluation tools
Nice to have: Experience building distributed evaluation or benchmarking pipelines
Nice to have: Familiarity with workflow orchestration tools such as Airflow, Dagster, or Temporal
Nice to have: Experience with containerized workloads and distributed compute infrastructure
Nice to have: Familiarity with Docker, Kubernetes, or comparable technologies
Nice to have: Experience working with AWS Lambda, SQS, EventBridge, or related cloud services
Nice to have: Familiarity with AWS SageMaker or comparable ML platform services
Nice to have: Experience with serverless infrastructure and infrastructure-as-code tools
Nice to have: Familiarity with SST or comparable application deployment frameworks
Nice to have: Experience developing data versioning and dataset management systems
Nice to have: Understanding of reproducible machine learning experimentation
Nice to have: Experience implementing model performance monitoring and drift detection
Nice to have: Familiarity with large language model APIs and model comparison workflows
Nice to have: Experience building internal developer platforms for machine learning teams
Nice to have: Understanding of HIPAA and healthcare data privacy considerations
Nice to have: Experience using AI-assisted development tools such as Cursor or Claude Code