Senior Data Scientist, Applied AI

Definitive Healthcare

. Independently build, test, and improve LLM, RAG, and tool-using agent workflows for an assigned use case using maintainable, tested Python code .

Posted 10/27/2026full-timeRemote • Massachusetts • United StatesSenior💰 $136,000 - $206,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and evaluating LLM applications using Python, with a strong focus on analytical problem-solving and performance optimization. Capable of effectively communicating technical findings and collaborating with diverse stakeholders.

Highest-signal resume keywords

Python ProgrammingLLM Application DevelopmentDatabricks ExperienceAnalytical Problem-SolvingEvaluation Design

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Machine LearningSoftware EngineeringTestingEmbeddingsSemantic SearchRetrieval SystemsRAG ArchitecturesBenchmarkingRegression ChecksPerformance Monitoring

Soft Skills

Clear CommunicationEffective CollaborationIndependent Operation

Tools & Technologies

DatabricksDelta LakeUnity CatalogLakebaseLakeflow JobsMLflow

Industry Keywords

Data ScienceApplied AIHealthcare DataAI Evaluation PlatformsAgent Frameworks

Tech Stack

Tools & technologies

PythonUnity

About the role

Key responsibilities & impact

Independently build, test, and improve LLM, RAG, and tool-using agent workflows for an assigned use case using maintainable, tested Python code
Select and apply evaluation criteria, baselines, and test datasets within the team's established evaluation approach
Compare models, embeddings, and retrieval methods using evidence on quality, reliability, latency, and cost; document recommendations and tradeoffs
Investigate retrieval misses, incorrect answers, and workflow failures; determine root causes and implement and validate fixes
Build and update test cases, benchmark datasets, and regression checks
Monitor and investigate performance changes for owned workflows and recommend and implement improvements
Communicate findings and technical tradeoffs to engineering, product, and domain partners

Requirements

What you’ll need

Bachelor's degree required; master's degree in a quantitative or technical field preferred
5+ years of relevant experience in data science, machine learning, applied AI, software engineering, or a related field
Hands-on experience building and evaluating LLM applications
Strong Python programming, software engineering and testing, and analytical problem-solving skills
Strong knowledge of LLMs, embeddings, semantic search, retrieval systems, and RAG architectures
Strong hands-on experience with Databricks, including Delta Lake, Unity Catalog, Lakebase, Lakeflow Jobs, and MLflow, for building and deploying scalable, production-grade data and AI solutions
Ability to design representative evaluations for an assigned use case, analyze errors, and use results to improve a system
Ability to weigh quality, reliability, latency, and cost when recommending an approach for their own project
Clear communication and effective collaboration with technical and nontechnical partners
Experience with agent frameworks and AI evaluation platforms
Experience with healthcare data is preferred, but not required
Individual contributor role with no people-management responsibilities
Ability to operate independently on an assigned project within the team's established evaluation framework and practices
Ability to escalate decisions involving changes to the team's evaluation approach, significant business risk, sensitive data, material cost commitments, or production release approval

Benefits

Comp & perks

Company bonus or commission plan eligibility, depending on the position
Medical coverage
Dental coverage
Vision coverage
Unlimited paid time off
401(k) plan with employer contribution
Industry leading products
Flexible and dynamic culture
Definitive Cares community service group
Competitive benefits package
401(k) match