Artificial Intelligence Quality Engineer, Full-time

Alliance Health

. Design and implement scalable AI Quality Engineering strategies for LLM solutions, agentic systems, and enterprise AI platforms .

Posted 10/14/2026full-timeMorrisville • North Carolina • United StatesMid-LevelSenior💰 $107,607 - $137,198 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in AI Quality Engineering, focusing on the design and implementation of scalable strategies for LLM solutions and enterprise AI platforms. Proficient in developing evaluation frameworks, automated testing workflows, and integrating quality controls within production environments.

Highest-signal resume keywords

AI Quality EngineeringPython ProficiencyLLM Evaluation FrameworksDistributed TracingAI/ML Model Deployment

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

AI Quality EngineeringPythonMachine LearningArtificial IntelligenceAutomated TestingEvaluation FrameworksModel OptimizationTelemetryVersion ControlRAG Architectures

Soft Skills

Analytical ThinkingCollaborationCommunication

Tools & Technologies

LangSmithGitHub CopilotObservability ToolsAI Assistants

Industry Keywords

Large Language ModelsAgentic SystemsQuality Scoring MethodologiesAgile DeliveryIncident Response

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Design and implement scalable AI Quality Engineering strategies for LLM solutions, agentic systems, and enterprise AI platforms
Develop and maintain AI evaluation frameworks, validation datasets, quality scoring methodologies, and automated testing workflows
Execute LLM-as-judge, code-based, and multi-turn evaluators on real production traces
Calibrate LLM judges against human expert preferences and clinical accuracy
Support model and prompt optimization and determine effective system prompts and model selections
Perform side-by-side performance analysis to assess agent component changes and prevent regressions
Trace agentic runs and conversations using distributed tracing and AI assistants to identify failure modes
Identify usage patterns and failure modes and provide executive-level reliability summaries
Partner with AI Engineering, AIOps, Security, and subject matter experts to build quality into architecture and validate agent outputs
Coordinate testing across agile delivery teams, including sprint planning, defect management, and release readiness
Integrate quality controls with observability tools, monitor model drift, and assist with incident response
Use telemetry and distributed tracing to maintain AI-enabled system availability and performance in production

Requirements

What you’ll need

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university
Three (3) years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions
Experience integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools within production environments
At least two (2) years of experience utilizing Python and related machine learning and artificial intelligence libraries and frameworks
Knowledge of AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators
Experience evaluating agentic workflows and Large Language Models using proprietary and open-source models
Experience using platforms like LangSmith for distributed tracing, monitoring agent runs, and analyzing production traces
Proficiency in Python and libraries used for automated testing and evaluation
Knowledge of RAG architectures and vector database evaluation
Knowledge of prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs
Ability to utilize AI tools such as GitHub Copilot
Ability to translate complex AI performance metrics and evaluation findings into actionable insights
Satisfactory background check and credit check required after offer acceptance and before start date

Benefits

Comp & perks

Medical, Dental, Vision, Life, Long Term Disability
Generous retirement savings plan
Flexible work schedules including hybrid/remote options
Paid time off including vacation, sick leave, holiday, management leave
Dress flexibility