Senior AI Systems Quality Engineer

Abacus Insights

. Build and ship production-grade automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle .

Posted 11/3/2026full-timeRemote • Massachusetts • United StatesSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in building and implementing automated validation frameworks and testing platforms for AI systems, with a strong focus on quality assurance, CI/CD integration, and measurable performance metrics. Proficient in Python and/or TypeScript, with a solid understanding of cloud-native architectures and AI testing methodologies.

Highest-signal resume keywords

AI Testing AutomationPython ProgrammingCI/CD IntegrationAWS Cloud-Native ArchitectureAI Testing Methodologies

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Automated Validation FrameworksTesting FrameworksLarge-Scale Test SuitesRegression FrameworksAI-Powered SystemsNon-Deterministic BehaviorQuality GatesEvaluation PipelinesScenario-Based TestingMeasurable Trust Thresholds

Tools & Technologies

DatabricksMLflowObservability ToolsAI Performance EvaluationPolicy Enforcement Frameworks

Certifications & Qualifications

Formal AI/ML Training or Certification

Industry Keywords

AI LifecycleQuality AssuranceGovernanceSafetyOperational Risk

Tech Stack

Tools & technologies

AWSCloudDistributed SystemsPythonTypeScript

About the role

Key responsibilities & impact

Build and ship production-grade automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle
Design and evolve an AI testing platform integrated with Databricks and MLflow
Create large-scale scenario-based test suites to validate agentic workflows, edge cases, long-tail scenarios, and failure modes
Validate orchestration behavior, including tool use, memory, and decision logic
Stress-test non-deterministic system behavior before production
Define system contracts, guardrails, and safe-degradation patterns
Define measurable LLM quality signals such as grounding, hallucinations, relevance, latency, and cost
Integrate automated quality gates into CI/CD pipelines
Ensure AI validation runs automatically on model, prompt, and code changes
Build reusable libraries and components for consistent AI quality practices
Own aspects of AI release readiness and define go/no-go criteria
Partner with AI, platform, security, and delivery teams to establish quality criteria, tradeoffs, and confidence levels

Requirements

What you’ll need

7+ years of software engineering experience, primarily in backend or platform systems
Proven experience designing and implementing AI testing automation in production environments
Demonstrated ability to build custom validation, evaluation, or testing frameworks for complex, distributed systems
Strong proficiency in Python and/or TypeScript within modern AI engineering stacks
Hands-on experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behavior
Experience designing or contributing to AI testing at scale, including regression frameworks, long-tail evaluation, and large test coverage
Deep understanding of CI/CD integration, including embedding automated tests and quality gates into deployment pipelines
Solid understanding of AWS cloud-native architectures
Track record of engineering for quality, reliability, governance, and safety as core system design principles
Working knowledge of security, privacy, and operational risk in regulated or mission-critical environments, including failure modes and recovery
Experience with AI testing methodologies, including evaluation of non-deterministic outputs, drift detection, bias/fairness testing, and robust regression strategies
Proven ability to establish measurable trust thresholds for AI systems, including query accuracy, hallucination limits, explainability, and PHI-safe behavior as enforceable release criteria
Experience working with domain experts to define correctness and real-world validation scenarios
Databricks-native environments and Medallion architecture are preferred, not required
MLflow, observability tools, LLM evaluation techniques, guardrails, policy enforcement frameworks, AI performance/latency/cost regression evaluation, documentation, formal AI/ML training or certification, prompt and agent behavior iteration, and AI-generated test scenarios are preferred, not required

Benefits

Comp & perks

Unlimited paid time off – recharge when you need it
Work from anywhere – flexibility to fit your life
Comprehensive health coverage – multiple plan options to choose from
Equity for every employee – share in our success
Growth-focused environment – your development matters here
Home office setup allowance – one-time support to get you started
Monthly cell phone allowance – stay connected with ease
Eligibility for performance bonuses
Equity grants