Senior QA Engineer – AI

Delta Exchange

. Build and maintain golden datasets for AI products, sourced from production traces or generated and verified against a documented source of truth .

Posted 9/25/2026full-timeRemote • IndiaSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in quality assurance for AI products, with a focus on building and maintaining evaluation frameworks, analyzing production traces, and defining quality metrics. Proficient in Python and experienced in API testing and observability tooling.

Highest-signal resume keywords

Quality Assurance For AI ProductsEvaluation Harness DevelopmentPython ProgrammingAPI TestingObservability Tooling

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Quality AssuranceEvaluation FrameworksAPI TestingPythonData AnalysisAutomationModel Behaviour AnalysisFailure Mode IdentificationRegression CoverageQuality Metrics Definition

Soft Skills

Clear Written Communication

Tools & Technologies

DeepEvalRAGASPromptfooOpikLangfuseLangSmith

Industry Keywords

AI ProductsTrading PlatformsFuturesOptionsCrypto DerivativesMarginSettlement Flows

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Build and maintain golden datasets for AI products, sourced from production traces or generated and verified against a documented source of truth
Design layered scoring for AI outputs using deterministic rules and model-graded evaluation
Manage evaluation runs end to end, including scheduling, execution, baseline comparison, and failure triage
Own and operate the release gate for prompt or model changes
Identify failure modes including hallucination, incorrect tool selection, loss of conversational context, and PII exposure
Investigate production traces to distinguish retrieval, generation, and tool failures
Convert findings into actionable engineering evidence and permanent regression coverage
Define and report quality metrics for AI surfaces and drive improvement

Requirements

What you’ll need

At least 1 year owning quality for AI products in a lead or primary-owner capacity
Direct experience with prompts, tool and function schemas, and model behaviour
Hands-on experience building or operating an evaluation harness for an AI system, using frameworks such as DeepEval, RAGAS or Promptfoo, or an in-house solution
4-6 years of QA / SDET experience
Working proficiency in Python
Ability to analyse traces and tool calls using observability tooling such as Opik, Langfuse or LangSmith, or an equivalent
Strong API testing experience
Sufficient engineering ability to build tooling and automation
Clear written communication
Experience with trading or exchange platforms, including futures and options, crypto derivatives, margin, or settlement flows is a bonus

Benefits

Comp & perks

Competitive compensation
Significant growth opportunities
Leadership exposure
Work directly with founders and senior leadership on high-impact strategic initiatives
Opportunity to shape the regulatory landscape for digital assets and Web3 in India
Mission-driven, fast-growing organisation at the forefront of financial innovation