AI Quality Assurance – Evaluation Specialist

Gramian Consulting

. Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.

Posted 9/24/2026contractRemote • Brazil, Argentina, Colombia, Chile, Peru, UruguayMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in technical QA, AI evaluation, and data analysis, with a strong focus on validating deliverables and ensuring quality through detailed documentation and evidence-based decision-making. Proficient in Python, SQL, and structured data interpretation to enhance evaluation reliability.

Highest-signal resume keywords

Technical QA ExperienceAI EvaluationData AnalysisPython ProficiencyStrong Analytical Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Technical QAAI EvaluationData AnalysisSoftware TestingPythonSQLShell ScriptsExecution LogsValidation of DeliverablesAttention to Detail

Soft Skills

Strong Written EnglishAnalytical SkillsIndependent InvestigationClear CommunicationEvidence-Based Decision Making

Industry Keywords

Grading LogicEvaluation CriteriaModel PerformanceTask DefectsGrader ErrorsQuality ControlActionable RecommendationsConcise DocumentationCollaborationDiscrepancy Investigation

Tech Stack

Tools & technologies

PythonSQL

About the role

Key responsibilities & impact

Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
Review AI agent execution traces, tool calls, and generated deliverables to assess whether outcomes are justified.
Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
Identify cases where valid alternative solutions are unfairly penalized.
Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
Independently assess automated QC findings rather than accepting them without verification.
Document concise, evidence-backed findings and actionable recommendations.
Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
Verify that task revisions resolve previously identified defects.
Collaborate with relevant teams to improve evaluation quality and reliability.

Requirements

What you’ll need

Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
Strong analytical skills, including the ability to validate calculations and reconcile conflicting evidence.
Experience assessing the correctness and completeness of technical or professional deliverables.
Strong written English and the ability to provide specific, reproducible feedback.
High attention to detail when reviewing task requirements, grading behavior, and model outputs.
Ability to independently investigate issues and make evidence-based decisions.