Python Engineer – AI Data & Evaluation

Gramian Consulting

. Design and refine evaluation rubrics and criteria for preference data .

Posted 9/21/2026contractRemote • Brazil, Argentina, Colombia, Chile, Uruguay, MexicoMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in Python or TypeScript for building and maintaining data pipelines, with a strong focus on evaluation methodologies and data quality. Proven ability to synthesize technical findings into clear documentation and collaborate effectively with cross-functional teams.

Highest-signal resume keywords

Python ProgrammingTypeScript DevelopmentData Pipeline InfrastructureEvaluation Criteria DesignTechnical Documentation

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data ReviewRubric CreationPreference Data CollectionEvaluation MethodologiesAI-Generated Content EvaluationQuality Judgment JustificationLarge Codebase NavigationMachine Learning ExperienceData Quality AssessmentTechnical Write-Ups

Soft Skills

Strong Communication SkillsCollaboration

Tools & Technologies

VSCode

Industry Keywords

Tier 1 CompaniesAI ProjectsEvaluation Projects

Tech Stack

Tools & technologies

PythonTypeScript

About the role

Key responsibilities & impact

Design and refine evaluation rubrics and criteria for preference data
Review and label data while identifying subtle quality issues and inconsistencies
Build and maintain Python or TypeScript pipelines and infrastructure supporting data generation, collection, and evaluation
Develop processes that translate qualitative assessments into scalable evaluation workflows
Synthesize findings from data reviews into clear technical write-ups, updates, and recommendations
Collaborate with researchers and engineers to improve evaluation methodologies and data quality
Iterate on evaluation processes based on observed issues and project requirements
Communicate technical findings in a structured, concise, and actionable format

Requirements

What you’ll need

At least 5 years of professional experience with Python or TypeScript
Hands-on experience building or maintaining pipelines and infrastructure using Python, or contributing to large TypeScript codebases
Practical experience with preference data collection, review, or rubric creation
Experience designing or applying evaluation criteria for AI-generated content or technical outputs
Strong written and verbal communication skills, demonstrated through clear technical documentation or summaries
Ability to explain and justify quality judgments using explicit evaluation criteria
Experience navigating or contributing to large codebases, such as VSCode, is an advantage
Prior experience working on AI, machine learning, data quality, or evaluation projects is an advantage
Mandatory experience working in Tier 1 companies