Gramian Consulting

Backend Engineer

• Design realistic and challenging coding tasks involving feature implementation and bug fixes.
• Create tasks that reflect authentic backend engineering workflows and practical software development challenges.
• Develop deterministic verifiers that accept reasonable valid solutions and reject incorrect implementations.
• Build and validate solutions using backend programming languages such as Java, Node.js, Go, Rust, TypeScript, Python, or C++.
• Develop tasks involving scalable services, APIs, debugging, performance optimization, and automated testing.
• Review code quality, maintainability, correctness, and adherence to technical requirements.
• Design evaluation criteria that distinguish robust solutions from superficially plausible implementations.
• Incorporate security considerations into tasks and verifiers where applicable.
• Document assumptions, expected behavior, edge cases, and validation requirements.
• Apply independent technical judgment throughout the task development and review process.

Software Engineer – Backend, AI Code Evaluation

Full-stack Engineer

• Analyze, debug, and resolve software bugs and security vulnerabilities across diverse codebases.
• Implement features and improve existing functionality using Python 3, Java, Rust, Go, C++, or TypeScript.
• Refactor legacy code to improve maintainability, security, and performance.
• Conduct codebase audits, penetration testing, and security assessments to identify and mitigate risks.
• Contribute to backend development and optimize applications for scalability and security.
• Apply secure software development practices and assess potential security risks.
• Document technical findings and recommend practical improvements.
• Provide real-world engineering and cybersecurity insights to improve AI training datasets.

Cybersecurity, Software Engineering Specialist – AI Projects

Artificial Intelligence

• Design complex scientific workflows across computational life sciences that require multiple analytical and computational steps.
• Develop realistic scientific datasets, input files, instructions, constraints, and expected deliverables.
• Create tasks requiring AI agents to inspect data, select appropriate methods, execute analyses, and troubleshoot intermediate results.
• Implement reproducible expert solutions and objectively verifiable ground truths.
• Design robust automated or semi-automated grading criteria for scientific tasks.
• Validate scientific assumptions, calculations, code, intermediate outputs, and final results.
• Ensure task dependencies, data, and computational resources can be reliably packaged in controlled environments.
• Develop workflows using Python, command-line tools, scientific libraries, and relevant domain-specific software.
• Ensure tasks assess scientific reasoning and execution rather than memorization.
• Maintain high task quality and throughput while incorporating reviewer feedback.

Computational Biology, AI Specialist

Research Analyst

• Design novel, challenging tasks in computational genomics and bioinformatics that test advanced AI agent capabilities.
• Develop realistic, multi-step workflows involving biological datasets such as FASTA/FASTQ, VCF, BAM/SAM, BED, TSV/CSV, expression data, and sequence annotations.
• Create tasks requiring scientific judgment, including identifying data artifacts, handling ambiguity, selecting appropriate methods, and revising assumptions based on intermediate results.
• Prepare task specifications, input datasets, expected output schemas, and objectively verifiable ground truths.
• Develop expert reference solutions and reproducible computational workflows using Python, command-line tools, and bioinformatics libraries.
• Validate scientific accuracy, solvability, and challenge level for advanced AI models.
• Design robust grading criteria distinguishing scientifically correct solutions from superficially plausible outputs.
• Ensure deliverables are reproducible, well documented, and suitable for client review.
• Incorporate reviewer feedback while maintaining high standards for quality and throughput.
• Communicate progress, blockers, and scientific or technical requirements to project leads and reviewers.

Bioinformatics Research Specialist, Python – AI

• Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified.
• Audit grading logic to identify brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties for valid alternative solutions.
• Investigate discrepancies between model performance, grader results, and expected outcomes.
• Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings rather than accepting them without verification.
• Document concise, evidence-backed findings and provide actionable, reproducible feedback.
• Flag uncertainty and verify that implemented revisions resolve previously identified issues.

Technical AI Evaluation Analyst – LATAM

• Design complex scientific workflows across computational life sciences requiring multiple analytical and computational steps.
• Develop realistic scientific datasets, input files, instructions, constraints, and expected deliverables.
• Create tasks requiring AI agents to inspect data, select appropriate methods, execute analyses, and troubleshoot intermediate results.
• Implement reproducible expert solutions and objectively verifiable ground truths.
• Design robust automated or semi-automated grading criteria for scientific tasks.
• Validate scientific assumptions, calculations, code, intermediate outputs, and final results.
• Ensure task dependencies, data, and computational resources can be reliably packaged in controlled environments.
• Develop workflows using Python, command-line tools, scientific libraries, and relevant domain-specific software.
• Ensure tasks assess scientific reasoning and execution rather than memorization.
• Maintain high task quality and throughput while incorporating reviewer feedback.

Computational Biology – AI Specialist

• Design novel, challenging tasks in computational genomics and bioinformatics that test advanced AI agent capabilities.
• Develop realistic, multi-step workflows involving FASTA/FASTQ, VCF, BAM/SAM, BED, TSV/CSV, expression data, and sequence annotations.
• Create tasks requiring scientific judgment, including identifying data artifacts, handling ambiguity, selecting appropriate methods, and revising assumptions based on intermediate results.
• Prepare task specifications, input datasets, expected output schemas, and objectively verifiable ground truths.
• Develop expert reference solutions and reproducible computational workflows using Python, command-line tools, and bioinformatics libraries.
• Validate scientific accuracy, solvability, and appropriate challenge level for advanced AI models.
• Design grading criteria that distinguish scientifically correct solutions from superficially plausible outputs.
• Ensure deliverables are reproducible, well documented, and suitable for client review.
• Incorporate reviewer feedback while maintaining quality and throughput.
• Communicate progress, blockers, and scientific or technical requirements to project leads and reviewers.

Bioinformatics Research Specialist, Python, AI

• Design novel, challenging tasks in computational genomics and bioinformatics that test advanced AI agent capabilities.
• Develop realistic, multi-step workflows involving FASTA/FASTQ, VCF, BAM/SAM, BED, TSV/CSV, expression data, and sequence annotations.
• Create tasks requiring scientific judgment, artifact identification, ambiguity handling, method selection, and revision of assumptions based on intermediate results.
• Prepare task specifications, input datasets, expected output schemas, and objectively verifiable ground truths.
• Develop expert reference solutions and reproducible workflows using Python, command-line tools, and bioinformatics libraries.
• Validate scientific accuracy, solvability, and challenge level for advanced AI models.
• Design grading criteria distinguishing scientifically correct solutions from superficially plausible outputs.
• Ensure deliverables are reproducible, well documented, and suitable for client review.
• Incorporate reviewer feedback while maintaining quality and throughput.
• Communicate progress, blockers, and scientific or technical requirements to project leads and reviewers.

• Design scientific coding tasks consisting of one main chemistry problem and at least 3 logically connected, progressively structured sub-problems.
• Develop verified golden solutions in Python with complete unit test coverage.
• Create discriminative test cases that distinguish correct from incorrect AI-generated outputs.
• Execute quality control validation through the Central Task Platform (CTP), including Tier 1 structural checks and Tier 2 quality rubrics.
• Iterate on task specifications, implementations, and test cases based on QC feedback.
• Optimize tasks to meet Pass@K evaluation criteria across multiple LLM judges, including GPT, Gemini, and Nemotron.
• Ensure scientific correctness, determinism, well-posedness, and adherence to project quality standards.
• Maintain high first-submission quality and minimize rework, targeting consistent L1 approval.
• Participate in review meetings, feedback sessions, and project standups during required overlap hours.

Scientific Chemistry, Python, AI Evaluation

• Author mathematical coding tasks containing one main problem and at least 3 logically connected sub-problems that progressively build toward the main solution
• Implement verified golden solutions in Python with complete unit test coverage
• Design discriminative test cases that distinguish correct from incorrect AI-generated outputs
• Perform quality control validation using the Central Task Platform (CTP), including Tier 1 structure checks and Tier 2 quality rubrics
• Refine problem specifications, solutions, and test cases based on QC feedback
• Optimize tasks to meet Pass@K evaluation criteria across multiple LLM judges, including GPT, Gemini, and Nemotron
• Ensure mathematical correctness, well-posedness, determinism, and compliance with project quality standards
• Maintain high first-submission quality and minimize rework, targeting consistent L1 approval
• Participate in review sessions, feedback meetings, and project standups during required overlap hours

AI Content Specialist, Mathematics, Python

Analyst

• Design scientific coding tasks with one main problem and at least 3 logically connected sub-problems
• Implement verified Python solutions with complete unit test coverage
• Create discriminative test cases that distinguish correct from incorrect AI-generated outputs
• Perform quality control checks using the Central Task Platform (CTP), including Tier 1 structure checks and Tier 2 quality rubrics
• Revise tasks and solutions based on QC feedback
• Optimize tasks against Pass@K evaluation criteria across multiple LLM judges, including GPT, Gemini, and Nemotron
• Validate scientific correctness, determinism, well-posedness, and expected outputs
• Maintain a low rework rate and high first-submission quality
• Participate in project reviews, feedback sessions, and standups during required overlap hours

Materials Science Analyst

Editor

• Source authentic photographs of real-world places, products, and media relevant to the target locale.
• Verify image licensing, copyright status, and provenance.
• Conduct multi-step visual research using Google Search, Google Lens, and reverse image search.
• Create cropped, masked, occluded, or degraded image versions that hide key identifiers while preserving realistic clues.
• Write clear, time-bounded prompts in English without revealing the answer.
• Create concise, objective grading criteria for each evaluation instance.
• Document the correct identity, research path, supporting evidence, and expected answer.
• Work with structured file-naming, shared Google Drive folders, and submission guidelines.
• Maintain consistency and quality across multiple evaluation tasks.

Photographer, Photo Editor

• Source authentic photographs of real-world places, products, and media relevant to the target locale
• Verify image licensing, copyright status, and provenance
• Conduct multi-step visual research using Google Search, Google Lens, and reverse image search
• Create cropped, masked, occluded, or degraded image versions that hide key identifiers while preserving realistic clues
• Write clear, time-bounded prompts in English without revealing the answer
• Create concise, objective grading criteria for each evaluation instance
• Document the correct identity, research path, supporting evidence, and expected answer
• Work with structured file-naming, shared Google Drive folders, and submission guidelines
• Maintain consistency and quality across multiple evaluation tasks