FREE ACCESS
5,000–10,000 jobs/day
See all jobs on Scoutfield
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in evaluating ML/AI research through critical analysis of methodology, experimental design, and scientific claims. Proficient in conducting literature searches and applying structured evaluation rubrics to assess AI-generated peer reviews.
Highest-signal resume keywords
Master's Or PhD In Machine LearningExperience In Peer Review For ML/AICritical Reading Of ML/AI Research PapersFamiliarity With Major ML/AI ConferencesStrong Analytical And Written Communication Skills
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Evaluation Of MethodologyAnalysis Of Experimental DesignAssessment Of Scientific ClaimsLiterature Search Using Google ScholarApplication Of Scoring RubricsIdentification Of Factual InaccuraciesComparison Of AI-Generated ReviewsTechnical Accuracy AssessmentNovelty Claims VerificationEvidence-Based Rationale Development
Soft Skills
Attention To DetailClear CommunicationAnalytical Thinking
Tools & Technologies
Google ScholarArXivSemantic Scholar
Industry Keywords
Machine LearningArtificial IntelligencePeer ReviewNeurIPSICMLICLRACLCVPRResearch MethodologyScientific Claims
About the role
Key responsibilities & impact- Read and scan ML/AI research papers to understand their core contributions, methodology, experiments, and claims
- Review original human peer reviews to establish an expert baseline for each paper
- Evaluate AI-generated peer reviews against that baseline using a structured scoring rubric
- Assess the technical accuracy, analytical depth, constructive value, and novelty/significance assessment of each AI review
- Identify hallucinations, unsupported claims, missed technical issues, or valuable insights surfaced by AI reviewers
- Compare two AI-generated reviews side-by-side and determine where one provides stronger or more useful analysis
- Search and verify relevant academic literature using Google Scholar, arXiv, or Semantic Scholar
- Check whether cited prior work was available before the paper’s submission date
- Provide concise, evidence-based rationales explaining evaluation decisions and consistently apply the project rubric
- Evaluate whether agentic AI reviewers provide meaningful value beyond expert human reviewers
Requirements
What you’ll need- Have a Master’s, PhD, or are currently pursuing graduate study in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical field
- Have contributed to at least one scientific/research paper, ideally as a first author, although co-authors and other substantial contributors are also welcome
- Have experience critically reading ML/AI research papers, including evaluating methodology, experimental design, results, limitations, and scientific claims
- Be familiar with major ML/AI research venues, such as NeurIPS, ICML, ICLR, ACL, CVPR, or comparable conferences and journals
- Have prior academic peer-review experience, ideally for an ML/AI conference or journal — strongly preferred
- Be comfortable conducting academic literature searches and verifying prior work, publication dates, citations, and novelty claims
- Have strong analytical and written communication skills and can distinguish meaningful technical concerns from superficial criticism
- Can provide clear, concise, evidence-based rationales for your decisions
- Can consistently apply detailed evaluation guidelines and scoring rubrics across multiple papers and reviews
- Have strong attention to detail, particularly when identifying factual inaccuracies or hallucinated technical claims
Benefits
Comp & perks- Fully remote and flexible — work from anywhere
- Part-time contractor role with flexible hours
- Work directly on the evaluation of cutting-edge agentic AI systems for scientific research
- Apply your ML/AI research expertise to help measure and improve the quality of AI-generated scientific peer review