Lightly

AI Research Scientist

• Read and scan ML/AI research papers to understand their core contributions, methodology, experiments, and claims
• Review original human peer reviews to establish an expert baseline for each paper
• Evaluate AI-generated peer reviews against that baseline using a structured scoring rubric
• Assess the technical accuracy, analytical depth, constructive value, and novelty/significance assessment of each AI review
• Identify hallucinations, unsupported claims, missed technical issues, or valuable insights surfaced by AI reviewers
• Compare two AI-generated reviews side-by-side and determine where one provides stronger or more useful analysis
• Search and verify relevant academic literature using Google Scholar, arXiv, or Semantic Scholar
• Check whether cited prior work was available before the paper’s submission date
• Provide concise, evidence-based rationales explaining evaluation decisions and consistently apply the project rubric
• Evaluate whether agentic AI reviewers provide meaningful value beyond expert human reviewers

AI Research Peer Review Evaluator – ML/AI