Can we trust LLMs to judge whether a research idea is novel? huggingface.co

Ai2 researchers released Old Ideas, Novel Problems, a benchmark, data and code for testing whether LLMs can judge the novelty of research ideas. Across six models, small prompt changes sometimes shifted accuracy by more than 50 percentage points; retrieval and extra reasoning did not reliably help, and two specialized evaluators performed worse than a simpler Claude Opus 4.6 baseline.