Putting AI Evaluation to the Test: Why We Must Also Evaluate the Tests Themselves

How reliable are the methods we use to evaluate the quality of artificial intelligence? Two joint research papers by CAIRO and PleIAs address this question. Both were accepted for workshops at the international conference Empirical Methods in Natural Language Processing (EMNLP 2026) and presented in Budapest in October.

The first paper, Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks, examines how reliably benchmarks measure the commonsense reasoning abilities of AI models. Benchmarks such as HellaSwag and PIQA are widely used to compare models, but their questions can contain grammatical errors, typos, misleading wording, or even multiple correct answers. The researchers found that around 68% of model predictions remained unchanged even when questions were removed or replaced with meaningless text. This raises questions about whether some benchmark results genuinely reflect the abilities they are intended to measure. The paper introduces BenCheck, a tool for auditing multiple-choice benchmarks and identifying potential problems.

The second paper, RAG Components Are Not Independently Rankable, investigates the evaluation of retrieval-augmented generation (RAG) systems, which combine large language models with external information sources, such as technical documentation or internal company data. The study shows that the performance of individual components—such as document chunking, information retrieval, and reranking—depends on how they are combined. A component that performs well in one configuration may have a different effect in another. Evaluating components in isolation can therefore lead to misleading conclusions. The researchers argue that evaluation methods should account for interactions between components rather than simply ranking complete systems by their overall scores.

“Evaluating AI is not just about looking at a single metric or benchmark. We also need to examine what our tests actually measure and how reliable their results are,” says Pavel Chizhov, co-author of both papers and a speaker at the workshops.

This is particularly relevant for industry, where evaluation provides an important basis for investment and technology decisions. Companies need to determine which model is suitable for a specific use case and whether an AI system performs reliably under realistic conditions. A high benchmark score alone does not guarantee this. Carefully designed evaluation helps identify weaknesses early, compare systems more fairly, and direct development efforts more effectively. This is relevant to applications such as searching technical documentation, knowledge management, and internal AI assistants—areas of interest to companies in Franconia and the Würzburg–Schweinfurt region.

Both papers address a common challenge: how can we know whether our methods for evaluating AI are themselves reliable? As AI systems become increasingly integrated into real-world applications, it is essential to scrutinise not only the systems themselves but also the methods used to measure their quality.

Research papers

• Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks – LUHME Workshop, EMNLP 2026. See publication at ACL Anthology
• RAG Components Are Not Independently Rankable – DocInsights Workshop, EMNLP 2026. See publication at ACL Anthology

Pavel Chizhov is a co-author of both papers and a research associate at CAIRO. The research was carried out in collaboration between CAIRO and PleIAs.