AGIEval
Test models on questions from human admission and qualification exams.
Overview
AGIEval evaluates foundation models with questions taken from standardized human exams rather than purpose-built machine-learning datasets. The exams include college entrance tests, law school admission tests, lawyer qualification exams, and math competitions, in English and Chinese. The aim is to gauge general ability on tasks that people actually sit for.
What a task looks like
A task item is an exam question, usually multiple choice, with an optional reading passage, the question, answer options, and a label. The repository's format example is a Chinese math question about the intersection of two sets with four options; a couple of tasks (for example Gaokao-Math-Cloze and MATH) are fill-in-the-blank.
How it’s scored
Answers are checked against the key and reported as accuracy per task plus an average, using the repository's post-processing and evaluation script. No LLM judge is described in the sources.
- Released
- 2023-04
- Size
- 20 tasks in v1.1: 18 multiple-choice and 2 cloze. The sources fetched do not give a total question count in text.
- Versions
- Paper v1 (13 April 2023) and v2 (18 September 2023). The GitHub data is at v1.1, which added 2023 Gaokao chemistry, biology, and physics questions, fixed annotation problems, and made every multiple-choice task single-answer; the English subset is unchanged from v1.0.
- Access
- Public data in the GitHub repository, MIT licensed, with the note that data use should follow the licenses of the original source datasets.
Summarized by Stubbi from the public sources listed below. Stubbi has not run this benchmark.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Twenty tasks from exams such as the SAT, LSAT, Chinese college entrance exams, law exams, and math competitions, in English and Chinese.
Methodology
Score multiple-choice and fill-in answers; report accuracy by exam and overall.