Can AI write psychometric test questions? Automatic item generation and its limits

Large language models can draft a plausible test question in seconds, and automatic item generation has been producing items by the thousand for over a decade. But a question that looks right is not the same as one that measures well. Here is what the research shows about AI-written test questions, and where the human work still lies.

Automatic item generation came before ChatGPT

The idea that computers can write psychometric test questions is older than the current wave of AI. Automatic item generation (AIG) became an established research field in the early 2010s, brought together in Gierl and Haladyna's 2013 edited volume, which framed it as the use of cognitive and psychometric theory to produce items with technology.^[1]

The classic method is template-based. Gierl and Lai describe three steps. First, subject-matter experts identify the content and the cognitive model: the knowledge and skills the item should require, and the features that make it easier or harder. Second, they build an item model, a template with a stem, options and the elements that can vary. Third, software fills the template under constraints, so that only sensible combinations are produced.^[2] Because this is combinatorial, a single well-built model can yield hundreds or even thousands of items.^[2] The approach works especially well for rule-based content, such as numerical, logical and medical-reasoning items, where the expert can specify exactly what changes and why.

The important point is where the intelligence sits. In template-based AIG, the expertise is in the cognitive and item models; the computer only does the assembly. Gierl and Lai also note that quality is typically established through field testing, which becomes hard to do when a model produces thousands of items, and that statistical precalibration is an active alternative.^[2]

What large language models add

Large language models change the picture because they can write fluent, varied text without a hand-built template. That makes them attractive for content that templates struggle with, such as personality statements and workplace scenarios.

Early results are genuinely encouraging. Hommel and colleagues fine-tuned transformer language models to generate personality items for specified constructs; around two-thirds of the generated items had acceptable psychometric properties, and about a third performed as well as established, carefully written items.^[3] Götz and colleagues built a GPT-2-based item generator and, in a preregistered validation with two samples, reported that short Big Five scales assembled from its items performed well against established measures.^[4] Li and colleagues extended the approach to personality situational judgement tests, using GPT-4 to draft workplace-style scenarios; a 40-item test covering five personality facets showed satisfactory reliability and validity for most facets in a sample of 443 people.^[5]

Where AI-written items fall short

The same studies also show the limits. In Hommel's work, a third of generated items did not hold up statistically.^[3] In Li's study, reliability was weak for some facets, one facet converged poorly with an established questionnaire, and the generated test predicted outcomes less well than the established measure.^[5]

Ability items show a similar pattern. Laupichler and colleagues compared exam questions written by a medical educator with questions generated by ChatGPT. Difficulty was similar, but the human-written questions discriminated significantly better between stronger and weaker candidates, with a mean discrimination index of .36 against .24.^[6] Discrimination is what makes an item useful in selection, so this is not a small difference.

Several other risks are specific to ability and reasoning tests. A language model can produce an item with no defensible correct answer, or with two. On True/False/Cannot tell items, for example, the line between "False" and "Cannot tell" depends on precise reasoning about what the passage does and does not establish, which is exactly where fluent text can mislead. Generated items may repeat surface patterns that test-wise candidates learn to spot, and they can carry cultural assumptions or reading-load differences that introduce unfairness. None of these show up by reading the item; they show up in data.

Why trialling and expert review remain essential

The Standards for Educational and Psychological Testing make no exception for how an item was written: test developers must document how items were developed, reviewed and trialled, and show evidence of reliability, validity and fairness for the intended use.^[7] An AI-drafted item enters the same pipeline as a human-drafted one.

In practice that means four stages. Expert review checks content, keying, clarity and sensitivity. Trialling on a representative sample produces item statistics: difficulty, discrimination, distractor behaviour and timing. Differential item functioning analysis checks that equally able people from different groups have the same chance of answering correctly. And items are then assembled into forms and the finished test is validated against job-relevant outcomes. The broader principles are covered in our article on what makes a psychometric test valid.

The balanced view

AI is a real productivity gain for test development. It can generate first drafts, suggest distractors, create parallel versions to reduce item exposure, and help developers explore a construct quickly. That matters at a time when large item banks are needed to keep online testing secure, a theme we discuss in can candidates cheat psychometric tests with AI?

What AI does not do is remove the need for evidence. The research to date supports using AI as a drafting tool within a disciplined development process, with expert review and trialling deciding which items survive. Employers buying tests should ask how items were developed and trialled, not whether a human or a model typed the first draft.