Item analysis: how test developers decide which questions make the cut

Most questions written for a new psychometric test never reach a candidate. Item analysis is the set of statistical checks that decides which ones survive, and understanding it explains a great deal about why a well-built test behaves the way it does.

What item analysis is

Item analysis is the examination of how each question in a test performs when real people answer it. After items have been written and reviewed, they are trialled on a sample that resembles the intended candidates, and every response is recorded: which option each person chose, how long they took and how they scored on the test as a whole. From these data the developer works out, item by item, whether the question is doing its job.^[1,2]

The classical approach, rooted in classical test theory, rests on three questions. How hard is the item? Does it separate people who are stronger on the construct from those who are weaker? And are its wrong answers working as intended? Item response theory offers a more model-based route to similar questions, which we cover in a separate article; the classical statistics remain the first look for most developers.

Difficulty: the facility value

The simplest statistic is the facility value, often called the p-value: the proportion of the trial sample who answered correctly. An item answered correctly by 95% of people tells you little about anyone, because almost everyone gets it right; an item answered correctly by 10% may be very hard, badly worded or miskeyed.^[1]

There is no single ideal facility. The right spread depends on what the test is for. A test meant to discriminate across the whole range of a graduate population needs items spread across the difficulty range, with many in the middle. A test designed to identify basic competence, such as an error-checking task, may legitimately contain easier items. What matters is that the difficulty profile matches the blueprint and the population. Facility values are also sample-dependent: the same item will look easier in a more able group, which is one reason trial samples must resemble real candidates.^[1,2]

Discrimination: does the item separate strong from weak?

An item earns its place by distinguishing between people who are higher and lower on what the test measures. The usual index is the item–total correlation, often the point-biserial: the correlation between getting this item right and the total score on the test. Developers normally use the corrected version, which removes the item from the total, so the item is not being correlated partly with itself.^[1]

A strongly positive value means people who do well overall tend to get the item right. A value near zero means the item is behaving almost randomly with respect to the construct. A negative value is a warning sign: the more able candidates are getting it wrong, which usually points to a miskeyed answer, an ambiguity that sophisticated readers notice, or content that measures something else. A long-standing rule of thumb associated with Ebel, originally framed for the upper–lower group discrimination index and often applied to item–total correlations too, treats values of about .40 or above as very good, .30 to .39 as reasonably good, .20 to .29 as marginal and in need of improvement, and those below .20 as candidates for revision or removal.^[3] These are guides for judgement rather than mechanical rules: a short, very easy or very hard item will tend to show lower values even when it is sound.

Distractor analysis

For multiple-choice items, the incorrect options deserve as much scrutiny as the correct one. Distractor analysis looks at how often each wrong option is chosen and by whom. A good distractor attracts some candidates, mainly those with lower overall scores. A distractor no one chooses adds nothing but reading time; a distractor chosen more often by high scorers than low scorers suggests it may be defensibly correct, or that the keyed answer is flawed.^[4]

Research on the number of options has found that, for many items, three well-functioning options perform as well as four or five, because the extra distractors are often implausible and rarely chosen.^[5] For verbal reasoning items with True, False and Cannot tell options, distractor analysis is particularly informative: if strong candidates are splitting between False and Cannot tell, the passage may be implying more than the developer intended.

Why items are dropped, and what else is checked

Items are typically removed or rewritten for one or more of these reasons: they are far too easy or too hard for the intended population, they discriminate weakly or negatively, they have a non-functioning or attractive-to-the-able distractor, or they show differential item functioning, meaning groups of equal ability perform differently on them. Developers also look at how much each item contributes to reliability, how many candidates fail to reach it under time limits, and whether it fits the content balance of the blueprint.^[1,2]

Statistics never make the decision alone. A weak item covering an essential part of the blueprint may be rewritten and re-trialled rather than dropped; a statistically fine item that reviewers judge unfair is removed regardless. Item analysis is also not a one-off: items in live use, especially in online ability tests, should be monitored for drift and exposure over time. For an employer, the practical point is simple: ask a supplier how items were trialled and selected. The answer is a good proxy for how much care has gone into the scores you will rely on, and for how valid and reliable the test is likely to be.