Item response theory explained: what it means for employers who use tests

Item response theory sits behind many modern online tests, including adaptive tests that give each candidate a different set of questions. You do not need the mathematics to use it well, but knowing what item response theory does, and what it cannot do, helps you ask suppliers the right questions.

Two ways of thinking about test scores

Classical test theory, the framework behind most psychometric tests for much of the last century, treats a test score as a true score plus error. It is practical and robust, and its statistics, such as an item's facility value and its item–total correlation, are easy to compute. But they have a known limitation: they depend on the sample. An item looks easy in an able group and hard in a less able one, and a candidate's score depends on which particular items they happened to take. That makes it awkward to compare people who sat different versions of a test.^[1]

Item response theory (IRT) approaches the problem from the level of the individual item. It models the probability that a person with a given level of the underlying trait, usually written as theta, will answer a particular item correctly. Person ability and item characteristics are placed on the same scale. When the model fits the data, item parameters do not depend on which sample was used to estimate them, and ability estimates do not depend on which items a person took.^[1,2] That property, often called invariance, is what makes much of modern online testing possible.

Item parameters in plain terms

Each item is described by a small number of parameters, summarised in an item characteristic curve showing how the probability of a correct answer rises with ability.^[2,3]

Difficulty (b) is the point on the ability scale where a candidate has an even chance of answering correctly; harder items sit further to the right.

Discrimination (a) is how steeply the curve rises around that point. A highly discriminating item separates people just below and just above its difficulty sharply.

Pseudo-guessing (c) is the lower asymptote: the chance that a very low-ability candidate still gets a multiple-choice item right, for example by guessing.

The one-parameter (Rasch) model uses difficulty only; the two-parameter model adds discrimination; the three-parameter model adds guessing. There are also models for rating-scale responses, widely used for personality and attitude measures.^[3,4]

Information, precision and adaptive testing

A practical benefit of IRT is the concept of information. Each item provides most information, and therefore most measurement precision, near its own difficulty level. Adding up item information gives a test information function, showing how precisely the test measures at each point on the ability range. Classical test theory, by contrast, usually reports one reliability figure and one standard error for everyone. With IRT, a developer can see that a test measures precisely around the middle of a graduate population, but less precisely at the extremes, and design forms to suit.^[1,2]

This is what makes computerised adaptive testing work. After each answer, the system updates its estimate of the candidate's ability and chooses the next item that will be most informative at that level. Able candidates are not slowed by easy items; less able candidates are not discouraged by a run of impossible ones. A well-designed adaptive test can reach a given level of precision with fewer items than a fixed test, and because candidates see different items, exposure of any single item is reduced.^[5] The same calibrated item bank also supports building many equivalent fixed forms.

The limits

IRT is not a cure-all, and its benefits depend on conditions that are not always met. Estimating item parameters accurately needs large trial samples; Hambleton and Jones suggested that more than 500 candidates are generally needed, with the exact figure depending on the model, compared with roughly 200 to 500 for classical item analysis.^[1] The standard models assume a single underlying trait, so a test that mixes, for example, reading load with numerical reasoning can violate that assumption. Invariance holds only when the model fits, so developers must check fit item by item. And the scoring is less transparent: under the two- and three-parameter models, two candidates with the same number of correct answers can receive different ability estimates, which can be hard to explain to candidates and line managers.^[1,3]

Above all, IRT improves the precision and efficiency of measurement; it does not create validity. A precisely measured construct that is irrelevant to the job is still irrelevant. The usual evidence of reliability, validity, fairness and relevant norm groups is still needed.

What to ask a supplier

If a supplier says a test uses IRT or is adaptive, reasonable questions include: which model was used, how large the calibration samples were, how model fit and unidimensionality were checked, how precise scores are across the ability range, how item exposure is controlled, and how scores are reported. Results from adaptive tests are best interpreted like any other test score: in bands, alongside other evidence about the candidate, and in the light of a job analysis, rather than as a single gate. For ability tests delivered online, IRT is often the most efficient route to secure, precise measurement, provided the groundwork has been done.