Differential item functioning: how test developers find biased questions

Differential item functioning is the main statistical tool test developers use to find questions that may be biased against a group of candidates. It works by asking a precise question: do people of the same ability, from different groups, have the same chance of getting this item right?

Group differences are not the same as biased questions

Suppose women and men, or two ethnic groups, have different average scores on a reasoning test. That difference alone does not show the test is biased. It might reflect a real difference in the skill being measured, shaped by education, experience or opportunity, or it might reflect something about the test that disadvantages one group for reasons unrelated to the skill. Looking at average differences cannot tell these apart.

Differential item functioning (DIF) analysis addresses this by comparing like with like. An item shows DIF when candidates who are matched on the ability the test measures, but who belong to different groups, have different probabilities of answering it correctly.^[1,2] Psychometricians distinguish this from impact, which is simply a difference in performance between groups that may well reflect genuine differences in ability. The distinction matters because removing every item on which groups differ would also remove items that measure the skill accurately.

DIF analysis is one part of the wider fairness picture, alongside the reliability, validity and fairness evidence described in what makes a psychometric test valid. Outcome monitoring looks at the test as a whole; DIF looks at the individual questions that make it up.

The Mantel–Haenszel procedure

The most established DIF method is the Mantel–Haenszel procedure, introduced to test development by Holland and Thayer.^[3] Candidates are divided into score levels using their total score on the test. At each level, the developer compares the odds of answering the item correctly for the reference group (often the majority group) and the focal group. The results are pooled across levels into a single common odds ratio, which can be converted into an effect size and tested for statistical significance.

Its appeal is practical: it is straightforward to compute, works with moderate samples and gives an interpretable index. Large testing organisations classify items into bands of DIF severity. At ETS, for instance, items are graded A (little or no DIF), B (small to moderate) or C (the largest differences), with C items used in test assembly only when essential to the specifications, and then only after documented justification and review by people not involved in writing them.^[4] A known refinement is to run a preliminary analysis and remove flagged items from the matching score, so that a biased item does not contaminate the yardstick used to judge the others.^[3]

The main limitation is that the standard Mantel–Haenszel test is designed to detect uniform DIF, where an item is consistently harder for one group across the ability range. It is less sensitive to non-uniform DIF, where an item favours one group at lower ability levels and the other at higher levels.

Logistic regression and IRT-based methods

Logistic regression offers a flexible alternative. Swaminathan and Rogers showed how the probability of a correct response can be modelled from the matching score, group membership and the interaction between them: a group effect indicates uniform DIF, and an interaction indicates non-uniform DIF.^[5] Zumbo later set out logistic regression as a unifying framework for DIF in both right-or-wrong items and rating-scale (ordinal) items.^[6]

Item response theory (IRT) methods take a model-based approach. Each item's characteristics, such as its difficulty and how sharply it discriminates between higher and lower ability, are estimated separately for each group on a common scale. If an item's parameters differ significantly between groups once ability is accounted for, it shows DIF.^[2] IRT methods are powerful and fit naturally with modern test construction, but they need larger samples and careful checking that the model fits.

Whatever the method, effect size deserves as much attention as statistical significance. With large samples, trivial differences become significant; with small ones, meaningful DIF can be missed. Sensible practice uses agreed effect-size thresholds, checks results across more than one method where possible, and treats any flag as the start of an investigation rather than a verdict.

Statistics flag, people decide

A DIF statistic does not prove an item is biased. It shows that matched groups perform differently, but the cause still has to be understood. The difference may come from something irrelevant to the skill being measured, such as an unfamiliar cultural reference, a needlessly complex word in a numerical item or a context more familiar to one group. Or it may reflect content that is genuinely part of the job requirement. Whether a flagged item is unfair therefore depends on whether the source of the difference is relevant to what the test is meant to measure.^[4]

This is why statistical analysis is paired with expert review. Before items are trialled, trained reviewers examine them for content that is offensive, stereotyping or likely to disadvantage a group for reasons unrelated to the construct. After trialling, items flagged for DIF are reviewed again, ideally by people independent of the original writers, who judge whether the cause is construct-irrelevant. Items judged unfair are revised or removed.^[1,4]

What employers should ask for

For organisations choosing tests, DIF analysis is one of the clearest indicators that a publisher has taken fairness seriously at the design stage. It is reasonable to ask which groups were analysed, which method was used, how large the samples were, what thresholds were applied and what happened to flagged items. A test can show no meaningful DIF and still produce group differences in overall scores, which is why item-level evidence complements, rather than replaces, monitoring outcomes in your own selection process. The same checks belong in any custom test development project from the outset.