Adverse impact in psychometric testing: what it is and how to monitor it

Adverse impact in psychometric testing occurs when a selection method produces noticeably different outcomes for different groups of candidates. It is not proof of discrimination, but it is a signal every employer using tests should be monitoring. Here is what it means legally, how to measure it, and what to do about it.

What adverse impact means

Adverse impact describes a situation where a selection procedure, such as an ability test, a situational judgement test or an interview, results in a substantially lower rate of progression for members of one group than another. The groups of interest are usually defined by protected characteristics such as sex, ethnicity, age or disability.

Adverse impact is about outcomes, not intent. A test can be carefully designed, administered identically to everyone and still produce different results across groups. Sometimes that reflects a problem with the test; sometimes it reflects real differences in the skills being measured, shaped by differences in education and opportunity. Either way, the difference has consequences for who gets hired, and employers are expected to know about it and be able to justify it.

The UK position: indirect discrimination and objective justification

In Great Britain, the relevant legal concept is indirect discrimination under section 19 of the Equality Act 2010. It arises when an employer applies a provision, criterion or practice to everyone, it puts people who share a protected characteristic at a particular disadvantage, and the employer cannot show that it is a proportionate means of achieving a legitimate aim.^[1] A selection test is a clear example of such a practice.

Two points follow. First, there is no fixed statistical threshold in UK law: whether a disadvantage is "particular" is a question of evidence in each case. Second, the defence is objective justification. An employer needs to show a genuine business aim, such as selecting people able to do the job, and that using the test in the way it is used is a proportionate way of achieving it. Evidence from job analysis and validation is central to that argument.

The Supreme Court's decision in Essop v Home Office illustrates how this works. Black and minority ethnic employees aged over 35 showed statistically that they were less likely to pass a core skills assessment required for promotion. The Court held that they did not have to explain why the test disadvantaged their group; they had to show that the test caused the disadvantage, and statistics could provide that evidence.^[2] That makes routine monitoring of test outcomes important for employers too, since it is often the first place a problem becomes visible.

The four-fifths rule: a rule of thumb, not a verdict

The most widely cited benchmark comes from the United States. The Uniform Guidelines on Employee Selection Procedures (1978) state that a selection rate for any race, sex or ethnic group which is less than four-fifths (80%) of the rate for the group with the highest rate will generally be regarded by US federal enforcement agencies as evidence of adverse impact.^[3] For example, if 50% of one group progress and 35% of another, the impact ratio is 0.70, below the four-fifths threshold.

The rule is useful because it is simple, and many organisations outside the US use it as an internal flag. But its limits are well recognised, including in the Guidelines themselves, which note that smaller differences can still amount to adverse impact where they are significant in statistical and practical terms, and that larger ones may not where numbers are small and the difference is not statistically significant.^[3] With small samples the ratio can swing widely on the outcome of one or two candidates. Morris and Lobsenz showed how significance tests and confidence intervals for the impact ratio can help, while also highlighting that such tests have low power with small groups.^[4] The sensible approach is to look at the ratio, a significance test and the actual numbers together, and to aggregate data over time where samples are small.

Monitoring by group

Monitoring starts with collecting equality data from candidates on a voluntary basis, kept separately from the selection decision. At each stage of the process, compare the proportion of each group who progress, and, for tests, compare average scores. Standardised score differences are informative even before any decision is made, because they show where impact is likely to arise. Large meta-analyses show that group differences on cognitive ability measures vary considerably with the type of test, the job and the applicant population, so local monitoring tells you more than general averages.^[5]

Monitor the whole process as well as individual tests. Impact can arise at sifting, interview or assessment centre stages, and a single test can look acceptable on its own while contributing to impact once combined with other stages. Where reasonable adjustments are made for disabled candidates, those cases should be recorded so they can be reviewed separately; we cover adjustments in our guide to reasonable adjustments in online testing.

Reducing adverse impact without losing validity

Ployhart and Holtz described the tension between maximising validity and minimising group differences as the diversity–validity dilemma, and reviewed sixteen strategies for addressing it.^[6] They range from using alternative predictors and measurement methods with smaller subgroup differences, assessing the full range of job-relevant attributes and statistically combining or weighting scores, to reducing unnecessary reading demands, removing items that function differently across groups, offering practice and orientation, and improving applicant perceptions. Their review found that strategies differ widely in effectiveness, that the strongest involve using predictors with smaller group differences or combining scores, and that no single strategy is fully effective on its own.^[6]

In practical terms, three steps make the biggest difference. Build a test mix that reflects the full range of what the job requires, informed by job analysis, rather than defaulting to the measure with the largest group differences. Avoid over-weighting any single test, particularly by using it as a hard gate early in the process; interpreting scores in bands alongside other evidence is more defensible than ranking candidates on small differences. And review content for job relevance and accessibility, choosing ability tests whose sub-tests match the role's demands and asking publishers what item-level checks for biased questions they have carried out.

Adverse impact does not mean a test should be abandoned. It means the employer should understand it, be able to justify the test's use, and design the overall process so that no single measure carries more weight than the evidence supports.