Norm groups: why the comparison group matters as much as the score

A percentile is not a property of a person. It is a statement about a comparison — and the comparison group is doing more of the work than most report readers realise.

What a percentile actually claims

"78th percentile" means: this candidate scored higher than 78% of the people in the comparison group. It says nothing in isolation. Change the comparison group and the same raw score can produce almost any percentile you like.

A graduate scoring at the 78th percentile against a general population norm might sit at the 45th against a graduate norm, and lower still against a norm of successful applicants to the same scheme. All three numbers are correct. They answer different questions.

Which is why the first thing to look at on a report is not the score but the norm group it was scored against. A report that does not name its norm group is not giving you enough information to interpret it.

Representativeness beats size

Suppliers quote norm group size because it is impressive and easy to measure. It matters far less than composition.

A norm of 500 people drawn from the population you actually hire from is more useful than 50,000 assembled from wherever data happened to accumulate. Ask what the group consists of: what roles, what sectors, what education levels, what countries, and how they were recruited.

Ask particularly whether the norm is applicants or incumbents. Applicant norms describe people who applied; incumbent norms describe people already doing the job, and those groups differ systematically because someone selected the second group. Comparing your applicants to incumbents flatters nobody and misleads everybody.

And ask about demographic composition. A norm that under-represents groups in your applicant pool will produce percentiles that mean something different for those candidates.

Norms go stale

Test scores drift. Populations change, education changes, familiarity with test formats changes, and a norm collected fifteen years ago describes a population that no longer exists.

Drift is not always in the obvious direction, and it is rarely dramatic year on year — which is what makes it easy to ignore. It accumulates. A norm nobody has revisited in a decade may be shifting every candidate's percentile by a meaningful amount without anyone noticing.

Ask when the norm was collected, not when the test was published. Ask whether the supplier monitors for drift, and what they do when they find it. Continuous norm updating is a sign of a maintained instrument; a norm dated to the original publication is a sign of one that is not.

Local norms are often better

For high-volume hiring, your own data eventually beats any published norm. A norm built from your own applicants for a specific role compares candidates to the population you actually see, which is the comparison that matters for your decision.

Local norms need enough data to be stable — a few hundred candidates before they are worth using, more before they are worth trusting — and they need refreshing as your applicant pool changes.

Until then, use the published norm that most closely matches your population, and be explicit in reports about which one was used. The number on the page is only as meaningful as the group behind it, and the group behind it should never be a mystery.