How a psychometric test is built: from job analysis to published test

A psychometric test that takes candidates twenty minutes can take a developer a year or more to build. Knowing how a psychometric test is built, and what evidence each stage should leave behind, is the quickest way for an employer to tell a well-founded instrument from a set of questions with a logo on top.

Start with the job, not the questions

Every defensible test begins by defining what it is meant to measure and why. For an occupational test, that means job analysis: interviews with job holders and managers, task inventories, critical incidents and a review of existing competency frameworks. The output is a blueprint, sometimes called a test specification, which sets out the construct, the content areas and their weighting, the item formats, the intended difficulty range, the time limit and the population the test is for. The Standards for Educational and Psychological Testing treat this specification as the foundation of test development, because every later decision, and every later claim about validity, traces back to it.^[1,2]

A blueprint also guards against a common failure: a test that drifts towards whatever is easiest to write questions about, rather than what the role actually demands.

Writing items and having experts review them

Item writers then draft considerably more questions than the final test will need, because many will not survive. Good practice for multiple-choice items is well documented: one clearly correct answer, plausible distractors, no clues in wording or option length, no trick negatives, and language no harder than the construct requires.^[3] For verbal reasoning items using the True, False and Cannot tell format, the passage must contain exactly the information needed to support the keyed answer, and nothing that lets general knowledge stand in for reasoning.

Draft items then go to independent reviewers. Subject-matter experts check accuracy and relevance to the blueprint; other reviewers look for cultural, gender or linguistic content that could disadvantage groups for reasons unrelated to the construct. Items are revised or discarded before any candidate sees them.^[2]

Trialling and item analysis

Surviving items are trialled on a sample that resembles the people who will eventually take the test. Trialling answers questions no reviewer can settle by reading: how hard each item really is, whether it separates stronger from weaker candidates, whether any distractor attracts the most able people, and whether the time limit is workable. Item statistics, either from classical test theory or from item response theory, decide which items make the cut and how they are assembled into forms. We cover these methods in our articles on item analysis and on item response theory.

Trial data are also used to check for differential item functioning: items that behave differently for groups of equal ability, which are a signal of potential bias and are usually rewritten or removed.^[1]

Reliability, validity and norms

Once the final form is fixed, the developer must show that scores are consistent (reliability) and that they mean what the test claims (validity). Reliability is typically reported as internal consistency, test–retest stability or, for multiple forms, alternate-forms correlation, together with the standard error of measurement. Validity evidence is accumulated rather than proved in a single study: content evidence from the blueprint, internal structure, relationships with other measures, and, ideally, relationships with job performance.^[1] Our article on what makes a psychometric test valid goes into these in more detail.

For a norm-referenced test, the developer then collects data from one or more clearly described comparison groups, so that a score can be expressed as a percentile, sten or band. The quality of those groups, including their size, recency and relevance, matters as much as the score itself.

The technical manual and independent review

All of this should be written down. A technical manual describes the purpose and construct, how items were developed and selected, the samples used, reliability and validity results, norm tables, subgroup differences and guidance on interpretation. The Standards make documentation of this kind an obligation of the developer, not an optional extra.^[1] The ITC International Guidelines for Test Use put a matching duty on users: to choose tests whose technical evidence fits the purpose and the people being assessed.^[4]

The final, and often skipped, stage is independent evaluation. In Europe, the EFPA Test Review Model gives reviewers a structured framework for describing and rating a test's documentation, materials, norms, reliability, validity, support for fair use and digitally generated reports.^[5] The current version, published in 2025, is a major revision of the 2013 model described by Evers and colleagues.^[5,6] In the UK, the British Psychological Society runs test reviews and its Registered Test scheme using the EFPA model, with each review checked independently by two reviewers and two editors.^[7]

What this means for employers

When you are choosing a test, ask for the blueprint or a description of how content was defined, the trial sample, the reliability and validity evidence, the norm groups and any independent review. A supplier who can answer these questions clearly has probably done the work. Then use the scores as the evidence was designed to support them: interpreted in bands, alongside interviews and other measures, never as a single gate. If no off-the-shelf instrument fits your roles, a custom-built assessment following the same stages is an option.