When working with an evaluator, you may have been encouraged to use a “validated” survey measure in place of a tailor-made questionnaire. At first, this might seem like a strange proposition: why use a measure that wasn’t specifically developed for your program or intervention? The short answer is that validated measures have special characteristics that tailored measures generally lack. And as for the long answer, we will unpack it in the next series of blogs.
What is a validated measure?
Validated measures are questionnaires that are created by psychometricians (statisticians who specialize in the measurement of attitudes and behaviors) and then rigorously tested using both qualitative and quantitative methods. For example, a questionnaire might be put through a series of focus groups, expert panels, statistical comparisons between groups, and models that test the performance of items using predetermined mathematical parameters. For a measure to be considered “validated” by psychometricians, a large amount of evidence is assembled and weighed in peer-reviewed publications (or made directly available to the public). One gold standard of validation is set out in the American Psychological Association’s Standards for Educational and Psychological Testing, but many psychometricians now strive to go beyond these standards as well.
At a basic level, validated measures are composed of three elements: 1) specific administration instructions, 2) carefully chosen items, and 3) a specific medium for delivery. For example, a validated measure will have precise directions for who should take the questionnaire and when they should take it. It will also include items (questions or tasks) that are worded in a way that makes their message clear and unmistakable. The medium for delivery of the questionnaire may be pen and paper, a static online questionnaire, or an interactive questionnaire that selects items based on answers to previous items.
Why are particular items important?
Many of our clients wonder about why particular items are chosen for a questionnaire and why they generally shouldn’t be changed. The first reason is that items that are included in a validated measure have been selected because they are well understood by members of the general population. If the measure is meant to be used with special populations, such as people undergoing mental health treatment, then items are selected because they are well understood by members of this special population. Second, the items are also selected because they work well together reliably to measure the same thing (e.g. stress) at different levels of severity. Third, the items are selected because they don’t unfairly bias the results in favor of any particular group of people – for example, it would be bad if a questionnaire systematically underestimated the level of anxiety of women. Validation studies test all the items to make sure that they aren’t biased in any way. When we gather the statistical evidence, it is very common to discover that items that do not appear biased actually are biased. In addition to these considerations, psychometricians select items for other statistical reasons, such as avoiding ceiling and floor effects, behaving uniformly for people along the entire range of the scale, and the ability to detect true differences between individuals.
What can a validated measure do?
Many validated measures have been so well refined that we can say with confidence that certain scores strongly indicate different levels of need or different mental health conditions. For example, scores from a validated measure can provide strong evidence for the claim that a person who scores above a certain threshold has severe, moderate, or mild mental health needs. These categories will have already been mapped onto scores by psychometricians and subject-matter experts. Another thing that a validated measure can do is to pick up on small changes – that is, it has high sensitivity. This makes validated measures much better suited for measuring change in pre/post studies. Most non-validated instruments cannot claim these special properties. Finally, using validated measures helps you to communicate your impact to funders and the public. That is, validated measures function as a standard that people can understand without going into all the details of your methodology.
After reading this post, you should have a better idea of the strengths of validated measures. In the next post, I’ll explain how an evaluator selects a validated measure, and how programs should implement it.