In my last blog post about measurement, I wrote about why an evaluation consultant may encourage you to use a validated measure in your project. In the conclusion to that piece, I promised that I would also explain when a tailor-made questionnaire would be more appropriate. To cut to the chase: custom measures will be your only option for certain types of constructs. They come with some limitations that an experienced evaluator can help to address.
What is a custom measure?
In contrast to validated measures, which are questionnaires created by psychometricians and then rigorously tested, custom measures are questionnaires that have been created specifically for a particular project and used right away. Custom measures can be rigorously tested, like validated measures, but this is usually done on a tight time constraint. For example, the project may allow only a few weeks to pilot and refine a key measure – hardly enough time for the extensive iterations in through which validated measures are tested.
Like validated measures, custom measures are composed of 1) administration instructions, 2) selected items, and 3) a medium for delivery. However, unlike a validated measure, the instructions, items, and medium may all be tailored for a specific program. If you begin with a validated measure and change any of these three elements, you now have a custom measure which can no longer be said to be validated.
Why not use a validated measure?
As I explained in a previous post, I recommend that evaluation consultants use validated measures whenever possible. The first step in creating data collection tools should always be a literature search for validated tools. However, there are times when validated measures can’t be used. Ask whether any of the following are true of your evaluation:
- Are you trying to measure a construct that no one has ever measured before?
- Are existing validated measures inappropriate for your participants?
- Are existing validated measures too long?
- Will existing validated measures work if only you could just “tweak” them?
If you answered positively to any of these questions you will need to develop a custom measure. Here are just a few real examples of times from my evaluations when clients needed a custom measure:
- Measuring an original construct: the client wished to measure participant’s comfort with using local public transit options in a particular county.
- Measuring a new population: the client wished to measure the “multimedia literacy” of students as young as 10 years old, when previously the youngest participants were 14 years old.
- Shortening a measure: the client wished to shorten a 60-item questionnaire about mental health to 10 items.
- Adapting a measure: the client wished to make a validated measure focusing on drug use into a measure focusing on both drug use and alcohol use.
As you might imagine when considering these examples, custom measures are frequently needed in real evaluation projects. Because creating custom measures is much more technically challenging than selecting a validated measure after a literature search, once you realize that custom measures are needed, assembling a capable evaluation team becomes much more important. In the next part of my series “Why Use Custom Measures?”, I will discuss several approaches to consider when developing a custom measure which require significant technical skills.
Read more of our latest blogs!