How should programs implement a custom measure?
If you are implementing a custom measure as part of a program evaluation or needs assessment, there are several things to keep in mind.
- Make time for a pilot phase for all custom measures. It is always possible that the measure will not work as expected for your participants and you will need time to refine it or start over. Sometimes, the first month, quarter, or year of the program serves as the “pilot phase” for the measure. There is nothing wrong with this in principle, provided you conduct the necessary analysis of the measure’s performance at the end of the pilot phase and adapt accordingly.
- As with validated measures, follow the administration instructions carefully. However, since custom measures are designed for your specific program and can be changed, keep an eye out for ways that the instructions could be improved to be more appropriate or understandable for your participants. Communicate any ideas you have about ways to improve the instructions to the evaluator so that changes can be made in a systematic way.
- As with validated measures, report any irregularities in implementation to the evaluator. Irregularities in implementation may or may not pose a problem, but they can also be a sign that the measurement process could be improved.
- Take careful note of organic feedback about the measure from participants. Since you are working with a custom measure, this feedback may suggest opportunities for improving the measure. Conversely, know that not every morsel of feedback provides a solid basis for changing the measure. For example, sometimes problems with understanding an item are specific to a particular participant and don’t affect the responses of other participants.
- Prepare translations of custom measures in advance of data collection and use the same translation every time. Don’t translate measures “on the fly” while implementing them, since once translator’s interpretation of the measure may differ from that of another translator. Listen for feedback from participants about each different translation of the measure, since issues can be specific to certain translations.
- Don’t take the scores for custom measures at face value. It is tempting to apply a simple scoring or weighting procedure to the results of the custom measure, such as summing, averaging, or standardizing. However, unless your measure has specific mathematical properties these methods will simply bias the results in new and exciting ways. Work with a measurement expert to determine the best way to interpret scores.
Once you have developed a custom measure, the best thing to do with it is to fully document its development and share it with the world. Not only will you be helping future researchers and evaluation consultants speed up their projects, you will also benefit from the accretion of validity evidence over time. The more people that use and test a custom measure, the more we understand about its strengths and drawbacks, and ultimately, the more we understand about our evaluations.
Read more of our latest blogs!
- New Team Member Highlight: Savanah Cole, Administrative Specialist
- 2024 National Rx and Illicit Drug Summit: EVALCORP Honored to Support our Ventura County Behavioral Health Partners
- New Team Member Spotlight: Stella Park, Research Assistant II
- Understanding the Power of a Data-Informed Culture
- Employee Spotlight: Kaylen Sanchez, My Journey as a Research Assistant