As EVALCORP’s lead methodologist, I was proud to represent the firm in a presentation: “Requirements for Measuring Subjective Experiences,” on measurement at the 2024 annual conference of the American Evaluation Association. Specifically, I was selected to talk about the measurement of subjective experiences, or the experiences that people have in their minds, such the experience of being satisfied with the help we get in a program or the experience of feeling supported in foster care. Many of the most important outcomes that we care about in human services programs are subjective experiences, so it is essential that evaluation consultants explore how to best measure this data.
Measuring subjective experiences is difficult but, I argue, not impossible. We just need to understand what the requirements are for measuring those experiences and be able to meet them. In this post, I’ll cover some of the main requirements and conclude with the questions I posed to the audience of professional evaluators.
Key Requirements for Measurement
The first requirement I spoke about for effectively measuring subjective experiences is that evaluators need to make sure the construct we are measuring is real. “Construct” is a word used by measurement experts to refer to whatever it is we are trying to measure. Using sophisticated statistical methods, it is easy to make up constructs that look nice on paper and might even sound real but aren’t. For example, in the education literature, “learning style” is a construct that was once widely thought to be real but given recent studies. probably isn’t. At AEA, I argued that evaluators need to find evidence that the constructs we are measuring are actually real so that we don’t mislead stakeholders with statistics.
A second requirement for measurement is that the construct itself needs to, at least partly, cause the variation we observe in our instrument. “Instrument” is the word used by measurement experts to refer to a tool designed to detect a construct; instruments can be surveys, eye-tracking devices, AI chatbots, medical devices, or other interactive technology. The chain of causation leading from the construct to the observed variation in the instrument is the key signal of measurement. For example, if there is something called ‘visual functioning’, then it is at least partially causally responsible for the person indicating that they have ‘moderate difficulty’ with ‘reading ordinary print in newspapers.’
Another requirement for measuring subjective experiences is that these experiences need to have a high degree of commonality between people. That is to say, we can’t measure experiences that are vastly different between people. For example, if everyone has very different ideas of what it means to feel “welcomed” at school, then we can’t measure that sense of being welcome on a needs assessment on school retention. This commitment is particular to the measurement of subjective experiences because the unit of measurement for these attributes is derived from comparing multiple people.
Even though we want to see a high degree of similarity in the basic definition of the experience between people, we also need to see variation in the level of that experience in order to measure it. This requirement is particular to subjective experiences (and doesn’t apply to tangible things, for example) because the unit of measurement is derived from comparing multiple people. If everyone in the sample experiences the same level of the construct, then they respond in the same way on our instrument and our statistical model collapses—this is part of the strange logic of measuring subjective experiences. For example, if elements in a program, such as a check-in procedure, are highly standardized such that all participants have the same subjective experience of them, this actually interferes with the ability to measure that experience.
In my talk, I emphasized that, if these requirements are not met, there are no alternative procedures that can allow us to measure subjective experiences anyway. Some of these requirements can be investigated using statistical methods, but others need to be investigated by old-fashioned qualitative, detective work.
Critical Questions to Consider
I concluded by asking the assembled evaluators to pose three questions the next time they were thinking about measuring a subjective experience:
- Is this construct actually experienced in some small or large way by everyone in the focal population or did we invent it, perhaps as a matter of bureaucratic shorthand?
- Are we likely to find variation in the level of the construct or just variation in the way that people talk about it or experience it?
- Does this construct have the power to cause all of the things that we think are its calling cards or are some of these observed indicators actually causing it?
Measuring subjective experiences is challenging but essential for meaningful evaluation. By ensuring that constructs are real, that instruments can capture variation, and that experiences are shared across populations, we can provide accurate and impactful insights that organizations can then use to develop data-driven initiatives or demonstrate the impact of their programs. Thoughtful inquiry and careful questioning help us capture what truly matters to the communities we serve.