One Take #15 - Questioning the Questionnaire

Have you ever wondered how researchers measure something as subjective as your comfort in a building? The latest episode of Air Quality Matters takes a surprising step back from specific pollutants to examine one of the most fundamental yet overlooked tools in indoor environmental research: the questionnaire.

When scientists ask you to rate how stuffy a room feels or how comfortable the temperature is, they're relying on scales and questions that may be fundamentally flawed. Marcel Schweiker's paper "10 questions concerning the usage of subjective assessment scales" exposes the messy reality behind these seemingly simple measurements. From the "wild west" of inconsistent scales across studies to the profound problems of language translation, we discover why comparing results between different research projects is nearly impossible under current practices.

The episode dives deep into the philosophical core of measurement itself. What are we actually capturing when someone circles a number on a comfort scale? Rather than obtaining clean data, we're glimpsing a complex psychological construct filtered through cultural expectations, sense of control, and even social desirability bias. A person who knows they can open a window will perceive air quality differently than someone who feels trapped in the same conditions. The podcast explores alternative measurement approaches including physiological signals and behavior observation, but concludes that questionnaires remain essential - if properly designed.

For anyone interested in buildings, air quality, or the science of human comfort, this episode offers a fascinating look at how the research community must evolve to better capture our messy, subjective experience of indoor environments. It's a call for more thoughtful, critical approaches to the science that shapes the buildings where we spend 90% of our lives. Listen now to gain a fresh perspective on what it truly means to measure comfort in the built environment.

Ten questions concerning the usage of subjective assessment scales in research on indoor environmental quality

Chapters

• 0:14 Introduction to One Take Format
• 0:40 Questionnaires: Our Overlooked Research Tool
• 1:20 The Wild West of Measurement Scales
• 3:04 Language Barriers in Comfort Surveys
• 5:08 What We're Actually Measuring
• 7:28 Beyond Scales: Alternative Measurement Methods
• 8:32 Conclusion and Call for Rigor

Transcript

Speaker 1: 0:14

welcome back to air quality matters. And one take one take my take on a paper or report on air quality ventilation in the built environment. One take in that it's well in one take and tries to summarise for you a scientific perspective on something interesting in well, usually 10 minutes or less, because who has the time to read all these amazing documents? Right? This week we're going to do something a little different. We're taking a step back from specific pollutants or ventilation systems and looking at one of the most common fundamental tools in our toolbox, a tool so common we barely even think about it. I'm talking about the survey, the questionnaire, the simple act of asking someone how do you feel on a scale? This paper is 10 questions concerning the usage of subjective assessment scales in research on indoor environmental quality, and it's by Marcel Schweiker and a really impressive list of co-authors from across the field. And the reason this paper is so important is that it forces us to stop and think about the very foundations of how we measure the human experience of buildings. We rely so heavily on asking people to rate their comfort, their satisfaction, their perception of warmth, stuffiness, glare or noise, but this paper poses 10 critical and maybe a little uncomfortable questions about whether we're actually doing it right. It's a bit of a soul-searching exercise for the whole indoor environmental community, so let's get into it.

Speaker 1: 2:00

The first big problem the paper lays out is what I'd call the wild west of scales. If you look across the research, you'll find an almost uncomfortable number of different scales used to measure the exact same thing. For thermal sensation, you've got 3 point, 5 point, 7 point scales. For air quality, you'll find simple, acceptable, not acceptable choices, Scales for odour ranging from from no odor to overpowering the scales for stuffiness. It's a mess and the consequences of this is that it becomes incredibly difficult to compare results from one study to another. It's like trying to build a global picture of human height, but everyone is using their own homemade ruler. It fundamentally limits our ability to build cumulative knowledge, but it gets more complicated than that. One of the really fascinating points in question two is how inconsistent we are between different environmental domains.

Speaker 1: 3:04

For thermal comfort and, importantly, for air quality, we have a pretty established two-step process. First we ask about perception. For thermal, it's the famous ASHRAE scale from cold to hot. For air quality it might be a scale from fresh to stuffy or a rating of odour intensity. Then we ask an affective question Is this comfortable or acceptable? But when we switch to say acoustics, we tend to skip the perception part and jump straight to the affective judgment how annoying is that noise? We don't typically ask how loud is it first. This makes a true holistic understanding of the indoor environmental quality, where you're trying to balance all of these factors really, really difficult because we're not asking the questions in a comparable way.

Speaker 1: 4:03

Then there's the language barrier, which is the focus of question three. The paper gives its most detailed examples here from the world of thermal comfort. But the principle applies everywhere. We take a term like slightly warm and assume it's universal. It's not. The authors point to research showing how these translations can go wrong. When translating into some languages the word warm carries a positive, cosy connotation which is very different from the neutral, scientific meaning intended. You can imagine the same problem for air quality. Does the word for stuffy in other language carry the same meaning, or does it imply something more like stale or unhealthy? It's a profound reminder that these aren't just abstract numbers, they are words, and words are loaded with cultural and linguistic baggage. So if the words are tricky, what about the structure of the scale itself. This is question four. People, it turns out, have something called a central tendency bias. We don't like picking the extremes.

Speaker 1: 5:19

The paper also warns against a classic mistake merging two different concepts into the same question. The old Bedford scale for thermal comfort was a famous example of this. It's like a doctor asking you, on a scale of one to seven, how much pain and happiness are you feeling? You can't answer that sensibly because they are two different things. Yet we see these kind of conflated questions all the time, whether that's mixing sensation and comfort in thermal studies, or perhaps acceptable and odour pleasantness in air quality research. This all leads to the deepest, most philosophical question in the whole paper, which is question five.

Speaker 1: 6:09

What are we really measuring with these scales? We like to think when we ask someone how they feel about air, we are getting a pure, objective report on their perception. But we're not. What this paper argues so powerfully is what we are measuring is a complex psychological construct that is filtered through a whole host of other factors. It's influenced by our recent past. It's influenced by our sense of control. If you know, you can open a window to get fresh air, you are far more tolerant of the current conditions. It's even influenced by your mood and perhaps most importantly, it's influenced by what researchers call social desirability bias Our tendency to want to give the correct answer. We might not be reporting how we actually feel, but how we think we should feel. So the scale isn't a clean thermometer of human experience. It's a cloudy, complex window into a person's entire relationship with their environment.

Speaker 1: 7:17

Given all these problems, the authors then ask well, can we just get rid of scales altogether? Questions seven and eight explore the alternative psychological signals and behaviour. The paper's conclusion is balanced and sensible. Basically, no, not really. Psychological signals are great at telling us the body is reacting to something, a stressor but they're not specific and behaviour is just as ambiguous. The paper gives a brilliant example here a window opening. We might assume someone opens a window because they're too warm, but multiple studies have found that co2 concentrations, our best proxy for indoor air quality is actually a better predictor of when people will open windows, and in some cases, completely the other way around. The behavior is the same, but the environmental driver is completely different. You can't tell which it is just by watching. These methods provide fantastic complementary data, but they cannot replace the simple, direct value of asking someone how they feel.

Speaker 1: 8:28

So to wrap this up, what's my take on this? The paper isn't trying to say that questionnaires are useless. Far from it. It's an urgent call to treat them with the scientific rigor they deserve. The author's main recommendation is for a fit-for-purpose approach. We need to stop grabbing the nearest seven-point scale off the shelf and instead consciously design our measurement tool for the specific question we're trying to answer. Ultimately, this paper is a reminder that the human element in our buildings, our experience of warmth, fresh air, light and sound is messy, subjective and beautifully complex. Measuring that experience is one of the hardest things we can do. This paper doesn't just give us all the answers, but it does an incredible service by making us ask all the right questions about the tools we have. It's a call for us to be more thoughtful, more critical and, ultimately, better scientists, and for any field, that's an essential breath of fresh air. Thanks for listening. I hope you enjoyed this week's episode of One Take. This podcast isn't possible without our partners Safe Traces and Imbiote. So thank you, See you again next week.