The Rogue HCI Blog

Have you heard? Small Ns are bad in quantitative projects. “Right”, you will say, “I will henceforth try to get as high of an N as possible”. Consequently, you also want to generalise your inferences to your population. Because why shouldn’t you. The decision-makers that read your research (the number of which is mostly zero, but let’s just pretend) need to know whether those inferences hold up at a population, or even human level. So you try to get representative samples for every consecutive project, if your budget allows for it.

Now, I have very bad news for you: Firstly, you might not even be methodologically savvy enough to generalize your inferences, or even build models that allow for generalization in the way you want (see here). But also: you might be wasting your money. There are (imho) not many use cases where representative sampling is actually worth the trouble. But first, let me give you a quick refresher on sampling.

Sampling is a procedure that we do for (mostly) two things:

1. To estimate population values without having full population data.
2. To introduce randomness in our research, enabling us to leverage probability theory for our estimation models.

In other words: sampling often is the thing that gives us probabilities (in contrast to likelihood) in the first place, if there is no randomized allocation. If you measure the entire population, you do not need to estimate. You can work descriptively. Alas, we almost never work with entire populations. That’s why we sample things and then estimate something, combined with a quantification of uncertainty. Thank you for attending stats philosophy 101.

Knowing all of this, representativity is a thing that depends very strongly on some assumptions – and those assumptions often are…let’s say overplayed. Because there are three questions that will tell you whether representativity is actually worth something for your project.

Representativity, Shmepresentativity.

The following questions are vital for any project that even thinks about representative sampling.

1. What is the nature of my research?
2. What is the goal of my research?
3. Which context does my research inhabit?

These questions will determine your choice of sample. Let’s go through them one by one.

What is the nature of my research?

Some research is more deserving of a representative sample than others. Especially when you conduct research with living things, representativity might be interesting (otherwise all you care about is power, or raw N). A very good reason for representativity is when your subject or hypothesis touches the lives or substance of every member of the population. Medicine is the best example: You want to ensure that new medicine works for the vast majority of patients (whatever majority means, that’s your personal definition). This is why you need to sample as diverse (representative, for certain people allergic to the word) as possible, with strong power to get effects that count for that majority. Likewise, you need representative samples if you want to explain general tendencies in humans, for example. Cognitive or social psychology is in dire need of such things, because we cannot guarantee that thinking or action is uniform across individuals – or even cultures. But this leads us to the second question.

What is the goal of my research?

Do you want to know whether your medicine works for most humans? Well, then you need to sample from most humans (this is why drug development is so damn expensive, up to a billion Dollars per approved drug). But cognitive or social psychologists don’t have that kind of money, often not more than 2k for one study. But they should, right? So in lieu of that, they are often satisfied with representative samples of one country, with rather low N (100-300). But the question remains: What are you estimating then? In most cases, you estimate the population parameters of that specific country. Sure, countries next to it might have very close or similar values – if we’re being generous. But people on other continents might not. This paper (finally back to HCI) is an interesting demonstration of this effect. Their models for privacy work (somewhat) in Germany but break down in the US, close to zero variance explained. This means you cannot generalise the findings across countries at all, despite representative quotas within countries. Which leads to the third question:

Which context does my research inhabit?

Whether representativity is worth something often strongly relies on the context of your research as well. In the case of the last paper, I’d argue that representativity is informative, as it is specifically tailored to inform national decision makers (legislative and otherwise). But for general tendencies, this paper tells us absolutely nothing. And how could it? As we can see from it, the whole deal is highly culture-specific. Representativity ends with cultures here. There is no general tendency.

This is what I mean by my question: Do you want to focus on specific aspects of a specific population? Great, representativity is your friend. Do you want to find out general tendencies that are present across populations? Then you should either sample from everyone, but I can tell you right now: “All Humans” is almost certainly not a population – for reference just read cross-cultural psychology papers. In case you already took a nationally representative sample out of habit, be really careful about inference. In most cases, you’re better off (financially and considering inference) doing an 80% power sample off of a reasonable SESOI from as diverse demographics and cultures as possible, while ignoring population kookie talk.

But what do I tell the reviewers?

While my gut reaction would be to tell them to fuck off with their BS representativity demands, this will neither increase your chance of publication nor foster learning on the other side. So what you need to tell them is the backstory of what sampling and generalizability means. Argue that representativity does not make sense in your case, or is financially infeasible, or (and this is mostly the case) narrow population-data (!) is borderline useless if one wants to research supposedly general tendencies. If they still fight back, just respond with a recommendation towards a book on sampling theory which they will never read anyways, and end the conversation.

What you should not do is yield to stupid demands. Many reviewers, especially in HCI, have close to zero statistical or methodological knowledge and are too ashamed to admit it. While this is just human, it does not serve the peer review process well. Should you encounter such people – and you will be able to tell from the reviews – it is best advised to remain calm and explain the situation to them with a lot of examples or graphs. Like you would explain it to a student. I had good success with this, and I hope this strategy might be useful to you, too.

In any case, stay (un)representative folks.

~R