In their 1974 paper, Judgment under Uncertainty: Heuristics and Biases, Amos Tversky and Daniel Kahneman identified a series of cognitive biases and heuristics that humans often use when making judgments under uncertainty. These shortcuts can lead to systematic errors in decision-making.

The ‘Insensitivity to Sample Size’ bias was one of these cognitive biases they discovered. Tversky and Kahneman showed that people often ignore the size of the sample when making judgments based on statistical data. This can lead to an overestimation of the reliability of the results from small samples and can impact various areas such as business decisions, policy-making, and scientific research.

For UX professionals, this bias can come into play during the research and synthesis phases. Imagine the following scenario:

You’re conducting usability studies for a new feature. After gathering feedback from three participants, you decide to implement significant changes based on their responses.

Insensitivity to sample size could lead you to draw premature conclusions about the effectiveness of the feature, as the feedback from such a small sample may not accurately represent the views and preferences of your entire user base.

Obviously, the size of the sample totally depends on your user base. If you’re building an internal enterprise tool with only ten key users, the sample size will be very different from a commercial application used by thousands of different people.