Recently I was talking with another consultant about different UX Research approaches and made the comment that both surveys and focus groups were Quantitative Research techniques. The consultant immediately corrected me pointing out that they were going to ask open-ended questions in their surveys, so it was Qualitative Research. It highlights a disconnect between many UX Researchers and Researchers with a Statistics or Market Research background.
Frankly, I can see both sides of the little talked about debate, although many on the Social Sciences side would argue there is no debate, UX’ers are just wrong.
Quantitative vs. Qualitative a Definition
Putting it simplistically, the Social Sciences definition of Quantitative Research is about gathering data in numerical format so the information can be easily organized, categorized and ranked. Think of surveys with closed questions, such as a series of Likert scales (an odd number of responses from strongly disagree to strongly agree). It’s very easy to tabulate the results, x% gave this response, y% that response. Perfect for statistics.
Qualitative Research is the other side of the coin, open ended questions and varied responses which require interpretive analysis. For example, if you ask a group of users about their impressions of a site, you’ll likely get 10 variations on the same idea/theme. It can be a bit of a judgement call if ‘there’s too much text,’ ‘the site is dull,’ and ‘it seems [inauthentic]’ are referring to the same problem or different problems.
In general, both sides agree on the most significant distinctions between Qualitative and Quantitative Research. Quantitative Research is about understanding the ‘what,’ what are users doing and how much, many, often. Qualitative Research is to better understand the ‘why,’ why are users doing it, why they feel that way, why they have the expectations they have.
So where’s the disconnect?
The reason why narrower definitions of Quantitative and Qualitative tend to chafe UX’ers is twofold.
First, any UX research technique could be either Qualitative or Quantitative depending on how many users you talk to and what kind of questions you ask. So, the narrower definitions don’t help as much in categorizing UX research techniques or in deciding which techniques to use.
Attitudinal Data vs. Behavioral Data
A more meaningful distinction between Qualitative and Quantitative in UX Research is whether the research method derives data from direct observation or an indirect method such as an analytics tool. Direct observation allows for verifying Attitudinal data (what the user says) with Behavioral data (what they do).
While Quantitative Data is helpful for identifying areas of concern, potential problems, demographics and user opinions/preferences, Qualitative Data is the key for in depth investigation into the root causes, distinguishing between symptoms and underlying causes and crafting solutions.
Observation vs. Users’ Recall
Second, regardless of whether you ask open-ended or closed questions, a survey or focus group both suffer from the same fundamental flaw when it comes to UX data gathering, they rely solely on a user’s recall. When a user fills out a survey or talks in a focus group, the user is interpreting what they have done based on what they can recall of events. However, a user’s memories of events can be wildly different than the reality because of gaps in memory and how they interpret what they do remember.
There was a fantastic study done in which people we asked to estimate how much they’ve given to charity. Based on their recall, participants gave responses that averaged 40% higher than their actual donations. Now the intention of the study was to demonstrate that people are generally better at estimating what other people do than what they do themselves, but it highlights an important point; users have interpretive recall particularly when it comes to things that make them look good or bad.
So, whose right? Probably the Social Scientists (and Merriam-Webster). But that doesn’t mean UXer’s can’t broaden the definition to better differentiate UX Research techniques…..just be ready for some cross-eyed looks from the other side.