User Research is NOT overrated

Last week I stumbled across this article on Medium — User Research is Overrated — that caught my attention. As a consultant in user research, the title alone is enough to make one defensive. But I was intrigued because I wanted to know why the author was claiming this, especially since I have mostly very positive experiences with the results of user research.

This paragraph is the gist of what this article is about:

It took a while, but eventually we started to notice a worrying pattern: We would do the pre-research for a specific product or service, do the interviews, create the personas, create the documentation then as soon as we got down designing and testing the actual product, we figured out that even though it was nice to “have the user in mind” when designing, the useful data came from the first user tests, not the research. In fact, more often than not, the personas and other documentation just started gathering dust while the rest of the process continued.

I too have seen cases where up-front, general user research was not the right answer. But it’s not because the recipe you tried out didn’t result in a nice dinner that you should abandon cooking altogether.

The problem with this article is that it starts from a definition of user research that is still too often common ground: user research is identified as a bunch of methods in a specific order.

Firstly, things like interviews, personas, empathy maps, customer journeys, and many others are just tools. They are enablers to get to the hidden insights. But on their own, without a proper reason for them to be used, they are completely meaningless. You don’t need personas (or any other tool for that matter) to work in a user-centered way.

Secondly, user research is not a checklist either. It is not by going through a fixed number of steps that you get to the insights that fit any project. Every research is different, every project requires its own approach. And on top that, you also need structures to collect, organise and analyse all the bits and pieces of information, while communicating properly to all other stakeholders to push them to do something with it as well.

A Scientific Approach for User Research

By now everybody knows that any project has at least these three parts: (1) a business/organisation part, (2) a technology part and (3) a user/audience part.

User research needs to be focused on figuring out how big an overlap there is between the user needs, business incentives and technological possibilities. There is usually no point in developing knowledge about the public in general. To do this properly you can/should use a scientific approach including all its basic steps: (1) observe/measure the facts, (2) identify the parts you don’t understand and formulate good research questions, (3) come up with a hypothesis that explains what is going on so you can predict the future measurements, (4) define some tests to validate/falsify your hypothesis and (5) confront your insights with other existing theories.

The 3 Basic Parts of User Research

So, like in a physics research project, you have 3 elements that constitute good research.

1 — Start from the facts, but look through the right lens

Any product or service should be built on facts, not untested assumptions. Just like it is a bad idea to spend money you don’t have, it is a generally a bad idea to build products your target audience doesn’t want.

So you need user insights to participate in your design decisions. But any observation, measurement or literature study is performed through some lens. Again, using an example from physics: you cannot see radiation with your eyes. You need special equipment to do the measurements. And those tools have their own set of limitations (and sometimes your measurement even alters the system as in quantum physics). So you need to think in detail about what you need to know and what is the best way to measure it.

This means that observations cannot go without a problem definition. Your method of observation follows from your problem statement. Obviously the influence works in the other direction as well: the observations alter the problem definition as well. Thus, for example, what you include in a persona depends on what you’re trying to achieve. This should be common practice but I hardly see any good examples. All too often, I see fancy, flashy user documentation that doesn’t contain any project-specific insights. Does it really matter if your target audience likes granola in the morning? So it is hardly surprising that this kind of effort is simply a waste of time. The example in the article mentioned above is no exception unfortunately.

Also, observations are taken far too literally as well. This phase is about the facts. It goes far beyond just seeing. It also includes all kinds of experiments, for example with early prototypes, analytics, measurements and even literature studies. Anything that you can collect, related to your project, that is not made up by yourself or the product team, should be considered.

2 — True insights have predictive powers

Once you have asked the right questions and have collected sufficient data, it is time to formulate a hypothesis. You need to explain what drives the patterns you have identified.

In almost all cases there are many explanations possible for the same data, yet my experience has taught me that companies often just explore one possibility. In many cases the hypothesis is not even written down and communicated to the other stakeholders. It remains a natural tendency for product teams to see a user behave in a certain way and to jump immediately to a solution, often leading to a fragmented user experience.

A hypothesis is needed to formulate true KPI’s later on. These indicators allow predicting future performance. But to define them properly you really need to know how your system works. If you know what matters for your users, you can come up with indicators that predict their future decisions. Without user research you end up with silly metrics that only serve the ones who came up with them.

3 — Testing is experimenting

The last part is testing your chosen hypothesis. You have to put it through its paces. If your theory proves to be true, so must the new measurements A, B and C lead to the expected outcome.

But, testing is more than showing your product to your users. User testing is building experiments that test very specific parts, set up in such a way that it allows one to learn something. Showing a finished product is not that, because they are filled with features, all intertwined with each other. How can you untangle the feedback if you have no idea about the experience of the parts that it contains?

Testing individual parts is not easy, and can quickly become tricky. Your users need some kind of context as well. You can’t just isolate one specific part from the whole UI and expect your test persons to behave in a natural way. But with some creativity it is often easier than you think to come up with modest experiments that teach you something very valuable.

So, is user research overrated?

No, clearly not. Proper user research is still in its early days. And it should be part of any evidence-based approach to developing products and services. Don’t just build personas because it looks hip and trendy. There is a place for these tools, but you need to understand when and how they make sense.

Looking back at the article that started this post, we should point out that the author is not against user research either. He is doing plenty of that using early prototypes. I hope he too will learn that by properly defining his problem space and selecting the right tools for observing facts, no more money is wasted on clueless research.