When trying to boost the efficacy of new functionality, placing a high emphasis on incorporating research practices into your product development processes is a good start. But research comes with a lot of hidden costs: time to design the research correctly, time to recruit, time to execute, time to analyze, and time to document. Not to mention the opportunity costs of other research you could have been doing in that time.
How do you ensure that you are not only doing research, but the right research on the right topics by using the right methods? And perhaps more importantly, how do you stack your research methods so that they effectively build on each other and lead to actionable and impactful outcomes?
Before diving into testing methods, it's important to prioritize what you want to test first. Then, you can choose the appropriate testing method based on your needs. Assumption mapping is often a great place to start, as it helps you visualise and prioritise what you want to test further.
Performing an assumption mapping exercise with your team is a great way to visualise the assumptions you have about your product, customer, and business outcomes on one map. This not only helps you align as a team and see all your assumptions in one place, but also helps you easily identify which assumptions to prioritise and gain more evidence for.
When listing assumptions within your team, make sure that you are listing assumptions that you have actionable control over getting evidence for and executing the outcomes on. This could mean limiting assumptions to your department, team, or project scope.
Assumptions that fall within the top right corner of the matrix should be prioritised, as they have the highest impact potential but require more evidence before being executed on. It's best to start by testing one assumption at a time, rather than trying to execute on all of them simultaneously.
Once you’ve defined your riskiest assumptions and decided what to test, it’s time to look figure out the how. Typically, the focus of your test will fall within one of six main areas:
Plotting your target assumptions against these dimensions helps you identify the best testing method more easily. For instance, surveys are particularly strong when gaining insights on user opinions and even testing value proposition, but fall short when predicting future behaviour or evaluating growth strategies.
While this varies case by case, here’s a handy guide that will help you narrow down your options broadly based on what you’re looking to test. The test examples provided are not explicit, and the 6 dimensions presented are more focused on early-stage validation rather than late-stage optimization. Ultimately, however, it’s important to remember that you should choose a test methodology based on your goals and not the other way around.
When doing early product discovery, the goal is to gain intuition about the market and your target customers as quickly as possible. This involves sacrificing statistical significance during research for the sake of gaining rapid feedback and running a large number of small-scale tests rather than a small number of large-scale tests. Indeed, tests during discovery are focused on shaping and gaining confidence in the solution, while tests during delivery will focus on fine-tuning and optimising the delivered product.
This will lead to test progressions that increase in both complexity and sample size as your research progresses. As you gain more confidence in and definition of your proposed solution, the tests you run will begin to more accurately resemble the final product you intend to launch. The sample sizes will also increase organically as a result, due to the fact that you have increased confidence in your solution and require larger burdens of proof to make small-scale tweaks and adjustments.
Lastly, combining quantitative and qualitative research methods helps you get a more complete picture of your users. Quantitative research can provide numerical data, while qualitative research can provide deeper insights into user behaviour and attitudes behind the numbers. Typically, you will start with qualitative research in order to gain initial insights before validating your findings with a larger audience during a quant study. In the case of a live application, as an additional example, this can involve performing in-person usability testing of a solution (qualitative data) before launching an A/B test with the larger audience to measure the design on a larger scale (quantitative data).
Last year, we were building a financial application targeted specifically at those with low financial literacy, little experience saving money, and a wide range of cultural backgrounds. We used a variety of testing methods, mixed quant and qual, and build in product complexity over time as we gained more confidence in our solution.
Our testing cadence was roughly as follows:
When building a research, you have no idea what insights you’re going to find along the way. This is the point of research, after all. It’s important, therefore, to leave room in your research plan to be flexible and adapt methods and targets based on the insights you find in previous research.
For instance, if you find a pain point in your purchase experience while testing chat support, dig deeper into the purchase journey with a usability test or with heat mapping. And ideally with both, if it makes sense. Don’t be afraid to shake up your previously planned research in order to dig into the outcomes you find along the way.
When it comes to your research plan, it is important to not limit yourself to research methods that you’re used to using. Many teams rely on interviews and surveys to make decisions and stick to them because they are familiar and teams are comfortable with them, even when they’re not the right tool for the job.
It is important to not limit yourself to a traditional research methodology or to the methods you’re most comfortable with using. Instead, explore how you can use methods like co-creation, card sorting, and remote usability tests to bring added insights that you wouldn’t get through regular surveys or interviews. Don’t be afraid to go outside your comfort zone and try a new testing method if you think it’s the right tool for the job. You may be surprised at the added value that comes as a result.
For more examples of testing methods by data type and stage of product, check out the UX cookbook repository.
Thumbnail Photo by Kelly Sikkema on Unsplash