Honing Your Mobile Apps With Savvy User Research

Executing Your Research Plan

If you’re conducting passive user research, watch your data closely for the first few days. It’s important to verify that pixels are tracked and data are stored correctly. After that, just sit back and relax until you collected enough evidence. How much evidence is enough? That highly depends on your hypothesis and type of experiment, but beware of calling the test results too early and making erroneous decisions by extrapolating conclusions from scant data.

Managing In-Person Tests

If you’re moderating an in-person user test, you need to be very engaged with your test subjects:

• Make them feel comfortable: Introduce yourself and the test, let them understand they can stop at any time, and ask them some background questions about their day to help them relax.
• Ease them into the flow with a scenario: As they relax, gently transition the casual chat into their first task with a scenario like: “Imagine you’re at home, browsing for clothes, and you stumble upon this …”
• Ask open-ended questions: Never ask your subjects a question that can be answered with a simple “yes” or “no”. Instead, ask questions such as “How does this make you feel?”, “What do you expect to find?”, “Describe what you think will happen if …”
• Listen: Don’t interrupt the user. The more they talk, the more data and evidence you will collect.
• Record: With the subject’s consent, record the session so you can rewatch it. At the same time have an observer take notes, so that you can review the session later. Do not tempt fate and rely on your memory after the fact.

Analyzing your Research Results

Once you have collected enough data, it’s time to draw some conclusions, figure out the next steps, and do it all over again!

Experiments

If the experiment ran for an appropriate amount of time and one of options is the clear winner, give yourself a pat on the back and ship it!

If the evidence is confused, don’t give up. Think about what could have gone wrong and verify it. Question your assumptions. Make sure that there’s nothing functionally wrong with the app, or with your test, and verify that the data is reliable. If the results are still inconclusive, shut down the experiment and move to the next one.

In-Person Tests

Gather all notes and recordings from you and other observers while the impressions are still fresh in your mind. Go through each task and each user to check how many subjects completed a task, find commonalities, and condense your findings.

Your goal is to convert these findings into actionable results and propose further iterations. Don’t forget to send your test subjects a thank you note after their session. It makes a difference. :]

Case Study: Results Analysis

At Rent the Runway we recently ran a series of user tests and experiments to better understand how our customers use reviews in their decision-making process. Using automated feedback forms, pixel tracking and online user tests, we were able to identify several opportunities to improve the display of reviews, which also led to an increase of sales.

The views involved in the tests were the product page, review list and review photos. In the product page we spotted the following issues:

1. Limited amount of rating information exposed, users want to get more “at a glance”.
2. The review picture UI was confusing, because users expected to be taken to a specific review, rather than a list.

The review list suffered from the following:

1. No sorting mechanisms. One of the biggest email complaints we got was: “Why can’t I sort the reviews?”

Finally, the reviews photos had the following problems:

1. Photo was too obscured by the textual information (even when very little), and users got annoyed by swiping up and down.
2. Text was small, and the typography nearly illegible. Hard to catch it all in a glance.

Users that paged through reviews converted at much higher rates than users who didn’t. Seeing how a dress fit others with similar proportions was a top factor in making the decision to buy. Our goal was therefore to make it easier for users to find reviews, and allow them to quickly find reviews posted by women with similar body size. These pain points and goals guided several iterations of improvements. In the end we settled on the following:

1. Display a featured review in the last image with a call to action to read more reviews
2. Highlight data about fit in the product page
3. Introduce different criteria to sort reviews
4. Improve the layout and text of reviews photos

After implementing these changes we noticed an uplift in the conversion from product page to purchase via reviews, which ultimately led to more sales. See how powerful user research is?

Where to Go From Here?

In this article we have I discussed many techniques that you can adopt to guide your product development and usability testing! Here are the key takeaways:

• User research should be a constant and consistent part of your product development process.
• Define your goals clearly before starting any kind of test.
• Every method has pros and cons, so choose the ones that make sense within your constraints.
• If you are the developer of the app, don’t moderate user tests yourself.
• Write very detailed scripts for online user testing to avoid miscommunication.
• Ask open questions, and encourage users to think out loud. Don’t interrupt them.
• Let experiments run for an appropriate amount of time before declaring the winner.
• Add a rating prompt and contact form to your app to collect feedback

If you have any comments or questions about this article, or testing in general, please join the discussion below!

Additional Reading

To learn more about active user research, the NNGroup has a fantastic collection of articles on research methods, and in particular user testing.

An Introduction to In-App A/B Testing is an excellent starting point for A/B Testing, but before you run your first experiment read when not to run tests.

Check out these great tips for moderating an in-person user test.

To learn more about usability in general, Steve Krug’s Don’t Make Me Think or Rocket Surgey Made Easy, and Don Norman’s The Design of Everyday Things are key books.

This article by WordStrem weighs the pros and cons of seven of the more popular survey tools. PCMagazine has an interesting chart showing the capabilities and limitations of many of these platforms to help you making the best choice for your needs.

Solving complex UX problems is easier when you understand the basic patterns for mobile Navigation. That link will take you to an excellent primer on the matter.

When you’re done with research and it’s time to start making some (re)design decision, check our article on UX Design Patterns for Mobile Apps. It’s a great starting point for the next iteration of your app!

To learn more about translating your goals into tasks and open-ended questions for a script, read this article about turning user goals into tasks for usability testing.

This excellent template by Steve Krug is a good starting point for a script.

Learn how easy it is to call test results too early. A must-read if you regularly touch A/B tests.