My Experience With Planning and Conducting a Remote Usability Test
How do you understand a user’s pain point when they’re thousands of miles away? You are not in the same geographical region as these people but need to understand their needs to give them the best possible experience with your product.
Fortunately, there is the option of conducting research remotely.
Recently, my team and I conducted a remote usability test on Bongalo mobile app — a platform for booking short-stay travel accommodations in Africa. And it was an eye-opening experience.
Background
When I joined Bongalo a couple of months ago, all property bookings and listings were through the web app, as the mobile app was still in the works. So, my first task was to get the mobile app up and running. After investing countless hours into the design process, I needed to ensure that users could use the product.
This article explains how I planned and conducted the usability test while working from home.
But first, what is a usability test and why is it important?
Usability testing involves testing a product with potential users. During a usability test, users will complete typical tasks, e.g. adding an item to a shopping cart, while an observer pays close attention to the user’s actions.
As a product designer, testing your product with users is essential as it not only helps you understand whether the product meets the users’ expectations but also helps to match your business decisions to real-world use.
Why Conduct a Remote Usability Test?
Some key reasons:
- Tight budget
- Short time frame
- Accessibility issues, etc.
In my case, remote testing was the most feasible option for two main reasons:
- The user base of our product is spread across different countries in Africa. We had people participating in the test from Rwanda, Cameroon, Mozambique, etc. It would have been impossible (or incredibly expensive) to get these people in the same physical location for an in-person test
- I live in Nigeria and work for a company in Rwanda.
My Concerns About Conducting a Remote Usability Test
Going into the testing phase, I had some concerns about not being in the same physical location as the participants. I worried that:
- Participants (or even myself) might run into some technical issues, thereby affecting the quality of the research
- I might not be able to read the participants’ body language, which is crucial to understanding their experience with the product.
- Participants might not take the test seriously since they weren’t under the level of supervision they would receive in a face-to-face test.
Planning and Conducting the Usability Test
While most of my concerns were beyond my control, I was determined to ensure that the factors within my control were well taken care of and that the test ran smoothly. To make sure of this, I followed a 4-step process.
Step 1: Creating a research plan
My first step was to create a research plan. Some key information in the plan includes:
- Research goals: (1) To understand whether users find the product easy to use, and (2) To identify the difficulties users face when booking or listing a property on the app.
- KPIs: Task Success Rate and Time on Task
- Target users: We targeted two main categories of people (1) Users already on the web platform, and (2) People who fit our target audience but have never used our product before eg. followers on our social media channels, and recipients on our email list.
- Test script: This contained some task-based scenarios to test the app’s main features. In this case, our tasks were around booking and listing a property on the app.
Note: If you need help with crafting a test script, you can download this test script by Steve Krug and modify it to suit your needs.
Step 2: Recruiting participants
To get volunteers to participate in the usability test, we sent emails and posted ads on our social media pages to target people that fit our criteria. The emails included all the necessary information to give the potential participants enough context on the research (what to expect, what is required of them, etc.). Interested participants were asked to fill out a Google form with a few screening questions to determine their eligibility.
Through this process, we recruited five participants to sign up for the usability test.
I used Calendly to schedule the event and sent a link to the participants to book a time and date at their convenience.
We offered them 10% off their first booking on the mobile app as an incentive.
Step 3: Conducting the test
We used Google Meet as our video conferencing tool.
We began each session with short introductions and ice-breakers to relax the atmosphere and prepare the participants for productive discussions.
The participants were then given a series of tasks to perform on the beta version of the app. For each task, I read out the prompt, set a timer, and then observe the actions the participant takes to complete the said task. As they navigate the platform, I encourage them to think aloud as much as possible. This helped us understand their thoughts on how the app works. Beyond what they were saying, I also paid close attention to their tone, expressions, and other subtle details that could indicate frustration, excitement, or utter disappointment.
At the end of each task, I graded the task success rate from a range of 0 to 1, with 1 meaning that the task was a complete success and 0 representing failure. We also recorded the time spent on each task.
With their permission, we recorded each session to ensure we captured every piece of information. The recordings also acted as a point of reference for the team working on the product.
Step 4: Synthesizing the Test Findings
Having completed all the usability test sessions, I presented a usability study report to the product team and other stakeholders. Key information in the report includes Study Details, Findings, Insights, and Recommendations.
The team focused on resolving the top priority issues while we saved the lesser priority ones for future iterations of the app.
So, were my initial concerns about conducting a remote usability test validated?
Short answer: No!
Interestingly, I found that having the participants engage in the test from their natural environment helped provide richer insights into how they would interact with the app in real-world situations. I was able to witness the distractions (e.g. incoming phone calls) they encountered while interacting with the product. Additionally, I took note of the small, yet important details like the WiFi signal strength in their locations and how they typically use their devices.
Best of all, they were in their homes, so they were naturally more comfortable and expressive.
Conclusion
Going by the quality of insights I got from conducting this remote usability test, it’s safe to say that remote testing can be just as effective as in-person usability tests with the right approach.
