5 Steps for Facilitating a Remote Usability Test

June 24, 2011 · by · in Usability Lessons, User Research

This post breaks down facilitating a remote usability test in 5 steps: Pre-Planning, Study Setup, Pre-Study Survey, The Study and Completing the Study. You’ll probably notice that most of the steps are very similar to an on-location usability test, but I wanted to put this post together to outline some of the difference, and tips that have worked for me. At the end I have included a video from Steve Krug for those looking for a usability test demonstration. The video is for an on-location test, but for those new to usability testing it’s a valuable watch to see how he facilitates, and the kind of feedback to expect from users.

  1. Pre-Planning
    1. You should have already chosen your user groups, scheduled participants, written your scenarios, setup the test environment, and run a pilot test. (Tips on these steps to come in future posts.  For more information in the meantime I recommend looking up Author Joe Dumas.)
    2. I find it helpful to send a copy of the Informed Consent via email prior to the test, as well as setting expectations that the participant will want to find a room where they can be alone, and won’t be disturbed. I include a line that even though others they know might also be interested in watching the study; we need them to be alone during the study so that we see how they would interact with the application on their own. (Of course this assumes you aren’t looking to observe social interactions surrounding use)
  2. Study Setup
    1. Begin the conference call and connect to your chosen screen sharing software 5-10 minutes ahead of schedule. I typically use WebEx with a separate conference line. WebEx allows me to enter the conference line information so that voice and screen will both be recorded, and it is the primary screen sharing software where I work.
    2. Once the participant has joined, welcome the participant by name, introduce yourself, and thank them for participating.
    3. At this time I ensure they received a copy of the informed consent and tell them I’m going to begin the recording.
    4. I then share my screen which has a copy of the informed consent open on it, and read it over with them to ensure they are comfortable with it and that their consent is recorded.
    5. The informed consent should describe the general outline for the study, about how long it will take, that they are being recorded, how the recording will be used, any security regarding their identity or personal information, and that they are free to stop or take a break at any time. I let them know that the recording is really there to help me out so I can pay more attention to what they are doing during the study, and less time worrying about taking notes since I can review the recording later.
    6. Next I remind them that they are helping us to test the system in a real-world scenario to see if any changes need to be made. So they can’t do anything wrong, because everything they do helps us understand where we need to make improvements. Therefore we are looking for their unfiltered honest feedback.
    7. I let them know that throughout the study I will need them to talk aloud about everything possible, from where they are looking, to their reactions, emotions, and thought process. I mention that this is especially important because I won’t be able to see them (often the case remotely) and so I will need them to describe emotions like confusion since I can’t see if they have a confused look on their face. The more they describe aloud about their thoughts, and reactions, the better the feedback for us to understand what improvements we can make to the system.
    8. Finally, I let them know that if they have questions during the test, I may not answer directly, but it’s only because I want to see what they would do if I weren’t there to help. And that at the end I will be happy to answer any outstanding questions they have.
    9. I conclude the setup by asking if they have any final questions over what we have just covered.
  3. Pre-Study Survey
    • At this point I will ask any questions I have set aside regarding background information that I still need to get regarding the participant and their experience. Questions like, time spent online, favorite websites, etc.
  4. The Study
    1. As an exercise I then bring up a generic webpage and ask them to describe the page in front of them, what they are motivated to click on first, and why. I ask if they have any questions before we start based on the exercise.
    2. Next I will open the website or application and start the first scenario. Typically I will have the scenarios open in a file in a window that isn’t shared, and copy/paste the scenarios as we get to them into the WebEx chat. I will then have them read the scenario to me and then start that scenario. I find this helps not only with getting them to talk aloud, but I can also typically note if they sound confused by the scenario or not. Additionally, it allows them easy access if they need to reference the scenario during the study to clarify what they are doing.
    3. I try to use acknowledgement tokens sparingly when the participant makes a comment to me, by saying things such as: ah, okay, that is good feedback, un huh, etc. However as much as possible I try to be a silent observer, only probing when needed such as if they feel stuck, by asking what they would do if I weren’t there, or by having them re-read the scenario. Occasionally, I find I will need to remind them to talk aloud with something like “so I noticed you’re a little quite, what are your thoughts about this?”
    4. Depending on the study I may ask them follow up questions after each scenario, or to rate their satisfaction with the task or system.
  5. Completing the Study
    1. Upon finishing the last task I ask any post-study questions I have, and then ask if they still have any outstanding questions that I could answer for them.
    2. I try to emphasize how valuable there time has been to me, and the company, and that we are gracious for taking the time out of their busy schedules to assist us.
    3. I ensure they have my contact information for any questions they may have later, and provide them with the stipend (if applicable) for the study.
    4. This ends the call, and the recording is stopped and saved.

 

Additional Resources:

Especially if you haven’t run, or observed a usability test before, I recommend watching Steve Krug’s You Tube video demoing a Usability Test. It is a demo of an on-location test, not a remote one, but the facilitating is pretty much the same.