Mobile testing: filmed in front of a live *stakeholder* audience

One of the best ways to build tech­nol­ogy that truly fits into people’s lives is to observe peo­ple using your tech­nol­ogy.  At my job, I fre­quently set up usabil­ity tests, so that we can observe peo­ple using the inter­faces we have designed.  It has become eas­ier and eas­ier to run these stud­ies, thanks to tools like GoToMeet­ing, or Morae, how­ever, one of the most chal­leng­ing tests to set up are tests on mobile devices.  You start out with many of the nor­mal tasks to con­duct web site usabil­ity tests: email cam­paign­ing, sched­ul­ing par­tic­i­pants, coor­di­nat­ing con­fer­ence rooms, tech­ni­cal set up… but then you have the addi­tional chal­lenges of set­ting up mul­ti­ple web­cam feeds, try­ing to stream live video into the next room, host­ing a remote obser­va­tion room, and keep­ing your eye on stake­holder ques­tions while mod­er­at­ing the test with the participant.

We know how pow­er­ful and inspir­ing it is for stake­hold­ers and devel­op­ers to watch ses­sions live, while you are con­duct­ing the tests with par­tic­i­pants.  But watch­ing mobile test­ing live? While in the field? My sister-in-law inspired me to believe this is pos­si­ble because she once live streamed her­self run­ning the Oki­nawah Marathon in Japan. It was almost like being there with her…the kind of expe­ri­ence you want your devel­op­ment teams to have, so they can under­stand the con­text your users are in…

So how do you run a mobile usabil­ity test in front of a live stu­dio audi­ence? Here are a few tips I picked up from watch­ing a lot of 30 Rock (NBC’s Com­edy Series, cre­ated by Tina Fey, about putting on a late night talk show).  The fol­low­ing are a few thoughts you can use to emu­late “Liz Lemon” traits:

1. Speak your mind.
When things are get­ting tough, go to your boss and be vocal about it. They are never going to know what you are strug­gling with unless you TELL THEM. Just tell them the whole story, and they will help you find a solu­tion… or at least tell you not to worry about it.

2. Keep track of the details.
Keep lists of to dos. Make check­lists for repeat tasks. Dou­ble check your work. Test. test. test. Rehearse like crazy. Nail your open­ing script. Be on time. Do every­thing in your power to cover the details. You might start to feel hope­less, if you are cov­er­ing all your bases, but not scor­ing any runs… if you know what I mean. But try not to worry about that. Just try not to drop any balls.

3. Stay focused on your work.
Many peo­ple will try to tell you how to run your test. This can be dis­tract­ing if you are not stay­ing focused on the test you crafted. They are great at offer­ing their opin­ion on the par­tic­i­pants you’ve brought in, the sce­nar­ios you wrote, and giv­ing their sug­ges­tions of what you should have done dif­fer­ently. Every­one has an opin­ion, and while it shows that they care about the research, they are almost always wrong. For exam­ple, peo­ple who com­plain about the peo­ple you recruited wouldn’t know the first step in find­ing the right peo­ple… most of them wouldn’t even know how  to describe who the “right” peo­ple are! Just stay focused on you… you need all the self con­fi­dence you can muster up for this “show” to go off well.

4. Shake off the mis­takes.
You can’t get held up with the mis­takes you make. You might for­get to record the audio prop­erly, or ask that extra ques­tion your stake­hold­ers asked you to put into the script last minute. Your com­puter might go to sleep, leav­ing your observers with noth­ing but a blank screen.  Just shake it off… take a deep breath, and ground your­self. Focus on what the par­tic­i­pant is say­ing, and fol­low their lead.

5. Don’t sweat the stress induced fore­head acne.
Your adren­a­line will kick in once you get doing dur­ing your tests. Then, all of a sud­den you will lightly brush your hand across your fore­head to dis­cover a sore lit­tle bump, freshly arisen, out of nowhere. Even if you swear to be calm, and not sweat the small stuff the week you are test­ing, your adren­a­line will kick in, pro­duc­ing stress hor­mones that cre­ate break outs on your face. Don’t pick at it, just let it run its course.  Your body needs to heal.

Just remem­ber, run­ning a usabil­ity test in front of a live stake­holder audi­ence may be stress­ful, but it is reward­ing. Watch 30 Rock, and pick up some tips.