spring 2003 assignments

assignment two: usability testing and test report

Each team will recruit four participants and perform four tests. At least two of these tests must be video-taped. The other two may be captured using Camtasia.

writing the report

Presumably, the web team will revise the site as necessary based on the results of your testing. Your job is to provide the results of your testing as persuasively as possible, in order to overcome the web team's natural temptation to ignore this feedback.

The focus of your report should be on helping your audience reconcile user needs with business goals and available human and technical resources. Analyze your audience carefully

• who they are
• what their goals are (connect your suggestions to their goals, address their concerns)
• what decisions do they actually have power to make
• how will they read their report (skim it all, skim some sections, assign it to someone else)

It's your job to make sense of the mass of information, to figure out the relationship between your observations and the underlying problems or errors or design flaws, to figure out which problems are the most urgent or will provide the highest payoff if fixed, then present that information to designers and decision-makers effectively and persuasively. You'll want to put the errors you identify into categories, such as issues related to layout, navigation, functionality, or visual tone. You may also use categories that reflect different areas or functionalities of the site.

Once you've identified and categorized usability problems, you need to evaluate how important they are. Evaluate problems in terms of their

• criticality (how much do the problems impact the site's success)
• severity (how hard will the errors be to fix)
• frequency (how often do the errors occur)
• cost (how much will the errors cost to fix; how much will the errors cost if unfixed)

You should also suggest possible solutions to the problems you identify. Deciding what revisions to make involves reconciling user needs with business goals and technical resources, reconciling cost with increased user satisfaction, reduced support costs, increased sales, more registration, or whatever the benefits will be. If appropriate, define qualitative and quantitative usability goals to be achieved through redesign.

It's your job to make the users real to your audience. As you evaluate your results, look for memorable quotes or images (fotos or video clips) from the tests, or artifacts from the user's workplace, that you can use to enhance your report or presentation. Use these quotes or images to make the users vivid and real to designers and executives.

usability report sections

• executive summary
• description of test methodology and test circumstances (perhaps in appendix)
• test participants
• test tasks
• results--sorted into categories, prioritized, with recommendations
• appendices (screener, script, other test materials, perhaps methodology, participants)

usability report checklist

• Have you written an executive summary that covers the major positive and negative results of your test?
• Does your executive summary speak effectively to decision-makers? Does it focus on the decision-makers' priorities and concerns?
• Have you described the circumstances of the test? (physical location, number of test participants, equipment, connection, version of interface, test team description, etc.)
• Does your report help readers to visualize the users as real people? Were you able to include a videotape of the highlights of the test?
• Have you identified the areas of the interface that need attention?
• Are your observations thorough and persuasive?
• Are your recommendations persuasive and courteous? Do you address feasibility issues?
• Have you categorized your observations into logical categories that accurately reflect the patterns you saw in your testing? Do your categories also relate logically and effectively to the concerns of the site owners and stakeholders?
• Have you categorized your observations appropriately in terms of their seriousness? Do you maintain credibility by not exaggerating or minimizing the importance of the problems you've found?
• Have you included an appendix containing your test script, screener, and any pre- or post-test questionnaires?
• Have you included an appendix of summary information about your test participants?
• Is your usability report professional and attractive in appearance? Are major sections clearly marked? Have you made the report accessible with effective use of headings and white space?