Bargain Basement Usability Testing

The subject of usability generates a dichotomy between what we think and what we do. We know it is good to focus on usability. We need only look at Apple and the iPod to know that it provides tangible benefits. However, in reality we often shy away. It is hard to prioritize usability when deadlines are looming and budgets are tight. Ultimately user testing gets pushed to the bottom of the agenda. It is as if the perceived losses of testing, outweigh the potential profit. But, are these assumptions true? Is user testing time-consuming and expensive?

The Perceived Losses of User Testing

There is a wide-spread perception that user testing is time-consuming and expensive, and to some extent with good reason. Traditionally user testing cost millions and took weeks to complete. It was held in expensive usability labs with two way mirrors, computer suites, and video surveillance. Large numbers of test subjects were required to provide statistically relevant data. Each had to conform to a specific demographic profile and so was hard to recruit.

A usability consultant, testing in a lab, with carefully selected subjects is effective. However, it is beyond the budgets and time frames of most organizations. User testing does not need to be like this. In-fact, it can be lightweight and inexpensive. It is also something you can do yourself. Although not the most effective approach, it is better than not testing at all. However, even the most lightweight user testing will require some time and budget. Do then the benefits outweigh this cost?

The Real Profit of User Testing
The benefits provided by user testing cannot be understated. Even the most lightweight approach will have a profound affect on your web presence.

The benefits of user testing include:

  • Fast problem detection
  • Increase user satisfaction
  • Reduced support costs
  • Increased efficiency

Bargain Basement Usability

The bargain basement approach to usability testing was pioneered by usability experts Steve Krug and Jakob Nielsen. Although they differ on the details, they agree on three key principles:

  • Test a little but often
  • Use a limited number of testers
  • Don’t become too concerned with who you test

If you want to user test while delivering on time and in budget, then these principles are important.

Test a Little but Often
Key to this approach is the principle of “little but often.” Testing should occur throughout a web project’s life-cycle and that means keeping testing lightweight.

At the most basic level this is what Steve Krug calls “cubicle testing”, where any new development is shown to colleagues to see if they can make sense of it. This simple form of testing is remarkably effective.
If however, you choose to be more sophisticated in who and how you test, never let that be at the expense of quantity.

Why ‘often’ matters?
Multiple rounds of testing identify more problems. When initially testing many users get stuck at the first hurdle. A second round provides the opportunity to fix initial problems, allowing users to progress further and uncover new issues. The number of rounds is largely dictated by budget and time. However, the more rounds, the more issues you will uncover. This is true even with a limited number of testers.

When to test?
Before the project commences, test your existing site or that of your competition. These sites act as a free prototype which help identify potential usability issues.

Test sketches and design concepts.
Your designer may resist showing users unfinished work. However, this is a chance to identify issues before too much time has been invested. When testing a design for usability focus on whether the user ‘gets it.’ Do they understand what the site is about and how it works? Did the user understand the terminology used and were they paying attention to the right screen elements?

Before beginning production consider testing against a wireframe. This is preferable to testing against a finished site, which cannot easily be changed. There are a number of tools that can reduce the time spent building wireframes. However, I think, the best time saver is to use a content management system (CMS) to construct your wireframe. If done well, this CMS driven wireframe can be reused as the basis of your final site. The wireframe is therefore no longer wasted.

User testing should not end with the wireframe. As you build the site itself, test that too. It is perfectly possible to test a half finished site. Select tasks for users to complete that focus on finished sections of the site and avoid the unfinished. Finally, do a round of testing pre-launch. By this stage the majority of issues will have been discovered. However, a last check will uncover minor problems that can be corrected.

You do not need to test at every stage. What I do recommend is that user testing should be an ongoing exercise even after launch. Periodically run test sessions both in development and post launch. Always seek new ways to make your site easier to use. Now I have told you when to test the next question becomes; who to test?

Beware of Decreasing Returns

Many website owners believe that usability testing is worthless if you do not test a large groups of people, who all fall within the target demographic. They are therefore discouraged from testing. However, this belief is not true. Statistical testing (as outlined above) is expensive and time consuming. It does not provide significant benefits over the bargain basement approach. In 2000 Jakob Nielsen wrote:
“Elaborate usability tests are a waste of resources. The best results come from testing no more than five users and running as many small tests as you can afford. ”

The reason is that the majority of users encounter the same problems when testing. The more users tested, the more overlap he found in the problems. Eventually the level of overlap is so high that the likelihood of discovering more issues is remote. As I said earlier and Nielsen reinforces above, multiple rounds of testing are more effective.

Steve Krug takes the logic further suggesting that you should only test three or four people. This catches the majority of serious issues, although admittedly some minor issues could slip through. He believes that this is a price worth paying as testing only three or four people allows testing and debrief in a single day. In short, the more people you test the lower your return on investment. ROI is also an issue for recruitment of testers.

Beggars Can’t be Choosers
It is easy to waste time and money recruiting the perfect user who matches the personas we created earlier. This is fine if you have limitless resources. However, it cannot be allowed to reduce the number of rounds of testing.
Although having the right users is good, it is not as important as you may think. Most users encounter the same problems when using a website, no matter who they are. While it’s good practice to design a site that is usable by all, not just your exact target audience, there are exceptions.

It is unwise to test with users who have radically different experiences of the web. You should also be aware of any accessibility needs your audience have that could affect how they use your site. A good example of this is a website aimed at the elderly. This audience have physical conditions that may affect their use of a site and generally don’t have as much experience using the web. You would therefore probably not want to test exclusively with 20 something technology geeks!

Running an Effective Test Session

Running your first usability session can be intimidating, especially if you have never even seen one. You probably have questions about where to run the session, what you are supposed to do and what kind of things to cover.
In actual fact running a test session is remarkably straightforward and can be done by almost anyone.

There are three simple principles you can use:

  • Be prepared
  • Understand the role of the facilitator
  • Work from a script

Be Prepared
It is possible to do usability testing with no preparation whatsoever using the cubicle test I mentioned earlier. However, if you want to run something more formal a little preparation goes a long way. Before you can begin recruiting answer two basic questions. Where and when are you going to test?

Where and when?
If can easily visit your testers, then you may wish to test where they normally access the web. This provides two advantages. First, they are more relaxed because they are in their own surroundings. Second, you are seeing them in their ‘native environment’ which is more realistic. However doing this is not always practical. The alternative is to ask your tester to come to a central location. This does not need to be anywhere sophisticated. It can be even be done at your own desk. The most important requirement of the test environment is that it is quiet and you will not be disturbed. Although a constantly ringing phone maybe realistic it does not aid successful user testing!

Once you know where you are going to test, draw up a schedule. Try to schedule all of your sessions on a single day. Each session should last a maximum of 40 minutes as people struggle to concentrate for longer. Place each session one hour apart. This give you opportunity for additional notes and preparing for the next session. It also allows time for users who have a lot to share or struggle with their tasks.

Always expect at least one person to drop out. In my experience somebody will always be too busy or sick to attend. Have a backup available to step in at the last minute. Even if this is a colleague who is not involved in the project. It is better than nothing.

Who Runs the Test?
The person responsible for running a usability test session is known as the facilitator. Unless you are using an expert, this role will probably fall to you. However, are you the best person to carry out the testing? Is there somebody you could use instead?

The problem may be that you are close to the project. You understand how the site works and could be tempted to guide users in the right direction. You are emotionally invested in the project and it is hard for you to remain impartial. A good facilitator should always remain calm. He or she needs to be patient and a good listener, capable of drawing opinions from others.

Unfortunately, not everybody is capable of that. I myself make a terrible facilitator. I talk far too much and get frustrated when users fail to ‘get it.’ Ask yourself, are you the right person for the job and if not who is?

The Facilitator’s Responsibilities
Whoever your facilitator is they need to understand the role. The position has three responsibilities.

  1. To encourage the tester to communicate. Many users sit in silence struggling with a site unless encouraged to speak. The facilitator should challenge users to think aloud. Ask open ended questions that cannot be answered with yes or no. Constantly ask what they are thinking and question their choices. A facilitator should always be calm, patient and a
    good listener.
  2. To Intervene When Necessary.
    There is a perception that the facilitator should never intervene to help a struggling tester. However, this is ultimately counter-productive. If a user becomes completely stuck, show them how to continue. This provides an opportunity to discuss why the way forward was not obvious and allows a chance to discover other issues deeper in the site.
  3. To Lead the Tester Through the Usability Script.
    It always pays to prepare a usability script beforehand. This outlines what you intend to cover. It is the role of the facilitator to guide the tester through this script.

Work from a Script

The answer of what goes into a usability script largely depends on what you are testing. If it is a design concept then your testing will be limited to questions about the navigation and communication of core messages. You could also carry out some flash testing (as
discussed in chapter 4) but beyond that your options are limited. However, testing against a wireframe or early versions of the site allows users to complete key tasks. For example, users could be asked to find the price of a particular product or the
contact details for a key member of staff. The choice of task should be based on activities that your personas would wish to complete. Think back to the persona we created in chapter 2 (Stress free planning). Jane was considering a visit to a health spa. We established that the first two pieces of information Jane wanted was price and availability. Any user testing for the spa should therefore
include tasks to find this information. Although what is tested will vary depending on the project and stage of development, some information that should ways be included. Below are highlights from a fictional transcript demonstrating the information that should always be covered.

Example Script

“Hi Jane. My name is Paul and I am going to be
running our session today. Joining me is Pete. I have
asked him along to take some notes as we talk. I hope
that is okay.”

By introducing yourself and others in the room you help to put the user at ease. Offering coffee can help too! Be sure to explain any recording equipment as this can also be intimidating.

“We are here to improve a website that is currently under development. You are going to help us test the site. Its important you understand, we are testing the site and not you. So you can relax!”

By explaining to the user that you are testing the site and not them, they will behave more naturally.

“You do not need to worry about messing up. There are no right or wrong answers in a usability test. What we do need is complete honesty. If you are
struggling with something or do not like the way it works, say so. You aren’t going to offend anybody.”

If the user perceives the session as a test
with right and wrong answers, they will tell you what they think is correct rather than what they feel.

“The most important thing to remember is that we need you to explain what you are thinking. Think out loud and talk about the options you are considering. Before you click on a link explain what other options you considered and why you picked that one.”

Getting the user to articulate their thoughts is fundamental to the success of the session. This cannot be stressed enough. Even though you explain the need up front, you will still need to prompt the user throughout the session.

“Finally, if you have any questions please feel free to ask. I might not be able to answer them straight away as I could prejudice the testing. However, I will answer them at the end.”

It is important to explain before you begin why you may not answer their questions during the session. However, if they do ask questions be sure to address them at the end.

“Let’s start off with something easy. Can you tell me a bit about yourself? Tell me about your job? Begin a session with simple questions such as family status, age and job title.”

This helps build the users confidence and provides some useful background information.

“Tell me a bit about your computer experience. How confident do you feel using a PC? Do you use them for work? What about at home? How much do you use the internet? What kind of sites do you use most and find useful?”

Building up an understanding of the users computer and web experience is useful context for the session. It also indicates how representative they are of the target audience.

“Lets talk about the site. It is for a health spa. Before I show it I want to ask about your expectations. What do you think a health spa website should look like and what information would it contain?”

It is often helpful before showing a site to users to ask them about their expectations. If the expectations do not meet the reality it can cause confusion. Also it provides the opportunity to find out more about what users want from the site.

After this generic introduction the session would become more specific. The choice of direction is dependent on the material available to test against and the type of site. However this specific testing should either fall into the category of ‘do they understand what they are seeing’ or task completion.

Cover of Paul Boag's book This extract is taken from Website Owner’s Manual, written by Paul Boag and published by Manning Publications.

Vitamin readers can enjoy 40% off the full price of the book until April 1st 2009. Just use the code ‘thivita40′.

Main image courtesy of (nz)dave.

Treehouse

Our mission is to bring affordable Technology education to people everywhere, in order to help them achieve their dreams and change the world.

Comments

34 comments on “Bargain Basement Usability Testing

  1. A great write up Paul, I am really looking forward to your book.I have yet to run into many clients who think a usability test is worthwhile. The budget is usually not big enough for it and most often they are resistant because the web designer is “supposed to know” how a site should be built. I find this is the biggest hurdle, in a way you have to admit that you don't know what the site needs.

  2. A great write up Paul, I am really looking forward to your book.

    I have yet to run into many clients who think a usability test is worthwhile. The budget is usually not big enough for it and most often they are resistant because the web designer is “supposed to know” how a site should be built. I find this is the biggest hurdle, in a way you have to admit that you don't know what the site needs.

  3. PaulGreat article. I'm building my own desktop app and usability is going to be a really key factor in my success or failure, particularly because my app will have very limited user interface.I heard Paul Buchheit, the author of GMail and creator of FriendFeed tell an audience in a presentation that you should aim for 100 happy users. He was talking more about the feature set of the product than usability testing, but I find it interesting that if you have 100 people who are happy with the product, this means you have most bases covered, as people just aren't that different from one another.Thanks again for a really useful article on testing – I'll certainly be applying this stuff to the usability testing for my app.

  4. Paul

    Great article. I'm building my own desktop app and usability is going to be a really key factor in my success or failure, particularly because my app will have very limited user interface.

    I heard Paul Buchheit, the author of GMail and creator of FriendFeed tell an audience in a presentation that you should aim for 100 happy users. He was talking more about the feature set of the product than usability testing, but I find it interesting that if you have 100 people who are happy with the product, this means you have most bases covered, as people just aren't that different from one another.

    Thanks again for a really useful article on testing – I'll certainly be applying this stuff to the usability testing for my app.

  5. I think usabilty testing is a must for every product to launch,be it a commercial product or just a website, in the long run, benefit will still goes for the producer and the consumers as well. Thanks for sharing this great article.Tiffany Inspired

  6. I think usabilty testing is a must for every product to launch,be it a commercial product or just a website, in the long run, benefit will still goes for the producer and the consumers as well.
    Thanks for sharing this great article.

    Tiffany Inspired

  7. Great article. I'm building my own desktop app and usability is going to be a really key factor in my success or failure, particularly because my app will have very limited user interface./Unique Picture frames

  8. Great article. I'm building my own desktop app and usability is going to be a really key factor in my success or failure, particularly because my app will have very limited user interface.
    /Unique Picture frames

  9. Usability is something I've wondered about when visiting some websites, like: who the heck designed this mess? Items that should be logical and well-placed are in need of time-wasting hide 'n seek discovery? Why waste my time?

  10. Usability is something I've wondered about when visiting some websites, like: who the heck designed this mess? Items that should be logical and well-placed are in need of time-wasting hide 'n seek discovery? Why waste my time?

  11. I have seen some interesting services online that offer usability testing with real users at very affordable prices – they may be worth looking into for some projects as well.

  12. Basements Planning and Design

    1. The first step toward any successful basement remodeling project is to come up with a
    plan. The decisions you make at this point will affect your entire project. You don’t want to have all the walls built, sanded, and primed only to realize you don’t have enough storage space!

    2. From getting started, to basement waterproofing and establishing a budget for this
    plan.

    3. Measure the available space. Decide how various areas will be used. Measuring most
    basements is relatively easy. With a little practice you can easily work out the total square footage of the space you have to work with. Having the square footage available is also very helpful when estimating the materials you will need.

  13. Usability is the bottom line to every website. Without testing the website is just a big random mess. Testing, testing, testing should be the goal of every company. Shame, Shame! To those creative design firms that outsource their web design work to others countries to save a buck. You are only adding to the mess of poorly designed websites.

    ~Jennifer Mahler