Block 4 Review Questions

Question 1

Why would you want to observe users later on in the interaction design lifecycle?

Answer 1

Early in design, observation helps designers to understand users’ needs. Later in design, observation is used to assess whether the prototype of the product meets users’ needs. Observing users is useful at any time during product development.

Question 2

Describe three types of observation. Why and when might each be used for evaluating an interactive product?

Answer 2

1. direct observation in the field would be most appropriate to evaluate a highfidelity prototype or an existing product that could be used in the target context.
2. direct observation in a controlled environment would be most appropriate to study specific aspects of a product.
3. indirect observation most appropriate to evaluate a high-fidelity prototype or an existing product.

Question 3

What does it mean to use the think-aloud technique for gathering data?

Answer 3

The think-aloud technique involves the participant saying out loud everything they are thinking and trying to do while using a product.

Question 4

What is a performance measure? What types of user performance measures are taken during a user test?

Answer 4

A performance measure is a quantitative measure related to the use of an interactive product. The following are example performance measures taken during a user test: the time taken to complete a typical task or tasks, the number of errors made while performing a task and the number of steps taken to complete a task

Question 5

Which data-gathering techniques are most commonly used in usability testing?

Answer 5

Direct observation, questionnaires, interviews and indirect observation through keystroke logging are the data-gathering techniques most often used in usability testing.

Question 6

Why are scripts useful for usability testing?

Answer 6

Scripts prepared in advance of a usability evaluation – and used for each participant – ensure that all participants are given the same information and are treated in the same way for the evaluation. This also ensures that it is possible to compare the results from participants. This was discussed in Section 14.2.2 of the Set Book, in the context of the MedlinePlus case study.

Question 7

Why is it important to select a representative sample of users whenever possible?

Answer 7

It is important to have a representative sample to ensure that the findings of the user test can be generalised to the rest of the user population. Selecting participants according to clear objectives helps evaluators to
avoid unwanted bias. For example, if 90% of the participants testing a product for 9–12 year-olds were 12, it would not be representative of the full age range. The results of the test would be distorted by the large group of users at the top end of the age range

Question 8

Suggest the kind of information that you would expect to include in a ‘welcome’ script for a user testing session. You might like to review Figure 14.9 on page 654 of the Set Book to help you answer this

Answer 8

Your welcome script would include:
the goal of the study and how the results will be used;
an outline of what the participant will be expected to do (but not in detail as this will come later);
the data recording that will be used, including video cameras, keystroke logging, etc.;
an introduction to the environment including the presence of recording equipment and one-way mirrors;
and a pointer to the informed consent form.

Question 9

Describe the differences between experiments and usability testing as discussed by Ben Shneiderman in his interview..

Answer 9

Experiments

• Experiments test a hypothesis that has practical implications and theoretical results.
• The results are of broad importance.
• At least two conditions are tested.
• Statistical tests are applied to the results to verify statistically significant differences.
• The results confirm or refute the hypothesis being tested.
• The experimental procedure is carefully described so that others can replicate it.

Usability test studies

• Usability test studies involve only small numbers of users who carry out required tasks.
• Statistical analysis of the results is less important.
• The goal is to refine a product as quickly as possible.
• The outcome is a report to developers which identifies frequent problems and possibly suggests improvements.

Question 10

What is the main purpose for conducting a field study evaluation?

Answer 10

Field studies are useful when you want to discover how products and prototypes will be used within their intended social and physical context of use.

Question 11

What evaluation methods are used in a field study evaluation?

Answer 11

Field study evaluations involve observing users and asking users (see Table 12.2 on page 596 of the Set Book).

Question 12

One of the example field studies provided in Chapter 12 of the Set Book is HutchWorld (Section 12.4.4, starting on page 605). For this case study, outline when and how field studies were used and what results were obtained.

Answer 12

Developers started by interviewing potential users and then observing their daily activity in the clinic and the hospital (page 606). After developing a prototype design they conducted further observations and focused on how and when patients used the system, how they interacted with it and what problems they faced (page 607). From these observations the evaluators realised that the user community was small so there were never enough people in the chat room for successful communication, and that many patients preferred asynchronous communication.

Question 13

What are the three steps in data analysis for evaluation? (These were introduced in Section 3.1.5 of Unit 1 of this block.)

Answer 13

The three steps in data analysis for evaluation are:.
• collating the data
• summarising the data
• reviewing the data.

Question 14

Briefly define quantitative data and qualitative data

Answer 14

Quantitative data is in the form of numbers, or can easily and meaningfully be translated into numbers. Qualitative data is difficult to measure, count or express in numerical terms.

Question 15

In what way(s) can question design in a questionnaire affect data analysis?

Answer 15

Open-ended questions are likely to produce a wide range of answers that will need to be analysed qualitatively. Closed questions which offer respondents a finite set of possible answers can be analysed qualitatively or quantitatively.

Question 16

Name and define the three different kinds of ‘average’? These were also introduced in Box 4 of Unit 1 in this block.

Answer 16

The three different kinds of ‘average’ are:

Mean
commonly known type of average where the figures are added and then divided by the number of figures you started with

Median
is the middle value of the figures when they are ranked.
mode. most commonly occurring figure in the set.

Mode
most commonly occurring figure in the set.

Question 17

What is a key advantage of analytical approaches to evaluation?

Answer 17

Analytical evaluations are relatively quick to perform and do not require recruiting of users to take part.

Question 18

What is a key disadvantage of inspections?

Answer 18

A key disadvantage of inspections is that the expert who is role-playing a user can only make ‘guesses’ about how a real user will react to a product. Putting yourself in the shoes of another user is difficult.

Question 19

Several researchers and evaluators have developed their own heuristics to complement Nielsen’s original set (listed on pages686–7of the Set Book). Why is this?

Answer 19

The original set of Nielsen’s heuristics was developed in the early 1990s. From your reading of Chapter 6 in the Set Book, you’ll have seen that many new and varied interface types have come onto the market since then. Nielsen’s original heuristics are too general for evaluating these new products.

Question 20

Both user observation and heuristic evaluation are undertaken to find what problems users may have when using an interactive product. Using the following criteria, compare these two techniques

Answer 20

• Involvement of users

In user observations, participants are asked to attempt tasks with the interface and to comment from their own point of view. In a heuristic evaluation, experts inspect the user interface while role-playing as typical users. They assess the user interface against each heuristic individually. The heuristics are meant to guide the expert in finding problems.

• Who is ‘observed’?

In user observations, the evaluator observes what the participant does. In a heuristic inspection, inspectors ‘observe’ themselves – noticing where typical users might look, and trying to predict what a typical user might do.

• Who answers the questions?

In user observations, participants may be asked to describe their thoughts in a think-aloud or retrospective protocol. In heuristic inspections, the inspectors asks themselves questions such as ‘Is this a simple and natural dialogue?’

• What are the outcomes?

In user observations, the key measurement is often effectiveness: ‘Can the user do this task?’ In heuristic inspection, the key measurement is: ‘How many violations of heuristics are there in this user interface?’

Question 21

What is a key benefit of pluralistic walkthroughs?

Answer 21

A key benefit is a strong focus on users’ tasks at a detailed level.

Question 22

What is GOMS? When is GOMS used for evaluation?

Answer 22

GOMS (an acronym standing for goals, operators, methods and selection rules) is a model which describes how a user performs a computer-based task in terms of goals, and the selection of methods and operations from memory that are needed to achieve them. GOMS is useful in helping make decisions about the effectiveness of new products. It is not often used for evaluation purposes.

Question 23

What is the keystroke level model? When is it useful for evaluation?

Answer 23

The keystroke level model is a model which provides actual numerical predictions of user performance. The model encompasses a standard set of approximate times for the main kinds of operators used during a task, which are used to predict the time it takes to execute a given task. The keystroke level model is useful for comparing the time it takes to perform a task using different strategies.

Question 24

What is Fitts’ Law? When is it useful for evaluation?

Answer 24

Fitts’ Law predicts the time it takes to reach a target using a pointing device. In interaction design it is used to describe the time it takes to use a mouse and other input devices to click on objects on a screen. Fitts’ Law is useful for evaluating systems where the time to physically locate an object is critical to the task at hand.