Block Four

Question 1

What is the difference between a usability goal and a user experience goal?

Answer 1

Usability tends to focus upon those aspects of an interactive product traditionally related to work, such as productivity and effectiveness. It also tends to focus explicitly on the user interface.

User experience relates to a wide set of concerns than those of usability, including issues such as whether an interactive product is fun to use, aesthetically pleasing and son on.

Question 2

Any kind of evaluation is guided either explicitly or implicitly by an evaluation approach. What is an evaluation approach?

Answer 2

An evaluation approach is based on a distinct set of values and assumptions as to how evaluation should be conducted, either implicitly or explicitly.

Question 3

What are the three main evaluation approaches?

Answer 3

The three main evaluation approaches are:

• usability testing
• field studies and
• analytical evaluation

Question 4

Which evaluation approach and associated methods would you use in the following situation:

You want to know whether a system, which you have designed, implemented and installed in particular organisation, is being used in the way you expected.

Answer 4

Since you want to evaluate the user of the system in a naturalistic setting, you would probably run a field study, sing observational techniques, and perhaps also interviews and questionnaires.

Question 5

Which evaluation approach and associated methods would you use in the following situation:

You want to convince a manager that the changes you made to an accounting system were worth the money he had invested in order to make the changes.

Answer 5

Mangers are likely to be convinced by figures, so usability testing including user testing might be relevant here. For example, you could compare the time taken on a particular task before and after the changes were implemented.

Question 6

Which evaluation approach and associated methods would you use in the following situation:

You want to evaluate different groupings of options on pull-down menus.

Answer 6

Usability testing is the approach, with the associated methods of asking users (about which groping they prefer), observing users (as they explore the alternative groupings) and user testing (measuring their performance on specific tasks using the alternative groupings).

Question 7

List the steps of the DECIDE framework.

Answer 7

D - determine the overall goals that the evaluation address.
E - explore the specific questions to be answered.
C - choose the approach and methods to answer the questions.
I - identify the practical issues that must be addressed.
D - decide how to deal with the ethical issues.
E - evaluate, analyse, interpret and present the data.

Question 8

Describe the key characteristics of an evaluation approach.

Answer 8

• the role of users
• who controls the process and the relationship between evaluators and users during the evaluation
• the location of the evaluation
• when the evaluation is most useful
• the type of data collected and how it is analysed
• how the evaluation finding are fed back into the design process
• the philosophy and theory that underlies the evaluation approach.

Question 9

What is a performance measure?

Answer 9

A performance measure is a quantitative measure related to the user of an interactive product

Question 10

What types of user performance measures are taken during a user test?

Answer 10

• the time taken to complete a typical task or tasks
• the number of errors made while performing a task
• the number of steps taken to complete a task.

Question 11

Which data-gathering techniques are most commonly used in usability testing?

Answer 11

Direct observation, questionnaires, interviews and indirect observation through keystroke logging are the data-gathering techniques most often used in usability testing.

Question 12

Why are scripts useful for usability testing?

Answer 12

Scripts prepared in advance of a usability evaluation- and used for each participant -ensure that all participants are given the same information and are treated in the same way for the evaluation. This also ensures that it is possible to compare the result from participants.

Question 13

Why is it important to select a representative sample of users whenever possible?

Answer 13

It is important to have a representative sample to ensure that the findings of the user test can be generalized to the rest of the user population. Selecting participants according to clear objectives helps evaluators to avoid unwanted bias. For example, if 90% of the participants testing a product for 9-12 year-olds were 12, it would not be representative of the full age range. The results of the test would be distorted by the large group of users at the top end of the age range.

Question 14

Suggest the kind of information that you would expect to include in a ‘welcome’ script for a user testing session.

Answer 14

Your welcome script would include:

• the goal of the study and how the results will be used
• an outline of what the participant will be expected to do
• the data recording that will be used, including video cameras, keystroke logging etc
• an introduction to the environment including the presence of recording equipment and one-way mirrors
• a pointer to the informed consent form.

Question 15

What is the main purpose for conducting a field study evaluation?

Answer 15

Field studies are useful when you want to discover how products and prototypes will be used within their intended social and physical context of use.

Question 16

Briefly define quantitative data and qualitative data.

Answer 16

Quantitative data is in the form of numbers, or can easily and meaningfully be translated into numbers.
Qualitative data is difficult to measure, count or express in numerical terms.

Question 17

In what way(s) can question design in a questionnaire affect data analysis?

Answer 17

Open-ended questions are likely to produce a wide rage of answers that will need to be analysed qualitatively. Closed questions which offer respondents a finite set of possible answers can be analysed qualitatively or quantitatively.

Question 18

What is a key advantage of analytical approaches to evaluation?

Answer 18

Analytical evaluations are relatively quick to perform and do not require recruiting of users to take part.

Question 19

What is a key disadvantage of inspections?

Answer 19

A key disadvantage of inspections is that the expert who is role-playing a user can only make ‘guesses’ about how a real user will react to a product. Putting yourself in the shoes of another user is difficult.

Question 20

Several researchers and evaluators have developed their own heuristics to complement Nielsen’s original set. Why is this?

Answer 20

The original set of Nielsen’s heuristics was developed in the early 1990’s. Many new and varied interface types have come onto the market since then. Nielsen’ original heuristics are too general for evaluating these new products.

Question 21

What is a key benefit of pluralistic walkthroughs?

Answer 21

A key benefit is a strong focus on users task at a detailed level.

Question 22

What is GOMS?

Answer 22

An acronym standing for:

• goals,
• operators,
• methods and
• selection ruels.

It is a model which describes how a user performs a computer-based task in terms of goals, and the selection of methods and operations from memory that are needed to achieve them.

Question 23

When is GOMS used for evaluation?

Answer 23

GOMS is useful in helping make decisions about the effectiveness of new products. It is not often used for evaluation purposes.

Question 24

What is the keystroke level model?

Answer 24

It is a model which provides actual numerical predictions of user performance. The model encompasses a standard set of approximate times for the main kinds of operators used during a task, which are used to predict the time it takes to execute a given task.

Question 25

When is the keystroke level model useful for evaluation?

Answer 25

It is useful for comparing the time it takes to perform a task using different strategies.

Question 26

What is Fitts’ Law?

Answer 26

Fitts’ Law predicts the time it takes to reach a target using a pointing device. In interaction design it is used to describe the time it takes to use a mouse and other input devices to click on objects on a screen.

Question 27

What is Fitts’ Law useful for evaluation?

Answer 27

It is useful for evaluating systems where the time to physically locate an object is critical to the task at hand.