Block 4 - Unit 2: Usability testing and field studies

Question 1

Usability testing. (4)

Answer 1

Involves users.

Emphasises ‘usable’ property - product is being tested rather than user.

(Often) controlled environment; performance of users on pre-planned tasks is repeatedly measured.

Goal - test if product is usable by intended user population to achieve the tasks designed for.

Question 2

Key components of usability testing. (2)

Answer 2

User test.

User satisfaction questionnaire.

Question 3

User testing. (Include examples)

Answer 3

Measures human performance on specific tasks.

Eg. reading different typefaces, navigation of menu types, info searching.

Logging of mouse / keyboard and video used to record performance.

Question 4

User satisfaction questionnaire.

Answer 4

How do users feel about using the product - rating along different scales, after interaction.

Question 5

Measures used in usability testing.

Answer 5

Time and number:

• time to complete a task (after a specified time away);
• Number of errors per task or per time unit;
• Number of views of online help / manuals;
• Number of users making a particular error;
• Number of users completing a task successfully.

Question 6

Number of users needed in usability testing.

Answer 6

5 - 12 considered acceptable.

Can use less if lower budget / schedule, or for quick feedback on eg. logo placement - 2 or 3 users.

Question 7

Remote usability testing (including advantage).

Answer 7

Users perform tasks in their own setting, logged remotely.

Advantage - many users tested at once, and logged data automatically compiled into statistical packages for analysis. Eg. no of clicks per page.

Question 8

Usability testing (UB)

Answer 8

Doesn’t have to result in accurate measures of user performance, or be under controlled conditions, but does involve some form of performance assessment and a level of evaluator control, most commonly the use of pre-set tasks.

Several methods may be used - eg. indirect observation, user test to measure performance, questionnaire.

User tests often conducted along experimental lines, involving hypothesis testing and statistics.

Question 9

Doing user testing. (Intro)

Answer 9

How to do user testing is peculiar to usability testing.

Controlling test conditions is central - careful planning; ensure all material is prepared, conditions are the same for each user, what is being measured is indicative of what is being tested and that assumptions are made explicit in the test design.

Question 10

Elements of doing user testing. (4)

Answer 10

Design typical tasks.

Select typical users.

Prepare testing conditions.

Plan how to run the tests.

Question 11

Design typical tasks (user testing).

Answer 11

Appropriate task to test users’ performance is critical.

‘Completion’ tasks set, eg. find website, create spreadsheet.

Quantitative performance measures obtained.

User tests most suitable for hi-fi prototypes, simulations and working products.
Task type depends on system type and evaluation goals / questions.
Eg. whether paper-prototypes, simulation or limited part of system’s functionality will influence breadth and complexity of tasks set.

Generally, tasks are 5 - 10 mins and designed to probe a problem.
Often straightforward, but occasionally more complex, eg. join an online community or solve a problem.

Question 12

Select typical users (user testing)

Answer 12

What characteristics?

Some products aimed at eg. old / young, novice / expert.

Equal male / female split, unless aimed at one.

Previous experience with similar systems?

If user population is large - short questionnaire can help identify testers.

Question 13

Prepare testing conditions (user testing)

Answer 13

Environment controlled to prevent unwanted influences and noise that distorts results.

Question 14

Plan how to run the tests (user testing).

Answer 14

Schedule an script, equipment tested, pilot test.

Start with easy task to build confidence and familiarisation.

Avoid long tasks; keep session under 1 hour.

Don’t create too much data to analyse.

Question 15

Potential sources of bias (user testing)

Answer 15

Users:
if don’t match profile, may get misleading results.

Evaluation tasks:
If users get unintentional clues from tasks, this may prevent exposing UPs.

Test setting:
Lab is artificial, and may need conditions adding to more closely simulate actual environment of use.

Evaluator / observer bias:
Behaviour may be altered when being watched. Observer may forget they shouldn’t ask leading questions, or help if user gets stuck.

Methodology:
‘Think-aloud’ alters behaviour.
Avoid ‘order effects’ - trying one design may give ‘practice’ to trying another. (So, counterbalance).

Reporting / analysis:
Data analysis and interpretation will be undertaken based on evaluator’s knowledge and experience - so biased to some extent.

Question 16

Field studies (intro)

Answer 16

Typically used to find how a product / prototype is adopted an used in people’s working / everyday lives.

Such settings are ‘messy’ - activities overlap and constantly get interrupted.

The way people interact with products is often different than in a laboratory setting; a better sense of the success of a product is gained in the real world.

Question 17

Trade-off of field studies.

Answer 17

Can’t test specific hypothesis about an interface or account, with the same degree of certainty, for how people react or use a product (than in a lab).

So, harder to determine what causes certain behaviour or what is problematic about the usability of a product.

Instead, qualitative accounts and descriptions of people’s behaviour and activities are obtained to reveal how they used the product and reacted to the design.

Question 18

Some characteristics of field studies. (3)

Answer 18

Can be minutes, months or years.

Primary data collection - observing and interviewing people; video, audio and field notes.

May ask for paper or electronic diaries to be filled in at points in the day - when interrupted, encounter a problem or are in a particular location.

Question 19

Things to do for field studies. (4)

Answer 19

Important to inform people of the length of study / session.

Need to agree part of site to be recorded and how.

Need to set up cameras unobtrusively, and if eg. in someone’s home - how to turn on / off.

What happens if product breaks?

Question 20

Planning an observational evaluation (intro).

Answer 20

Expand DECIDE to present a procedural approach to ‘doing’ observational evaluation.

Primary focus - evaluation of low-fi prototypes (especially card-based and interface sketches).

Question 21

Kinds of task to consider (‘I’)

Answer 21

Core tasks (frequent)

Those very important to users or the business.

Those that have some new design features / functionality.

Critical tasks.

Ones you feel have to be validated with users for greater clarity and understanding of the design team.

Those that scenarios were based on, which you used for developing the conceptual design of the UI.

Question 22

How to choose tasks.

Answer 22

Choose tasks that help in validating usability or UX goals, or that focus on any particular design features you want to assess - metaphor suitability, task flow, icon design, etc.
But, depends on prototype as to what tasks you can evaluate.

Question 23

Other materials for evaluation. (9)

Answer 23

1. Briefings for evaluators.
2. Introductory material for participant.
3. Background info for participant.
4. Pre-session questionnaire or an interview plan.
5. Permission to gather and keep data.
6. Task cards.
7. Data collection forms.
8. Post-session interview plan or questionnaire.
9. Data analysis, interpretation and recommendation forms.

Question 24

First steps before evaluation session.

Answer 24

Pilot study.

Revising any procedures and materials as required.

Question 25

Steps in an observational evaluation session.

Answer 25

Greet user, brief them about the study / equipment and obtain informed consent.

Conduct pre-session questionnaire / interview.

Show prototypes. If low-fi, explain their rough nature.

Give first task; check understanding.

Give each of the other tasks.

Observe while each task is attempted - what they do, or say if ‘think-aloud’. Record observations on paper / data collection forms.

Explore as appropriate other concerns / aspect of design - suitability of metaphor, task flow, icon design, structure, layout, tools, etc.

Conduct post-session questionnaires / interviews (if appropriate).

Thank user, and de-brief with post-evaluation discussion.
If others present, have group discussion - gives people chance to share their impressions with designers.

If evaluation is done with colleagues, can be useful to compare notes.

Question 26

What are paper-prototypes good for exploring? (6)

Answer 26

Concepts and terminology.

Navigation, work and task flow.

Content.

Documentation / help.

Requirements / functionality.

Interface layout.

Question 27

Comments on script for evaluating paper-prototypes.

Answer 27

Add any questions thought of to script.

Where possible cluster questions around relevant task, so you’re exploring at one time as many of the issues related to the task being attempted as you can.

For aspects (eg. UX goals) that may not cluster - ask separately.

Take care not to ask leading questions - we want users’ own opinions.

Question 28

Quantitative and qualitative data (analysis)

Answer 28

Quantitative - data in the form of numbers, or easily translated to numbers.

Qualitative - data is difficult to measure, count or express in numerical terms in a sensible fashion.

Fallacy that certain forms of data gathering will only result in quantitative data and others in qualitative - all forms previously discussed can result in either.

Quantitative analysis users numerical methods to ascertain magnitude, amount or size of something.

Qualitative analysis - nature of something, represented by themes, patterns an stories.

Question 29

Use and abuse of numbers.

Answer 29

Often unwarranted to turn qualitative data into numbers to manipulate and interpret with respect to goals, although people tend to believe numbers offer strong / clear conclusion.

Better to use % for 10+ only, but still better to use raw numbers to make understanding clear.

Question 30

Initial processing of interview data.

Answer 30

Raw data - audio or notes.

Notes - write up and expand while memory fresh.
Audio can help, or may be transcribed (large effort - maybe just relevant parts).

Closed questions - quantitative analysis, eg. % in age range.

Open questions - qualitative analysis.

Question 31

Initial processing of questionnaires.

Answer 31

Raw data - written responses, or electronic records.

May need to ‘clean’ eg. misunderstood question responses.

Data can be filtered by sub-populations, or by questions (eg. to understand reactions to colour).
Allows analyses to be conducted on subsets of data - can draw detailed conclusions for more specific goals. (May use tools, eg. spreadsheets).

Closed questions - quantitative, open - qualitative.

Question 32

Initial processing of observations.

Answer 32

Raw data - wide variety: notes, photos, data logs, think-aloud, video / audio.

Rich picture, but hard to analyse without structured framework.

Initial processing - write / expand notes, transcription.
In controlled setting - sync different data recordings.

Transcriptions / note - qualitative
Photos - contextual info.
Logs and some elements of notes - quantitative.

Patterns / themes emerge - note initial impressions for further analysis, but don’t rely on these - you may be biased by them.

Question 33

Simple quantitative analysis.

Answer 33

Type of average chosen can change meaning of results.

Before analysis - data needs collating into data sets.
Can usually be put in rows and columns - eg. spreadsheet for easy manipulations and data set filtering.

Question 34

Example of how the question affects data analysis.

Answer 34

“How do you feel about Cybelle?” - responses will be treated qualitatively, which is hard with many responses.

Instead, could use a closed question, eg. “Is Cybellee amusing or irritating?” (or neither).
Reduces options and can give % for each.

Or, use Likert scale: “Cybelle is amusing” - strongly agree, etc.

Question 35

Advantages of spreadsheets.

Answer 35

Commonly available and understood.

Offers a variety of numerical manipulations and graphs - good for overall view.

Question 36

How graphs are helpful.

Answer 36

Can help identify outliers in error rate data - may be removed to avoid distortion, but worth investigating why.

Fairly straighforward to compare 2 sets of results using graphs.

Question 37

3 main groups of methods for summarising quantitative data.

Answer 37

Tabulations, charts and rankings - visual representations.

Descriptive statistics - eg. mean, median and mode.

Inferential statistics - results based on tests of statistical significance that give the probability that a claim arising from data can be applied to your user population as a whole.

Question 38

Comment on size and frequency of evaluation tests.

Answer 38

Mostly looking for info to help decide what to do next with design - small, frequent tests are more helpful than a single large-scale technique.

But, small samples don’t support maths of inferential statistics - not enough data for a statistical analysis that supports claims - avoid eg. ‘significant result’.

Question 39

Outcome of data analysis?

Answer 39

List of any usability problems (defects) found during evaluation.

Presence of usability defects indicate an interactive product’s usability goals and UX goals are not met.
They can lead to confusion, error, delay or failure to do a task.

Question 40

Characteristics of usability defects.

Answer 40

Irritates / confuses.

System hard to install, learn or use.

Mental overload.

Poor user performance.

Violates design standards or guidelines.

Reduces trust / credibility of system.

Tend to cause repeated errors.

Question 41

Simple qualitative analysis - first steps

Answer 41

(As with quantitative) first step is to gain an overall impression of the data and look for patterns.

Some patterns emerge during data gathering, but need to confirm and re-confirm findings to make sure initial impressions are not biasing analysis.

Using a framework for observational data helps notice patterns, by asking eg. “Who is present?”, “What is happening?”, “Where?”.

Question 42

3 types of qualitative analysis (M364).

Answer 42

Identify recurring patterns / themes.

Categorising data.

Analysing critical incidents.

Question 43

What aspects do patterns / themes relate to?

Answer 43

Behaviour, user group, places / situations where certain events happen, etc.

Each may be relevant to goals (ie. a main theme)

Question 44

How can problems found in analysis of qualitative data be grouped?

Answer 44

Categorised according to incidents, people looking puzzled, or don’t know what to do.

According to patterns - common and differences in data from users, eg. ‘All liked ……’.

Ordering according to severity of problem.

Dividing into lists - eg. whether you know the cause.

Order according to the level of difficulty in fixing a defect.

Question 45

Identifying causes of usability defects.

Answer 45

To identify causes of defects, need to look in more depth at the various sources of evaluation data collected (video, notes, etc.).

Process of reviewing and summarising evaluation data usually makes it obvious which problems require most urgent attention.

Question 46

Severity rating.

Answer 46

A measure given to a usability defect to indicate the criticality of its impact on the usability of the UI design.
Eg. 0 - 4

Prioritisation can be used to distribute resources - time, money, design effort - to required changes.

Question 47

Points recommendations for change may cover. (3)

Answer 47

Successes to build on.

Defects to fix.

Possible defects or successes not proven - evaluation data doesn’t have enough evidence to decide. Requires further evaluation.

Question 48

Categorising data - transcripts

Answer 48

Transcripts can be analysed at a high level of detail (stories, themes), or a fine level (each word, gesture, etc, analysed).

Either way elements are usually categorised first using a categorisation scheme.

Question 49

Categorisation challenges.

Answer 49

Categories used are largely determined by study goal.
Challenges:
- determine meaningful categories that are othogonal (don’t overlap).
- decide appropriate granularity (word, phrase, paragraph) - depends on goal and data.

Question 50

Content analysis.

Answer 50

Typically involves categorisation of data, then studying frequency of category occurrences.

Question 51

Discourse analysis (for transcripts).

Answer 51

Focuses on dialog - meaning of what is said, and how words convey meaning.

Strongly interpretive; attention to context.

How people use language to construct versions of their worlds.

Question 52

Conversational analysis.

Answer 52

Very fine grained form of discourse analysis.

Semantics of the discourse are examined in fine detail, and focus is on how conversations are conducted.

Question 53

Critical incident technique?

Answer 53

Flexible set of principles. 2 basic principles:

a) Reporting facts regarding behaviour is preferable to the collection of interpretations, ratings and opinions based on general impressions.
b) Reporting should be limited to those behaviours which, according to competent observers, make a significant contribution to the activity.

Question 54

ID and critical incident technique.

Answer 54

In ID context, use of well-planned observation sessions satisfies principle (a);
(b) refers to critical incidents - significant or pivotal to the activity being observed, in either a desirable or undesirable way.

In ID, the main focus is to identify specific significant incidents, then focus on these and analyse in detail, using the rest of the data as the context to inform their interpretation.
May be identified by:
- User - by retrospective discussion of recent event.
- Observer - by studying video; observation in real time.
Eg. when observer is obviously stuck.