L7- Evaluating interactive systems

Question 1

What is the difference between formative and summative evaluation

Answer 1

Formative evaluation is used in the early stages of a project to compare, assess and refine design ideas.
Formative evaluation often involves OPEN research questions where the researcher is interested in learning further information that may inform the design
Summative evaluation is more likely to be used in the later stages of a project and involve CLOSED research questions with the purpose of testing and evaluating systems according to predefined criteria

Question 2

What’s the difference between analytical and empirical evaluation methods

Answer 2

Analytical : based on applying a theory to analysis and discussion of the design, in the absence of real world users
Empirical : making observations and measurements of users

Question 3

What’s the difference between quantitative and qualitative evaluation

Answer 3

Numbers versus words/pictures/audio/video

Question 4

Why are analytical methods useful for formative evaluation?

Answer 4

Analytical methods are useful for formative evaluation, because if the system design has not yet been completed, it may be difficult to observe how it is used (although low fidelity prototypes can be helpful here)

Question 5

Give some examples of qualitative evaluation methods

Answer 5

Qualitative analytic methods include cognitive walkthrough (useful for closed research questions), and the cognitive dimensions of notations framework (useful for open research question).

Question 6

Give examples of quantitative analytic methods

Answer 6

The Keystroke Level Model is a quantitative analytic method, which can be used to create numerical comparisons of closed research questions.

Question 7

Give examples of qualitative empirical methods

Answer 7

think-aloud, interviews, and field observation –> ethnographic approaches
They are usually associated with open research questions, where the objective is to learn new information relevant to system design or use

Question 8

Give examples of quantitative empirical methods

Answer 8

Quantitative empirical methods generally require a working system, so are most often summative
Examples include the use of analytics and metrics in A/B experiments, and also controlled laboratory trials

Question 9

Explain how to run RCTs

Answer 9

Decide on a performance measure
Find a representative sample of the target population (who have given informed consent to participate)
Find an experimental task that can be used to collect performance data

Question 10

How might we measure the results of an RCT?

Answer 10

Effect size – impact on the mean performance
Measure correlation with factors that might improve performance
Report significance measures to check whether the observed effects might have resulted from random variation or other factors than the treatment

Question 11

What problems are associated with RCTs?

Answer 11

Overcoming natural variation needs large samples
RCTs don’t provide understanding of why a change occurred
This means that it is hard to know whether the effect will generalise (for example to commercial contexts)
If there are many relevant variables that are orthogonal to each other, such as different product features or design options, many separate experiments might therefore be required to distinguish between their effects and interactions
Thus RCTs aren’t often used for design research in commercial products
A more justifiable performance measure is profit maximisation, but sales/profit are often hard to measure with useful latency
Companies therefore tend to use PROXY MEASURES such as the number days that customers continue actively to use the product

Question 12

What is internal validity?

Answer 12

What the study done right
Reproducibility
Scientific integrity
Refutability

Question 13

What is external validity?

Answer 13

Does the study tell us useful things
Focussing on whether the results can be generalisable to real world situation, including factors such as representativeness of the sample population, the experimental task and the application context

Question 14

Describe two ways of analysing qualitative data

Answer 14

While we can use statistical comparison of quantitative measures from controlled experiments; interviews and field studies require analysis of qualitative data
Qualitative data is often recorded and transcribed as written text, so the analysis can proceed using a reproducible scientific method

Question 15

What is categorical coding, explain how to do it.

Answer 15

Categorical coding is a qualitative data analysis method that can be used to answer ‘closed’ questions, for example, comparing different groups of people or users of different products
The first step is to create a “coding frame” of expected categories of interest
The text data is then segmented (for example on phrase boundaries)
Each segment is assigned to one category, so that frequency and correspondence can be compared
In a scientific context, categorical coding should incorporate some assessment of inter-rater reliability, where two or more people make the coding decisions independently to avoid systematic bias or misinterpretation
Compare how many decisions agree relative to chance using a statistical measure such as Cohen’s Kappa for 2 people, or Fleiss’ Kappa for more and comparing to typical levels (0.6 - 0.8 is considered substantial agreement)
Inter-rater reliability may take account of how many decisions still disagreed after discussion, which may involve refining and iterating the coding frame to resolve decision criteria
It is often useful to ‘prototype’ the coding frame by having the independent raters discuss a sample before proceeding to code the main corpus

Question 16

What is grounded theory?

Answer 16

Qualitative data analysis method that can be used to explore open questions where there is no prior expectation or theoretical assumption of the insights that the researcher is looking for
First step: read the data closely, looking for interesting categories (‘open coding’)
The researcher then collects fragments, writing ‘memos’ to capture insights as they occur
Emerging themes are organised using axial coding' across different sources of evidence It is important to constantly compare memos, themes and findings to the original data in order to ensure that these can be objective justified The process ends when the theoretical description has reached saturation’ in relation to the original data, with the main themes complete and accounted for

Question 17

Explain how to get ethical clearance

Answer 17

Inform ethics committee before you collect any data or recruit any participants
Describe the study, who will participate, what you will ask them to do, what data you will collect
What precautions are being taken, as appropriate to the nature of the research, including the approach taken to informed consent, and whether participants will be anonymous

Question 18

What are three analytical evaluation options?

Answer 18

Cognitive walkthrough
KLM/GOMS
Cognitive Dimensions

Question 19

When would you use cognitive walkthrough?

Answer 19

Cognitive Walkthrough: Is normally used in formative contexts – if you do have a working system, then why aren’t you observing a real user, which is far more informative than simulating or imagining one? However, Cognitive Walkthrough can be a valuable time-saving precaution before user studies start, to fix blatant usability bugs.

Question 20

When would you use KLM/GOMS?

Answer 20

KLM/GOMS: It is unlikely that you’ll have alternative detailed UI designs in advance, so there is not much to be learned from using these methods in the context of a Part II project. If do you have a working system, a controlled observation is superior

Question 21

When would you use Cognitive Dimensions?

Answer 21

Is better suited to less structured tasks than Cognitive Walkthrough and KLM/GOMS, which rely on predefined user goal and task structure

Question 22

What empirical approaches could you choose from?

Answer 22

Interviews/ethnography
Think-aloud / wizard of oz
Controlled experiments

Question 23

When would you collect data using interviews/ethnography?

Answer 23

Useful in formative/preparation phase where an open research method is helpful in developing design ideas or capturing user requirements

Question 24

When would you use think-aloud/wizard of oz?

Answer 24

Valuable for both paper prototypes and working systems
Highly effective at uncovering usability bugs as long as the verbal protocol is analysed rigorously using qualitative methods

Question 25

When would you use controlled experiments?

Answer 25

Can help to establish the engineering aspects of the work
Important to ensure you can measure the important attributes in a meaningful way (with both internal and external validity)
need to test significance and report confidence interval of observed means and effect sizes

Question 26

When would you use surveys and informal questionnaires

Answer 26

Be clear what you are measuring
Is self reporting likely to be accurate
Use a mix of open questions, which capture richer qualitative information, and closed questions that make it easier to aggregate and test hypotheses
Open questions require a coding frame to structure and compare data, or grounded theory methods (if you have a broader research question)
Collecting survey data via interviews is likely to give more insight but questionnaires are faster so that you can collect data from a larger sample
Remember to test questionnaires with a pilot study as it’s easier to get them wrong than with interviews

Question 27

When would you use field testing

Answer 27

If a working product exists it may be possible to make a controlled release and collect data on how it is used
Make a risk assessment
Seek ethics approval before proceeding

Question 28

When would you use standardised survey instruments

Answer 28

These are standard psychometric instruments to evaluate mental states such as fatigue, stress, confusion and emotional state
There are also standard methods to assess individual differences (e.g. personality, intelligence)
Use standardised approaches wherever possible, so your results can be compared to existing scientific literature
Making changes to these standardised surveys generally invalidates the results

Question 29

What are some bad evaluation techniques?

Answer 29

Don’t use purely affective reports
Don’t ask a biased group – e.g. your friends – experimental demand
Don’t make claims that sound as though they result from a formative analytic process but are actually subjective
Don’t use introspective reports made by a single subject – might be biased and subjective

Question 30

How would you evaluate a non HCI project

Answer 30

Approach testing as a scientific exercise
Define goals and hypotheses and understand the boundaries and performance limits of your system by exploring them
Keep in mind that it’s often necessary to test to point of failure so that you can make comparisons or explain limits
For non-interactive projects, still necessary to decide whether evaluation should be analytic (proceeding by reasoning and argument, in which case you should ask how consistent and well-structured is your analytic framework) OR empirical (proceeding by measurement/observation, in which case you should ask what you are measuring and why, and ensure that you have achieved scientific validity, where the measurements are compatible with your claims).
All projects can include a mix of formative and summative evaluation
If you only evaluate formatively – did you finish your project?
If carrying out summative evaluation, be clear whether the evaluation criteria are internal (derived from some theory) or external (addressing some problem)
Need to establish objectivity of qualitative data (i.e. that it isn’t simply your own opinion).