L5 Efficient systems Flashcards
What are we trying to do when designing efficient systems?
concerned with measuring and optimising human performance through quantitative experimental methods
Why are probabilistic models useful for designing efficient systems?
Probabilistic models can be used to model human action, and also (in part) to predict human action.
Such models can be updated in real time, as demonstrated in predictive text entry systems, but they also allow efficiency of a user interface to be predicted, and also measured at design time.
What’s the fundamental tradeoff in human performance?
Work slowly and carefully, or quickly at the expense of making more mistakes
Speed versus accuracy tradeoff
Characterise as an information channel – fast and inaccurate actions result in more channel noise meaning that the information gain per unit time does not increase as quickly as the number of movements made
Explain Fitts’ Law
the time taken to point at something is proportional to the Distance to target, while inversely proportional to Width of target.
The ratio of width to distance is the index of difficulty, and can be understood as the potential amount of information gained by the system when the user points (selecting a small target from a wide range reflects greater information gain).
Time = k log (2D / W)
How can Fitts’ Law be used to design more efficient user interfaces?
- if we have an accurate prior expectation for the actions that a user is likely to take. “Semantic pointing” modifies the mapping of mouse motion to screen pixels, so that the effective width of more likely targets is increased, and the effective distance between them decreased.
What is the keystroke level model? What assumption is made?
The speed with which an expert user can complete a task in a user interface can be modelled as a series of unit operations - mouse movements and key strokes - with the KLM.
(Note that the user must be expert, because the model does not include learning time, errors, or reasoning about an unfamiliar task).
In KLM how do we estimate the time taken to point at a target, keypresses, homing?
Fitts’ Law
The time taken to point at a target is estimated using Fitts’ Law. Other components of the model include time taken to press a key or mouse button (about 200 ms), time taken to ‘home’ the hands on mouse or keyboard (about 400 ms), mental preparation time between sequences of more automated actions (about 1500 ms) and the time that the system takes to respond.
Describe GOMS
- Successors to the KLM approach have attempted to more accurately predict mental preparation time by using detailed cognitive models, for example in the GOMS (Goal/Operator/Method/Selection) technique
What is an A/B test
When companies are able to collect large volumes of behavioural data, for example in click-throughs from an online listing, it is possible to make randomised controlled trials by giving different users different versions of the interface, and observing which version is more likely to result in desired behaviour. This procedure is called an A/B test.
Explain how to statistically analyse the result of a controlled experiment to evaluate efficiency in a user interface
Controlled experiments to evaluate efficiency in a user interface often measure the completion times for a task, comparing the distribution of times for two or more versions of the user interface.
A statistical significance test is then carried out, to see whether the difference between the sample means might be due to chance.
The null hypothesis is that there is no difference in performance between the versions, and that the samples differ only as a result of random variation
A simple significance test such as the t-test compares the effect size (the difference between sample means) to the variance in the experimental data.
Experiments in HCI generally aim to minimise variance, and maximise effect size, to demonstrate that an improved user interface has resulted in better performance
What sorts of hypothesis tests are available?
The most straightforward tests for comparing sample distributions rely on the data following a normal or Gaussian distribution
If user performance does not vary so consistently, then a non-parametric statistical test such as the sign test can be used, to compare sets of matched samples.
A typical way to collected matched sample data is by carrying out a within-subject comparison, asking each experimental participant to complete an experimental task with both the original and the improved version of a user interface
For each participant, we note whether the sign is positive (improved version faster) or negative (original version faster), and then compare the proportion of each sign.
What sorts of factors have to be controlled for user studies?
Successful user studies rely on controlled experiments to minimise variation in the data from factors unrelated to the effect of the design change.
The include individual differences between subjects (e.g. IQ), errors or misunderstanding of the task,
distractions during the trial (e.g. sneezing),
motivation of the participant (e.g. time of day),
accidental intervention by experimenter (e.g. hints), and other random factors.
Difference in means should always be reported with confidence intervals or error bars.
Discuss whether a significant result is always interesting
a significant result is not always interesting - very small effects can be shown to be reliable, if the variance is small or the sample size very large.
HCI research for design applications usually focuses on large effect sizes, rather than statistical significance.
What are some drawbacks in experimental user studies?
Hawthorne effect
Taylorism
Discretionary use systems
Phenomenology
What is the Hawthorne Effect?
productivity seemed to improve if lighting was increased but also that productivity improved if lighting decreased
turned out that worker motivation and therefore productivity, improved at any time that an experiment was carried out, just because the workers liked the fact that someone was taking an interest in them
The same thing often happens with user interface modifications - an interesting design change may result in apparent efficiency improvements in the experimental context, but not have any long-term benefits.
This is especially likely to occur when the experimenter has a personal investment in the new design (perhaps because they designed it), and reveals this during the experiment.
