10: Experimental Design Flashcards
Stages in usability experiment
- identify objectives
- formulate hypothesis
- choose variables
- choose experimental design
- choose the tasks
- recruit participants
- run the experiments
- perform statistical test on data
- analyze and interpret results
- identify objectives
determine objective, goal of the experiment
- formulate hypothesis
- make a prediction about the outcome of the experiment
- must be testable
- should be presented in terms of independent, dependant variable
what makes a bad hypothesis
- vague
- hypothesis does not predict outcome - complex
- hypothesis able to explain any results
- choosing variables
- independent
- characteristics that is changed to produce different condition
- manipulated by researcher - dependent
- used to test hypothesis
- variable that is being measured
- outcome is driven by independent variable - control variables
- variable to be held constant to ensure validity
- depends on experimental design - confounding variable
- variable that can alter outcome of experiment
- not dependent
- choose experiment design
- specify what independent variables needs to be manipulated
- which dependent variables to measure
- ensure results are due to manipulation of independent variable only
- ensure that some variables are controlled to ensure validity
main types of experimental design
Between subjects (independent samples)
- participants are randomly grouped
- each group take part in only one condition
- compare one group with another
within subjects (repeated measures)
- same group of subjects take part in more than one condition
- compare each participant against himself
between subjects strengths
- no learning effect
- lower chance of participant having carry over effect
- lower possibility of gaining practice and experience
- lower chance of skewed results - less fatigue
- only subjected to one experiment - multiple variables can be tested simultaneously
- divide into groups
- each group test different condition
between subject limitations
- need many participants
- individual variability
- difference in abilities and expertise
- what if bad group of participants - assignment bias
- control and experimental group are from different population
within subject strengths
- need fewer participants
- less chance of variation between participants
- compares everyone with themselves(lower confounding variables)
within subject limitations
- carry over effect
- participation in one condition may affect performance in another
- confounding variables that vary with independent variables - fatigue effect
- tired
- negatively affect result
- give break - practice effect
- get used to experiment
- positively affect result - order effect
- outcome is due t order of experiment
- randomise order of experiment(counter balancing)
Matched experimental design
- participants matched in pairs
- pairs can be formed on gender, expertise, personal relationship
- each pair allocated one condition
- matching criteria may affect result
ladder of experimental validity
- content validity
- does result reflect variable of interest
- is the result interesting - construct validity
- does the result align with the theoretical concepts - internal validity
- does the hypothesis hold/ valid? - external validity
- does outcome hold for other data sets or sample users
- can it be generalised - ecological validity
- is the test meant for type of user? eg. developed for bling users tested on normal users
- choose the tasks
experimental tasks should be:
- constrained to test just the thing of interest
- need to collect data to analyse
- more data = less influence of outliers
writing experiment software
- finalise design before developing
- decide on best platform
- desktop
- mobile - test that it works on yourself
- pilot test on others
- run the experiment
- pilot the experiment
- test prototype
- fix problem if it occurs
- every participant run through should be the same - noise level
- information given
- task and training - explain what experiment will involve
- do not explain hypothesis - give standardise information
- get informed consent
- explain what experiment does
- participants sign to give consent - provide necessary training
- practice tasks to understand how it works
statistical test
compare data sets gathered after experiment and find statistical differences
Null hypothesis
H0: changes in behaviour due to chance
Alternate hypothesis
H1: hypothesis you are trying to demonstrate
if test results are significant, reject null hypothesis
choosing statistical test depends on:
- type of comparison
- difference
- correlation - type of data
- nominal/ categorical
- ordinal/ scale
- interval
- ratio
- parametric - number of conditions
- 1 independent variable can have 2 or 3 conditions
use parametric tests when
- data is interval/ ratio
- data is normally distributed
- data can be categorised by measure of central tendency
between subjects
- independent T test
within subjects
- paired T test
procedure
- test statistic t calculated
- significance value based on t calculated
- descriptive statistic calculated
use non parametric test when:
- data is ordinal/ nominal
between subjects
- Mann-whitney test
within subjects
- wilcoxon signed rank
procedure
- difference between ranked position of scores in the 2 groups
- test statistical U calculated
- U mapped to critical values to obtain significance value P
statistical significance
- produce value P
- P value indicates the likelihood that differences between conditions is due to chance
- lower P means less likely due to chance
minimum value = 5%
moderate value = 1%
high significance = 0.1%
if P<0.05, null hypothesis is rejected
if P>0.05, null hypothesis is not rejected
P>0.05 does not prove null hypothesis, not enough evidence or data may be noisy
types of errors
type 1
incorrect rejection of true null hypothesis (false positive)
type 2
fail to reject a false null hypothesis (false negative)
statistical power depends on:
significance value
sample size
effect size
- analyse and interpret result
conclusion
hence we reject the __ hypothesis
explain reason of outcome
