Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Comp 3020 > Experiments > Flashcards

Experiments Flashcards

1
Q

What is the difference between a usability test and experiments?

A

Usability Testing

  • Goal: Check that the system being developed is usable by the intended user population for their tasks

Experiments

  • Goal: Test hypotheses to discover new knowledge by investigating the relationship between two or more things
    • Involves things like conditions, variables, etc.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is the experimental Process?

A
  • Formulate a hypothesis
  • Identify independent, dependent variables
  • Design a controlled experiment
  • Check for:
    • Confounds
    • Validity
    • Reliability
  • Select representative participants
  • Randomly assign to conditions
  • Run experiment, collect data
  • Analyze results
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is an experiment Hypothesis?

A

A suggested explanation of a phenomenon

  • “If I change A, then B will change in this manner…”

In experimentation, want a hypothesis to be as specific as possible

  • makes it easier to test

To test hypothesis, must identify what variables we think will lead to expected outcome

Must identify how manipulating these variables will result in expected outcome

  • “If I provide keyboard shortcuts, users will be able to complete the tasks faster than with just menus”
  • “If I use pie menus rather than vertical context menus, users will be able to select items faster”

Clearly identify which variables will influence what outcomes, and how

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What are independent variables?

A

Independent variables are those directly manipulated as part of the experiment

Examples

  • Menu type: pie or vertical context
  • Keyboard shortcuts: available, not available

Everything else should be kept constant

They are not dependent on anything in the experiment, you change them on purpose.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What are dependent variables?

A

Dependent variables are those that change in response to the independent variables

ExamplesCompletion time

  • Error rate
  • User preference
  • Quality of user response

Their value is dependent on the changes you made to theindependent variables

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the relationship between independant and dependant variables?

A

Independent variables are assumed to produce an effect on dependent variables’ values when manipulated

  • “If I use pie menus rather than context menus, users will be able to select items faster”
  • Pie menus vs. context menus (independent variables)
  • Item selection speed (dependent variable)

Only manipulating independent variables increases our confidence that any observed changes in dependent variables due to changes in independent variables

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is the difference between causation versus correlation?

A

you notice that people seem to be faster with your interface for people who use a mouse, and slower for people with hotkeys

Therefore, the mouse is more effective for your interface than hotkeys. True?

no – what if it was luck? what if there were other variables that you missed? What if you just noticed what you wanted? What if some other reason is why you have mouse vs hotkey users

This is a correlation – you noticed that two things seem to be linked. This does not mean that one caused the other

You can use experimental design to, on purpose, change one variable to see if the other is impacted. Test causation

Causation vs correlation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What is a null hypothesis?

A

In testing hypothesis, we are seeking to reject the null

hypothesis

Null hypothesis

  • There exists no relationship between manipulating the independent variables and the resultant changes in the dependent variables
  • Example:
    • “There is no difference in selection speed between pie-menus and vertical context menus”
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What are nuisance variables?

A

Any other factors that can affect the dependent variables

Examples

  • Time of day
  • Handedness

Goal is to have as few of these as possible

Use techniques to mitigate the effect (e.g., counterbalancing)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What are the details needed for experimental design?

A

Need at least two conditions

  • Control condition
  • Experimental condition

Control condition

  • No experimental manipulations performed

Experimental condition

  • Experimental variable is manipulated

Results are compared between two conditions

  • Statistics tell you whether the differences are expected randomness or not

Validity

  • Are we measuring what we say we are measuring?

Reliability

  • If we run the experiment multiple times will we get the same results?

Confounds

  • Are there variables we didn’t control for which may be influencing the results we’re obtaining?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q
A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What are the types of validity?

A
  1. Internal
  2. External
  3. Ecological
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What is Internal validty and External validity and what is the tradeoff between the two?

A

Internal validity

  • The changes in the dependent variables are caused by the independent variables

External validity

  • Results can be generalized to other settings, populations, tasks, etc.

There is often a tradeoff between the two

  • The more tightly you control the experiment (to increase internal validity), the less generalizable the results
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What is ecological validity?

A

To what extent do the study conditions mimic those in the real world

Related to external validity, but not the same

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q
A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What are the options for how participants will be assigned conditions?

A

Between-subjects

  • Each participant does one of the experimental conditions
  • Doesn’t account for individual variability
  • Need more participants
  • No learning effects (good)

Within-subjects

  • Each participant completes all experimental conditions
  • Better able to account for individual differences Requires fewer participants
  • Allows participants to make direct comparative statements
  • Learning effects are possible
    • To account for these, order of conditions are usually counterbalanced
17
Q

How many participants should be used?

A

The more participants the better

  • Increases confidence of findings

For large effects, fewer participants needed to achieve statistical significance

  • The smaller the effect, the more participants needed
18
Q

Which is faster?

A

We don’t know with what’s shown in the previous picture.

Statistics!

General Idea: compare to random chance

19
Q

What can you do with your data once you have it?

A

Consult your friendly statistician…

Different tests include:

  • t-tests, ANOVA, CHI-squared, sign test important to
  • pick the right one for your data
20
Q

What are the advatages vs disadvantages for controlled experimentation?

A

Advantages:

  • High confidence in the phenomena that we observed

Disadvantages:

  • Confidence applies to only narrow phenomena
  • Can be difficult to create the right motivation for participants
  • Can’t see things like workarounds that participants develop over time in the field
  • Results can be statistically significant without being practically significant

Proper study design is very hard – you need expertise, mentorship, iteration with trial and error, etc.

  • get a mentor, do an MSc!

Often, a study fails to find a result, but in retrospect you can see confounds, nuisance variables:

  • re-design and re-conduct the study to fix it
21
Q

What are the variables vs dependent variables for the pictures experiment: Keyboard vs Touchscreen

A

independent variable?

  • Interface: existing vs new touch screen

dependent variables?

  • Time, error rate, cognitive load (nasa task-load index)

Comp 3020 (14 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions