Planning Flashcards
Steps in evaluating a prototype
build state hypothesis identify variables design experiment choose popularion apply for review if human subjects run pilot studies run experiment perform analysis draw conclusions
elements of HCI evaluation plan
objectives - questions to answer
participants - plan for their selection
experiment design - what will they do, how connects to research questions
limitations - scope of methods etc.
answerable
All research questions should have expected answers (aka hypotheses)
non-triviality
Is the answer to the research question well-known or easily discovered?
If so, it is not a good research question for postgraduate study.
testability
Can we find the answer to a research question?
Specifically, can we construct an experiment or conduct an analysis to determine the answer?
Falsifiability
Falsifiability means that it MUST be possible to prove a deductive research question to be false. Some condition must exist where, if it is true, implies a negative answer to the hypothesis.
Reproducibility
The answer to a research question should not be vague or applicable to specific data or a specific time (not specified in the question itself).
Variable types in experiment design - name and explain 5
independent - you control them and vary to see effect
dependent
hypothesis
control variables - fixed throughout
random variables - attributes that do vary but that you dont deliberately vary - includes external variables
common dependent variables in HCI
performance metrics
- task success , completion time, errors, efficiency
satisfaction metrics
two types of data
qualitative
quantitative
external validity
participants are representative of target audience
study conditions reflect real world use
internal validity
experiment set up should not favor any one condition
how to do limitations and reflections
reflect onmethods
sources of limites
population representation methods environment prototype fidelity time money bias
WOZ studies
user interface obsercer computer interface elements
Likert scale
repondents rate level of agreement with a given statement
Considerations when using Likert scale
feel like quant measures
nuances in language make unsuitable for multilingual
scalse are not evenly distibuted
quick and dirty vs methodologically sound
what are focus groups
Interviewing people in a group – usually 3-7 people
Participants can respond to and inspire one another
Lots of opinions at once!
what are workshops
Larger groups (10-20) – with planned interactions Locus for training, creativity 90min to multi-day workshops Great for sharing of ideas and data Community-based data collection
key goals for moderating workshops and focus groups
A safe space for participants to interact
All participants can contribute
Interaction around ideas
qualities of a good moderator for workshops and focus group
Has a plan Listens well! Observes and responds to cues from participants Invites participation from all Understands overall goal Maintains topic focus - respectfully
between subject design
different groups of users use different interfaces
within subject design
everyone uses all interfaces (fred and whoever and betty and wilma use both interfaces)
pros and cons of between subject design
Participants cannot compare conditions (+/-)
Can collect more data for a given condition (+)
Need more participants (-)
Often the choice is forced
If the task is time-consuming each subject will only be able to complete one condition Between Subjects is necessary
pros and conds of within subject design
Compare one person across conditions to isolate effects of individuals (+)
Requires fewer participants, less overhead (+)
Fatigue effects (-)
Bias due to ordering/learning effects (-)
If the task is short, filling a one hour session will consume several conditions Within Subjects is natural
