10 - User Research and Evaluation Flashcards
Why is it important to test our product with its users?
- Product must appeal to its users
- Allows you to find problems before selling, minimizing the amount customers lost
In what circumstances can a product be evaluated in?
- Natural settings (In the wild)
- Controlled setting (In the lab)
- Remotely
When should an evaluation be conducted?
In general:
- Before starting
- After prototypes
- After every iteration of the design cycle
- After adding new features
Final decisions have to be based on the data collected
What information is relevant about a user?
- Their goals
- Their behavior
- Their reaction to technology
Who makes up the user population?
People with different goals, capabilities, needs and wants
Important: The designer is not the user
What is a sample?
A subgroup of your users
What are four sampling methods?
- Convenience sampling
- Snowball sampling
- Random sampling
- Stratified sampling
What is convenience sampling?
Selecting a sample based on the people you know/have access to
What is snowball sampling?
Selecting a sample based on people you know and the people they know…
What is random sampling?
Selecting a random subset of the population
What is stratified sampling?
Selecting a sample in proportion to the users (e.g. If 40% of the user population is female, 40% of the sample is female)
What are key elements of proper research conduct?
- Consent forms
- Anonymized data
- Proper data storage
- Research ethics
What are three common user research methods?
- Interviews
- Questionnaire
- Observations
Explain the evaluation flow.
Define the task > Define performance measures > Develop the experiment > Recruit participants > Do the experiment (with corresponding conditions) > Analyze the data > Summarize the data
What metrics do you want to observe when doing usability testing?
- Number of users completing a task successfully
- Time to complete a task
- Time to complete a task after a specified time away from the product
- Number and type of errors per task
- Number of errors per unit of time
- Number of navigations to online help or manuals
- Number of users making a particular error
What are the steps in usability testing?
Define a goal > Generate a research question > Test - Script > Test > Analyze results > Implement results
What is within - subject testing?
When the same participant tests all conditions corresponding to a variable
What is between - subject testing?
When different participants are assigned to different conditions corresponding to variable
What are the benefits of within - subject testing?
- Requires fewer participants
- Increases the chances of discovering the true difference among the different conditions
What are the benefits of between - subject testing?
- Shorter sessions
- Easier to set up and analyze
What is the drawback of between - subject testing?
- Lowers the chances of discovering the true difference among the different conditions
What are the key steps in experimental design?
Welcome participants > Demographics questionnaire > Experiment > Presence questionnaire > Thank participants
What is Fitts’ Law?
It is a predictive model that predicts user performance.
