§3-gathering and analysing data

Question 1

Purpose of data gathering, analysis, interpretation

Answer 1

for requirements gathering

1. want to ensure our requirements will be stable
2. want statistical evidence about what kinds of users we have, what kinds of goals they have, their contexts of use

Question 2

list of issues re data gathering

Answer 2

 Observation
 Interviews
 Focus groups  Card sorting
 Questionnaires
 Studying documentation
 Scenarios / Use cases
 Researching similar products  Web analytics

Question 3

Alexa case study

Answer 3

1. dealing with ambiguous questions
2. not personalsed to possible end users
3. perceived as odd if responds without being called
4. known via focus groups and user testing
5. sensitive issues, avoid
6. policy teams,
7. data gathering to find what issues users consider sensitive

Question 4

Pros/cons of ethnography/observation in the field

Answer 4

1. realistic
2. costly to set up
3. intrusive, affecting performance, skewing data
4. privacy, reliability

Question 5

Pros/cons of observation in the lab

Answer 5

1. less privacy intrusion,
2. not realistic setting
3. capture errors

Question 6

P/C of direct observation

Answer 6

1. users say what they’re doing and why, while performing task
2. good for understanding nature/content of task
3. too much data, time consuming

Question 7

P/C of indirect observation

Answer 7

1. users do task, then later explain what they did and why

2. users may skip steps

Question 8

P/C of unstructured interviews

Answer 8

1. open questions
2. rich data
3. too much data, off topic, long answers

Question 9

P/C of structured interviews

Answer 9

easy data collection, on topic,

rigid, need to know right questions to ask

Question 10

P/C of semi structured interviews

Answer 10

rich and targeted data

Question 11

P/C of interviews in general

Answer 11

forum for talking to people
good for exploration of issues
need to avoid long questions
jargon that interviewee may not understand
leading questions that make assumptions
unconscious biases e.g gender stereotypes

Question 12

Purpose of focus groups

Answer 12

1. identify conflicts in terminology or expectations from users
2. participants need to be representative of target users

Question 13

Purpose of card sorting, pros/cons, types

Answer 13

1. what are natural groupings
2. apply e.g. to menu design
3. apply to potential users
4. groupings of terminology, relationships (similarity of concepts), categories (names of groups that make sense e.g. shopping till – to many different user groups).
5. open card sorting – no preestablished groupings
6. closed – given groups, place into
7. hybrid – combine the two

Question 14

Issues re questionnaires **

Answer 14

1. elicit specific info
2. good for getting specific answers from large, decorrelated group of people
3. can provide both quan/qual data
4. use in conjunction with other techniques
5. ordering of questions may influence responses
6. different versions for different populations?
7. need clear completion instructions
8. avoid long questionnaires
9. decide whether all phrases will be positive/all negative or mixed

Question 15

Open questions vs closed questions

Answer 15

open – free to answer in any way

closed – select answer from set of possibilities - checklist/rating scale/ranked order

Question 16

How to get good response rates

Answer 16

1. make purpose of study clear
2. promise anonymity
3. easy to complete / unambiguous questions
4. short version + long version
5. incentives

Question 17

What is the point of studying documentation – Pros/cons of studying documentation as a way of gathering data about user requirements

Answer 17

1. find out about steps involved in an activity
2. regulations re a task
3. no stakeholder time, which is a limiting factor on other techniques
4. not in isolation – because may be idealised

Question 18

Pros/cons of researching similar products

Answer 18

1. prompt possible requirements
2. generate alternative designs
3. inhibit creativity

Question 19

Why do web analytics

Answer 19

1. number of web visitors and views
2. demographics, other interests
3. figure out who we need to be surveying/interviewing

Question 20

How would one choose between data gathering techniques

Answer 20

1. time, level of detail, risk
2. knowledge (or lack of) possessed by analyst
3. kind of task
   3a sequential or parallel/fragmented
   3b high/low complexity
   3c layman or skilled practitioner

Question 21

General guidelines for gathering data

Answer 21

1. focus on identifying stakeholder needs rather than technical stuff
2. involve all stakeholder groups
3. multiple representatives from each group
4. triangulation – combination of data gathering techniques + more than one type of data
5. maintaining data – database?
6. should be able to update data over time
7. compromise between analysis time and gathering time
8. support with prototypes and task descriptions
9. compromise between time devoted to functional and non functional requirements

Question 22

Purpose of data analysis and interpretation

Answer 22

1. interpret and present findings from gathered data

Question 23

Quantitative vs qualitative analysis

Answer 23

numerical methods (compute summary stats) 
vs 
express themes/patterns which are harder to quantify (e.g. complaint letters) 

performed after initial processing (e.g. putting into spreadsheet, transcription of interviews)

Question 24

How would we differ in our use of quantitative analysis for small vs large projects of card sorting?

Answer 24

small – look at piles of cards for patterns

larger – cluster analysis, tabulation

Question 25

Buzzwords on how we can analyse results of card sorting

Answer 25

1. distance matrix — (i, j) = fraction of times i was placed in same category as j
2. perform cluster analysis, dendrogram, hierarchical structure
3. similarity rating

Question 26

What is the similarity rating and how is it used in cluster analysis

Answer 26

• • Every time two cards are in the same pile you assign them 1 point
• • Similarity rating = add up all of the times that two cards appear together and divide by the number of groups

Question 27

What are we looking for in qualitative analysis

Answer 27

1. recurring patterns, emergent from data
2. categorisations
3. critical incidents (E.g. complaints)

Question 28

issues re presenting findings from data analysis

Answer 28

1. depends on audience, purpose,

2. graphical e.g. pie charts