L9, Data collection 2 - qualitative Flashcards
Pros and cons with using available information?
PRO
1. inexpensive
—> data already there
2. available to see trends over the past
CON
- difficult to access
- ethical issues concerning confidentiality
- information may be imprecise or incomplete
Pros and cons with observations
PRO
- more detailed and context-related information
- tests reliability of responses in questionnaire
CON
- ethical issues with confidentiality and privacy
- observation bias
- presence of data collector high influence results
- thorough training of research assistants is required
Pros and cons with interviewing?
PRO
- suitable for use with both literates and illiterate
- permits clarification of questions
- has higher response rate than written questionnaires
CON
1. The presence of the interview might influence the response
- reports of events might may be less complete than with observations
Pros and cons with small scale flexible interview?
PRO:
1. permits collection of in-depth information and exploration of spontaneous remarks by respondents
CON:
- Interviewer might influence interviewer
- analysis of open-ended data is more difficult and time-consuming
Pros and cons with larger scale fixed interview?
PRO
1. easy to analyze
CON:
1. Important information may be missed due to spontaneous remarks by respondents are not recorded or documented
Pros and cons with Adminestering questionnaires?
PRO 1. Less expensive 2. permits anonymity —> honest results 3. no need for research asistance 4. eliminates bias due to phasing questions differently with respondents
CON:
- cannot be used with illiterate respondents
- often low rate of response
- questions may be misunderstood
Pros and cons with participatory and projective methods?
PRO:
- provide rich data
- positive spon offs for knowledge and skills by researchers and informants
CONS:
1. require extra training of researchers
How should you make the choice of method for a market study?
It should be determined by the purpose of the study
— Is it explorative?
— Confirmative?
The purpose of market studies can be assumed to differ in different phases of the product development process
In what ways could requirements for solutions be expressed?
- as descriptions of problems
- as comparisons
- as assumptions
- as behaviors
- as actions
- as requirements
- as solutions
How does requirements emerge?
- they do NOT emerge in a vacuum
- In a situation
—> “the use situation”
—> “the use system” - the methods chosen must be able to grasp this system
The choice of method is a matter of four different choices!
- data collection method
- participants
- context
- Mediating tool /stimuli
Choice of participants?
— The qualitative dimension?
— The quantitative dimension?
Depends in the sampling: 1. statistical sampling? 2. theoretical sampling? —> choose participants with a particular property 3. Critical sampling? 4. Convenience sampling?
How can you divide users into groups?
- The common
—> age
—> gender
2. Some other —> product user experience —> problem experience —> interests —> physical properties —> personality
Some thoughts on participants?
- Think about what characteristics are really important when you decide on your sampling strategy
- participants should be part of the identified user group
- avoid representatives of the users
- should preferably have experience of the use
—>or experience of the problem
What kind of participants could be difficult to find?
- elderly frail
2. busy people
what kind of participants are easy to find?
- unemployed
- students
- women with academic background
the quantitative dimension of qualitative studies?
- how many
—> the more the better - Find the saturation point
- The rule of diminishing returns
How many people should be subject for qualitative studies according to literature?
- Griffin and Hauser
—> 30 interviews = 99% requirements
—> 98% of customer requirements after 25 hours of interviews or focus groups - ulrish and eppinger
—> 10 is not enough
—> 50 is too many - virzi and Lewis
—> 12-20 people
What are the two main categories for data collection methods?
- Question-based methods
2. Observation-based methods
Question-based methods?
- one-to-one interview
—> in-depth interviews - group interviews
- focus group interviews
- surveys
- Questionnaires
—> telephone
—> post
—> e-mail or the web
How should questions be structured?
- Structured
The questions are formulated and the order of the questions are defined in advance - Semi-structure
—> Questions are formulated in order to guarantee that certain topics are covered but without the exact formulation and the exact order may vary between interviews - unstructured
—> the interview governs the content of the interview
—> the interviewer’s role is more passive.
—> the questions are not formulated in advance
How should the questions be structured in general?
- Move from the general tp the more specific
—> as a funnel - Move from the neutral topics the more “sensitive! topics
- Move from questions on facts to questions which require reflection and evaluation
Interview technique?
- Probing
—> what do you mean?
—> why?
—> can you demonstrate? - Confirming questions
—> have I understood you correctly…?
How should the questions be formulated?
- be specific
—> only one question at the time - Not be complicated
—> do not elaborate too much in the questions - Be formulated in terms that could be understood without problems
—> terms that are used by the target group - You should not be leading
- Be neutral
—> don’t be loaded in any direction (positive, negative= - do not include “prestige bias”
- minimize demands on memory
- Avoid to be hypothetical
Describe open and closed questions
- Open questions
—> requires explanations
—> develops the diaglog
2. closed questions —> concern confirmations or denials —> have you (or not) —> is it (or is it not)? —> Can you (or can't you)?
Give an example of a question put as opened and close!
CLOSED QUESTIONS
Do you think it’s difficult to book a ticket on the company’s websight?
OPEN QUESTION
What are your thoughts about booking a ticket on the company’s web sight?
What is an indirect question?
- An indirect question directs the interviewee’s attention to another person (or to other persons) than the interviewee
- Can be a way to adress more sensitive questions or areas where a true answer may not be anticipated
EXAMPLE
Direct:
— Do you use the safety devices
Indirect:
— Do people working here use the safety devices?
What factors will influence the quality of the interview?
1. The interviewer —> social skills —> training and experience —> motivation —> safety, security
- The content
—> sensitivity
—> complecity
—> interest. - The situation
—> time
—> place
—> other people - The interviewee
—> social skills
—> ability to answer
—> safety, security
Tips on interviewing?
- don’t be afraid of silence
- try not to pose leading questions
- use probing
- let the interviewee lead the conversation
- do not let the interviewee lead the conversation in the wrong direction
- do pre-tests of your interview guide
- mediating tools are useful
what is probing?
what do you mean?
why?
can you demonstrate?
What is a focus group?
- A group interview with a moderator
- The aim is that the participant shall discuss with each other not with the moderator
- 6-10 participanats
- useful for “sensitive topics”
- Often advocated that the Ps should be similar but…
Fun facts or whatever:
— Developed during WW2 - for propaganda film
— forgotten until 1980
what is the role of the moderator?
- Lead and focus the discussion
- Help the silent
- Keep the talkative down
Practicalities during focus group?
- one moderator
- one helper that takes notes can speed up the process
- start with fika
—> hungry people are not as engaged - Sandwich is good
—> candy is better
What are the relative benefits of focus of focus groups
- There are no scientific proofs that focus group have benefits compared to interviews
- There are lots of anecdotal evidence that focus groups work as advertised
- Focus groups are fun
What mediating tools are there?
Mediating tool or mediating object is here a “stimulus”
- something which enhances reflection
- something which enhances discussion
- something which brings out focus
- The purpose of using a mediating tool may be divergent or convergent
what type of mediating tool is proper?
— It depends on the purpose of the study
— Mediating tools may be used in user studies with the purpose to:
- describe problems
- elicit requirements
- generate ideas and concepts
- evaluate design solutions
give examples of mediating tools!
- pictures
- video
- collage
- sketches
- prototypes and mock-ups
- “cultural probes”
why is sketches a good mediating tool?
It is good to accompany with story boards
Describe observation based methods?
— Observations are an efficient way to study situations
— Requirements are shown
— As descriptor of problems: —> as comparisions —> as assumptions —> as compensating behaviour —> as actions —> as requirements —> as solutions
What different types of observation are there?
- Direct observation
— Observation of something that takes place before our eyes.
— Typically meetings, what goes on in a railway station etc.
2.Participatory observation
— Common anthropological and sociological research
— The observation takes part in the activities studies
—> The social system that is under observation during a shorter or longer period of time
- Self observation
— A special form of self observation is when the researcher observe themselves.
— e.g. diaries or photo assignments - natural or constructed observation
- obtrusive or unobtrusive observation
(requries acceptance and trust)
Describe the structure during observations
The same spectrum as for question based methods:
1. unstructured
—> trying to grasp the situation
- Semi-structured
—> having ideas what to look for
—> a plan for recording and analyzing - Structured
—> you look for specific things according to an observation protocol or similar
Name the motives for observation studies
- the customer/ user does not always want to describe problems, requirements, wishes etc.
- It is common to blame oneself for different failures, not the design of the environment or product
- The customer cannot always provide the information
—> lack of language-skills
—> compensating behaviour
—> actions having become non-conscious
—> regarding the requirement as self-evident
What could be observed?
- actions
—> what people do as as well as what people do not
—> work postures
—> work sequences of action and sub-actions
—> duration of actions
—> ineraction patterns
—> compensating behaviors - Communication patterns
—> ““A communicates with B, while C communicates with D”
—> someone looking for information (when not needed)
—> someone is not looking for information (when needed) - Alternative usage of the product
—> product is ised for different purpose than the assumed one
—> used in a different manner then the assumed one - Modifications to the product
—> in order to better fit the individual
—> in order to better fit the task
Negative thoughts about interview methods for evaluation?? and pros for observations?
- Questions could disturb peoples natural way of doing things and are efter based on pre-assumptions
2.People
—> people don’t ask questions to problems they don’t know or they don’t think can be solved
—> give poor results on everyday life issues, routines and habits
—> often provide answered they think is expected from them
- observation
—> cannot rely on what is reported on actions, problems etc.
—> by observation the customers own modifications can be identified - Communication is non-verbal
Important to remember with regards to observations
- to observe is not simply too look
- to observe is a state of mind, a mission
- The one who needs the observations done, is the one wh should carry out the observations
Definition of context?
- The part od a text or statement that surrounds a particular word or passage and determine it’s meaning
- the circumstances in which an event occurs
- A setting
- Context is sometimes referred to as environment
- More than the physical environment
—> circumstances in which the product will be expected to operate
Aspects of context?
1. Physical environment —> light —> sound —> pollutants etc.
- Social environment
—> collaboration
—> attitudes
etc. - Organizational environment
—> infrastructure
—> support - Technical environment
name some different approaches where context is stressed?
- Ethnography
—> writing the culture, studies with the purpose to understand, to make the implicit explicit
—> participation and involvement of the researchers - Naturalistic studies
- emphatic design
- Contextual design
—> contextual interview (combo of interview, observation and reconstruction of events)
—> understanding develops in collaboration between user and developer
—> does not impl the degree of user involvement
—> more or less structures
Name some methodical consequences?
- the time that has to be spend
- enough time to undersand the individual’s objective, the motive behind an activity
- the individual’s objectives change over time
- actions become operations over time
- Different data collection methods and analyses are needed
- triangulation of different data collection methods e.g.
—> question based
—> observation based
—> earlier documentations/ research - Researcher’s perspective
—> approach and ambition will influence the outcome of the study
what is triangulation?
— Different method provide different types of data, often complimentary
— Qualitative data is often needed to explain quantitative data
