Questionnaire Design and Analysis Flashcards
why use questionnaires?
- Objective (factual, standardised) measurement of skill, knowledge, attitudes and behaviours
Comparisons between individuals or groups, associations between factors etc
Cost-effective
why use standardised measures?
- Reliability (consistency)
○ Test-retest- Validity (measure what intended to?)
○ Face validity
○ Ecological validity - Ease of comparison with other studies
- Time/money
- Validity (measure what intended to?)
what are standardised scales?
Identical materials
Consistent scoring procedures
Clear guidelines on how to administer the test
benefits of standardised scales?
Norms provided for comparison
Can make inferences about your sample in comparison to published norms
Can also be used to determine clinical cut-offs
definition of latent variable
“hypothetical constructs that cannot be directly measured” (MacCallum & Austin, 2000)
questionnaires and latent variables
Questionnaire design involves developing a pool of items or questions that can tap into measuring the latent variable
what are uni-dimensional scales?
- All of the items on the scale measure the same thing
○ No identified subscales- Global score determined by one underlying construct
what are multidimensional scales?
All items in scale are still related to and measure the latent variable but there may be groups of questions that intercorrelate more highly than others.
why use multiple items/variables
Individual differences
Differences in interpretation
Differences in context
Accidently missing a question
Response biases (circling 4 for the previous 2 questions)
development process when constructing a scale
Hypothesis the conceptual framework
Generate item pool and draft instrument
Confirm conceptual framework and assess properties
○ Once you have the item pool…
Pilot the instrument with the relevant group
Perform item analysis
* Identify which items relate most closely to the latent variable
* Examine item-total correlations and internal consistency statistics
Collect and analyse data (item and factor analysis)
Modify instrument
internal consistency statistics - cronbach alpha
Cronbach alpha calculates the average of every possible half of the items correlated with every other possible half of the items
○ >0.7 critical value (Kline, 1999)
○ (not bigger than 0.95 though)
what is factor analysis
- Collect sufficient sample size to allow analysis of the internal structure of data –> factor analysis
- A method to assess the extent to which a questionnaire is measuring a latent variable
- Explores correlation patterns between items to establish an underlying structure
types of factor analysis
confirmatory and exploratory
advantages of factor analysis
○ Can reveal patterns within the data
○ Reduces data to ensure only items related to underlying construct are retained ○ Can lead to the development or refinement of new theories
disadvantages of factor analysis
○ Researcher bias
○ Lack of consistency in methods and cut-off criteria ○ Garbage in, garbage out ○ Time and resources
modification of questionnaires
- Change items and factor structure over time as necessary
- E.g. trialling the measure in a population for which it was not originally designed
internal consistency - reliability types
split half and cronbach alpha
split half reliability
correlation between one half of the items with the other
cronbach alpha reliability
average of all possible split-half reliability scores
type of reliability for stability over time
test-retest
test-retest reliability
extent to which responses on a measure remain stable over time
what is validity?
does it measure what it intends to?
types of validity
face
content
concurrent
construct
preliminary checks for factor analysis
sample size
levels of measurement
normality and outliers
factorability
eigenvalues - what are they?
- The amount of variance which a factor can account for in the data
which eigenvalue is the largest?
always the first one
thresholds for eigenvalues
- Different thresholds: some say eigenvalues over 1 are considered ‘stable’; some say >0.7