Questionnaire Design and Analysis

Question 1

why use questionnaires?

Answer 1

• Objective (factual, standardised) measurement of skill, knowledge, attitudes and behaviours

Comparisons between individuals or groups, associations between factors etc

Cost-effective

Question 2

why use standardised measures?

Answer 2

• Reliability (consistency)
  ○ Test-retest
  • Validity (measure what intended to?)
    ○ Face validity
    ○ Ecological validity
  • Ease of comparison with other studies
  • Time/money

Question 3

what are standardised scales?

Answer 3

Identical materials

Consistent scoring procedures

Clear guidelines on how to administer the test

Question 4

benefits of standardised scales?

Answer 4

Norms provided for comparison

Can make inferences about your sample in comparison to published norms

Can also be used to determine clinical cut-offs

Question 5

definition of latent variable

Answer 5

“hypothetical constructs that cannot be directly measured” (MacCallum & Austin, 2000)

Question 6

questionnaires and latent variables

Answer 6

Questionnaire design involves developing a pool of items or questions that can tap into measuring the latent variable

Question 7

what are uni-dimensional scales?

Answer 7

• All of the items on the scale measure the same thing
  ○ No identified subscales
  • Global score determined by one underlying construct

Question 8

what are multidimensional scales?

Answer 8

All items in scale are still related to and measure the latent variable but there may be groups of questions that intercorrelate more highly than others.

Question 9

why use multiple items/variables

Answer 9

Individual differences

Differences in interpretation

Differences in context

Accidently missing a question

Response biases (circling 4 for the previous 2 questions)

Question 10

development process when constructing a scale

Answer 10

Hypothesis the conceptual framework

Generate item pool and draft instrument

Confirm conceptual framework and assess properties
○ Once you have the item pool…
Pilot the instrument with the relevant group
Perform item analysis
* Identify which items relate most closely to the latent variable
* Examine item-total correlations and internal consistency statistics

Collect and analyse data (item and factor analysis)

Modify instrument

Question 11

internal consistency statistics - cronbach alpha

Answer 11

Cronbach alpha calculates the average of every possible half of the items correlated with every other possible half of the items
○ >0.7 critical value (Kline, 1999)
○ (not bigger than 0.95 though)

Question 12

what is factor analysis

Answer 12

• Collect sufficient sample size to allow analysis of the internal structure of data –> factor analysis
  • A method to assess the extent to which a questionnaire is measuring a latent variable
  • Explores correlation patterns between items to establish an underlying structure

Question 13

types of factor analysis

Answer 13

confirmatory and exploratory

Question 14

advantages of factor analysis

Answer 14

○ Can reveal patterns within the data

	○ Reduces data to ensure only items related to underlying construct are retained

	○ Can lead to the development or refinement of new theories

Question 15

disadvantages of factor analysis

Answer 15

○ Researcher bias

	○ Lack of consistency in methods and cut-off criteria

	○ Garbage in, garbage out

	○ Time and resources

Question 16

modification of questionnaires

Answer 16

• Change items and factor structure over time as necessary
  • E.g. trialling the measure in a population for which it was not originally designed

Question 17

internal consistency - reliability types

Answer 17

split half and cronbach alpha

Question 18

split half reliability

Answer 18

correlation between one half of the items with the other

Question 19

cronbach alpha reliability

Answer 19

average of all possible split-half reliability scores

Question 20

type of reliability for stability over time

Answer 20

test-retest

Question 21

test-retest reliability

Answer 21

extent to which responses on a measure remain stable over time

Question 22

what is validity?

Answer 22

does it measure what it intends to?

Question 23

types of validity

Answer 23

face
content
concurrent
construct

Question 24

preliminary checks for factor analysis

Answer 24

sample size
levels of measurement
normality and outliers
factorability

Question 25

eigenvalues - what are they?

Answer 25

• The amount of variance which a factor can account for in the data

Question 26

which eigenvalue is the largest?

Answer 26

always the first one

Question 27

thresholds for eigenvalues

Answer 27

• Different thresholds: some say eigenvalues over 1 are considered ‘stable’; some say >0.7