Developping Questionnaires

Question 1

Questionnaires are being used increasingly in health care research because:

Answer 1

• the traditional, narrow definition of health (morbidity/mortality) replaced by WHO (1948) broader definition of health as “a complete state of physical, mental, and social well-being and not merely the absence of disease or infirmity”

o increasing recognition of the limitations of traditional measures in evaluating health care interventions that may affect several domains including physical, psychological and social well-being

o increasing recognition of the importance of evaluating patient-based health outcomes that are more subjective and complex (e.g. quality of life, disability, quality of care, patient satisfaction, etc)

o increasing consumer demand that the patient’s point of view be incorporated into evaluations of health care o increasing demand from policy makers and providers of health care for evidence of cost effectiveness, e.g. benefits vs. costs of health services

• gold standard scientific methods, borrowed from the social sciences for application in health care, are now available to ensure that these more “subjective” outcomes are measured in a scientifically rigorous (reliable and valid) way. This allows regulatory bodies, clinicians, researchers and patient advocacy groups to determine whether an instrument is a “good” measure that provides scientifically credible information.

Question 2

Systematising data collection by using questionnaires has some powerful advantages over less structured approaches:

Answer 2

• efficiency: simple and cheap to administer, lend themselves to mass production (e.g. census), yield the most information in the least amount of time, allow researchers to obtain only the information they want
• consistency, comparability, generalisation: standardised format ensures that all respondents are asked the same questions, in the same order and manner, this reducing potential error and bias
• summary and analysis: provide quantitative data (numbers, counts, categories, scales) that can be quickly summarised and subject to statistical analysis
• scientific rigour: questionnaires can be evaluated for reliability, validity, responsiveness

Question 3

The degree of structure provided by questionnaires is also the chief weakness:

Answer 3

• limited depth: cannot generally provide an in-depth view
• inflexibility: structured, standardised format is limiting
• cost: significant time and resources required to test and develop questionnaires properly
• error and bias: in questionnaire design and administration, response rates

Question 4

How will the questionnaire be administered? Who will provide the information?

Answer 4

 self-completion
 interviewer-administered (face-to-face vs. telephone vs. postal)
 observer rating (staff/carer/family member)
 data abstraction from medical records

Is it culturally attuned?

Question 5

EVALUATING PSYCHOMETRIC QUESTIONNAIRES

Answer 5

1. Pre-test for appropriateness of content, language, skip patterns
    informal pre-test: home, office, interviews
    formal qualitative pre-test: face-to-face with a small sample of respondents (N=25-30) from target group using think aloud techniques, cognitive interviewing, etc.
2. Field test the questionnaire to evaluate its scientific (psychometric) properties The final step in developing a psychometric questionnaire is to evaluate its scientific rigor using gold standard methods derived from psychometric theory and methods (see Appendix B). There are 3 main criteria for evaluating the scientific rigor of a questionnaire
   - Acceptability: Low proportion of missing data No floor/ceiling effects Responses should be well distributed across response categories
   - Reliability: The precision of measurement regardless of what is being measured
    the repeatability, consistency and stability of a measuring instrument
   - ValIdity The extent to which an instrument measures what it is supposed to measure (e.g. the underlying construct); what the instrument measures and how well it does so

• Responsiveness The ability of a measuring instrument to detect clinically important change/change over time (sensitivity to change)

Use an Existing Measure or Develop a New Measure?
Use a standardised measure with proven reliability, validity and responsiveness
Adapt or modify an existing measure
 use instrument as is; do not “pick and choose” relevant items as psychometric adequacy may be compromised
 adapt/modify and re-assess psychometric properties of adapted instrument OR Develop a new measure
 Need to assess psychometric properties (reliability, validity and responsiveness) of the new instrument

Question 6

Questionnaire measurements

Answer 6

A questionnaire is a structured schedule used to elicit predominantly quantitative information, by means of direct questions from respondents, either by self-completion or via interview (McColl et al. 2001). There are two main types of questionnaires:
• survey questionnaires: these use individual questions that are examined on a one-at-a-time basis; individual questions are the unit of analysis
• psychometric questionnaires (multi-item or summated scales): these measure complex constructs (underlying factors) that cannot be measured directly or captured by a single item; uses multiple items (questions); items are combined to form summated scales; summary/composite/index scores are the basic unit of analysis

Question 7

What to measure

Answer 7

 descriptive information (e.g. demographic characteristics, risk factors, previous/current illness, medications, etc)

 exposure (e.g. urban residence, poor diet, stress, etc)

 explanatory variables–confounders (e.g. smoking history), effect modifiers (e.g. age, gender, etc)

 outcomes (e.g. symptoms, adverse events/complications, quality of life, patient satisfaction, disability, treatment success, etc)

 health-related knowledge (e.g. about illness/treatment), attitudes (e.g. smoking, exercise, use of contraception, preference for different payment systems, preference for different systems of delivery/organisation of care etc) or behaviour (e.g. use of health services, smoking, exercise, etc)

Question 8

Types of validity

Answer 8

 content validity

 criterion-related validity
o concurrent validity
o predictive validity

 construct validity
o convergent validity
o discriminant validity
o group differentiation (extreme groups, group discrimination)
o experimental interventions/hypothesis testing o factor analysis