Questionairre Design Flashcards
Individual differences
Psychological traits / characteristics
Five Factor model of personality
Ability
Intelligence tests
Attitudes
Measure particular beliefs toward something (e.g. work)
Types of research
Individual differences
Ability
Attitude
Ethics / morals
●Need to ensure that the measures being used are reliable and valid.
Equal opportunities
Cultural biases
Creating questionnaires
●Question formats ●Writing your questions ●Clarity of questions ●Avoid leading questions ●Reverse wording ●Response formats ●Clear instructions
Question formats
Advantages: Leads to more qualitative data.
Disadvantages: Time consuming to analyse.
● Open vs. closed format questions
Open format asks for some written detail, but has no determined set of responses, e.g.,
“Tell us about the occasions when you have been academically vindictive.”
Question fo
● Open vs. closed format questions
Closed format questions are short questions or statements followed by a number of options.
Writing your questions
Theoretical literature: Ideas that appear in the theoretical literature should be used as a basis.
Writing your questions
Experts: Recruit experts in the area to suggest items.
Colleagues: Can help you generate more items.
Clarity of questions
“If I had the opportunity, resources and ability to change other students’ exam grades so that mine was the best, I would do it.”
Solution - remove ambiguity:
“If I had the opportunity to change other students’ exam grades so that mine was the best, I would do it.”
Problem:
Respondents may concentrate on opportunity, resources, and ability to different extents.
Avoid leading questions
“In the past I have falsely told other students the wrong exam date, but only when they were too lazy to find out themselves, and only when I was in a bad mood, so they would miss the exam.”
Leads the respondent in a particular direction by potentially excusing the behaviour.
Problem:
“In the past I have falsely told other students the wrong exam date, so they would miss the exam.”
Response formats
Dichotomous scales (yes/no, true/false):
Frequency of behaviour:
Strongly agree – strongly disagree:
Numerical scales:
Classical theory of error in measurement
●Standard error of measurement
●Universe of items
●All items correlate to some extent with the true score
●Reliability is related to the average correlation between items and test length
Observed score = true score + error
Reliability
●Internal
–Split-half reliability
–Parallel forms
–Cronbach’s Alpha
–KR-20
●External
–Test-retest
●Inter-rater
–Kappa
●Intra-rater
Reliability and validity
Reliability: A questionnaire is reliable if all of the questions in your test are consistently measuring the same underlying concept, and that this remains stable over repeated times that the test is administered.
Validity: A test is valid if it is actually measuring what you intend it to measure.
Reliability is a necessary but not sufficient condition for validity
Predictive validity
Assesses whether a measure can accurately predict future behaviour.
Scores on the academic vindictiveness scale should be able to predict people acting in an academically vindictive way in the future:
– Not sharing notes
– Not helping other people revise
Construct validity
Discriminant validity:
That the measure is NOT related to things that it should not be related to.
Academic vindictiveness should be measuring something different from Five Factors of personality, so should not correlate highly with extraversion, neuroticism, agreeableness, openness and conscientiousness.
Construct validity and convergent validity
Seeks to establish a clear relationship between the construct at a theoretical level and the measure that has been developed.
Convergent validity:
That the measure shows associations with measures that it should be related to, e.g., academic vindictiveness should be related to other aspects of vindictiveness; such as a tendency to seek revenge, or spitefulness.
Content validity
The extent to which a measure represents all facets of the phenomena being measured.
So, in the case of academic vindictiveness:
There might be different types:
Academic vindictive behaviours
Academic vindictive attitudes
Academic vindictive feelings
Face validity
If something has face validity, it looks like a test that measures the concept it was designed to measure.
In relation to the academic vindictiveness scale:
Find experts in academic vindictiveness, and ask them to judge whether the questionnaire represents a good measure of that construct.
Faith validity
Simply a belief in the validity of an instrument without any objective data to back it up, and the evidence is not wanted!
Types of validity
●Faith ●Face ●Content ●Construct -Convergent -Discriminant ●Predictive
Sources of unreliability
●Guessing ●Ambiguous items ●Test length ●Instructions ●Temperature, illness ●Item order effects ●Response rate ●Social desirability
Inter-rater reliability or ‘agreement’
The same assessment is completed by the same rater on two or more occasions.
These different ratings are then compared, generally by means of correlation.
Problem: Since the same individual is completing both assessments, the rater’s subsequent ratings are contaminated by knowledge of earlier ratings.
Inter-rater Ability or agreement
Cohen’s Kappa: Values up to +1.00, larger numbers indicate better reliability, used when there are two raters.
Fleiss’ Kappa: An adaptation which works for any fixed number of raters.
Inter-rater reliability determines the extent to which two or more raters obtain the same result when coding the same response.
Measures agreement, not accuracy.
External reliability
–Test-retest reliability (‘stability’ over time)
●Perform the same survey, with the same respondents, at different points in time.
●The closer the results, the greater the test-retest reliability of the survey.
●The correlation coefficient between the two sets of responses is often used as a measure of the test-retest reliability.
Internal reliability
●Internal
–Kuder-Richardson Formula 20 (KR – 20)
●Measures internal reliability for measures with dichotomous choices (i.e., 2 choices Yes/No).
●Values up to +1.00
●Usually a figure of +0.70 or greater indicates acceptable internal reliability.
Internal reliability
Cronbach’s Alpha (α)
●Cronbach’s Alpha is mathematically equivalent to the average of all possible split-half estimates.
●Values up to +1.00
●Usually a figure of +0.70 or greater indicates acceptable internal reliability.
Internal reliability
Parallel forms
● Create a large pool of items.
● Randomly divide the items into two separate tests.
● Administer the two tests to the same participants.
● Calculate the correlation between the two forms.
Problem: Difficult to generate the large number of items required.
