Questionairre Design

Question 1

Individual differences

Answer 1

Psychological traits / characteristics

Five Factor model of personality

Question 2

Ability

Answer 2

Intelligence tests

Question 3

Attitudes

Answer 3

Measure particular beliefs toward something (e.g. work)

Question 4

Types of research

Answer 4

Individual differences
Ability
Attitude

Question 5

Ethics / morals

Answer 5

●Need to ensure that the measures being used are reliable and valid.
Equal opportunities
Cultural biases

Question 6

Creating questionnaires

Answer 6

●Question formats
●Writing your questions
●Clarity of questions
●Avoid leading questions
●Reverse wording
●Response formats
●Clear instructions

Question 7

Question formats

Answer 7

Advantages: Leads to more qualitative data.
Disadvantages: Time consuming to analyse.
● Open vs. closed format questions

Open format asks for some written detail, but has no determined set of responses, e.g.,

“Tell us about the occasions when you have been academically vindictive.”
Question fo

● Open vs. closed format questions

Closed format questions are short questions or statements followed by a number of options.

Question 8

Writing your questions

Answer 8

Theoretical literature: Ideas that appear in the theoretical literature should be used as a basis.
Writing your questions
Experts: Recruit experts in the area to suggest items.
Colleagues: Can help you generate more items.

Question 9

Clarity of questions

Answer 9

“If I had the opportunity, resources and ability to change other students’ exam grades so that mine was the best, I would do it.”
Solution - remove ambiguity:
“If I had the opportunity to change other students’ exam grades so that mine was the best, I would do it.”
Problem:
Respondents may concentrate on opportunity, resources, and ability to different extents.

Question 10

Avoid leading questions

Answer 10

“In the past I have falsely told other students the wrong exam date, but only when they were too lazy to find out themselves, and only when I was in a bad mood, so they would miss the exam.”

Leads the respondent in a particular direction by potentially excusing the behaviour.

Problem:

“In the past I have falsely told other students the wrong exam date, so they would miss the exam.”

Question 11

Response formats

Answer 11

Dichotomous scales (yes/no, true/false):
Frequency of behaviour:
Strongly agree – strongly disagree:
Numerical scales:

Question 12

Classical theory of error in measurement

Answer 12

●Standard error of measurement
●Universe of items
●All items correlate to some extent with the true score
●Reliability is related to the average correlation between items and test length

Observed score = true score + error

Question 13

Reliability

Answer 13

●Internal

–Split-half reliability

–Parallel forms

–Cronbach’s Alpha

–KR-20

●External

–Test-retest

●Inter-rater
–Kappa

●Intra-rater

Question 14

Reliability and validity

Answer 14

Reliability: A questionnaire is reliable if all of the questions in your test are consistently measuring the same underlying concept, and that this remains stable over repeated times that the test is administered.
Validity: A test is valid if it is actually measuring what you intend it to measure.
Reliability is a necessary but not sufficient condition for validity

Question 15

Predictive validity

Answer 15

Assesses whether a measure can accurately predict future behaviour.
Scores on the academic vindictiveness scale should be able to predict people acting in an academically vindictive way in the future:
– Not sharing notes
– Not helping other people revise

Question 16

Construct validity

Answer 16

Discriminant validity:
That the measure is NOT related to things that it should not be related to.

Academic vindictiveness should be measuring something different from Five Factors of personality, so should not correlate highly with extraversion, neuroticism, agreeableness, openness and conscientiousness.

Question 17

Construct validity and convergent validity

Answer 17

Seeks to establish a clear relationship between the construct at a theoretical level and the measure that has been developed.
Convergent validity:
That the measure shows associations with measures that it should be related to, e.g., academic vindictiveness should be related to other aspects of vindictiveness; such as a tendency to seek revenge, or spitefulness.

Question 18

Content validity

Answer 18

The extent to which a measure represents all facets of the phenomena being measured.
So, in the case of academic vindictiveness:
There might be different types:
Academic vindictive behaviours
Academic vindictive attitudes
Academic vindictive feelings

Question 19

Face validity

Answer 19

If something has face validity, it looks like a test that measures the concept it was designed to measure.
In relation to the academic vindictiveness scale:
Find experts in academic vindictiveness, and ask them to judge whether the questionnaire represents a good measure of that construct.

Question 20

Faith validity

Answer 20

Simply a belief in the validity of an instrument without any objective data to back it up, and the evidence is not wanted!

Question 21

Types of validity

Answer 21

●Faith
●Face
●Content
●Construct
-Convergent
-Discriminant
●Predictive

Question 22

Sources of unreliability

Answer 22

●Guessing
●Ambiguous items
●Test length
●Instructions
●Temperature, illness
●Item order effects
●Response rate
●Social desirability

Question 23

Inter-rater reliability or ‘agreement’

Answer 23

The same assessment is completed by the same rater on two or more occasions.
These different ratings are then compared, generally by means of correlation.
Problem: Since the same individual is completing both assessments, the rater’s subsequent ratings are contaminated by knowledge of earlier ratings.

Question 24

Inter-rater Ability or agreement

Answer 24

Cohen’s Kappa: Values up to +1.00, larger numbers indicate better reliability, used when there are two raters.
Fleiss’ Kappa: An adaptation which works for any fixed number of raters.
Inter-rater reliability determines the extent to which two or more raters obtain the same result when coding the same response.
Measures agreement, not accuracy.

Question 25

External reliability

Answer 25

–Test-retest reliability (‘stability’ over time)

●Perform the same survey, with the same respondents, at different points in time.
●The closer the results, the greater the test-retest reliability of the survey.
●The correlation coefficient between the two sets of responses is often used as a measure of the test-retest reliability.

Question 26

Internal reliability

Answer 26

●Internal
–Kuder-Richardson Formula 20 (KR – 20)

●Measures internal reliability for measures with dichotomous choices (i.e., 2 choices Yes/No).
●Values up to +1.00
●Usually a figure of +0.70 or greater indicates acceptable internal reliability.

Question 27

Internal reliability

Cronbach’s Alpha (α)

Answer 27

●Cronbach’s Alpha is mathematically equivalent to the average of all possible split-half estimates.
●Values up to +1.00
●Usually a figure of +0.70 or greater indicates acceptable internal reliability.

Question 28

Internal reliability

Parallel forms

Answer 28

● Create a large pool of items.
● Randomly divide the items into two separate tests.
● Administer the two tests to the same participants.
● Calculate the correlation between the two forms.
Problem: Difficult to generate the large number of items required.