Threats To Psychometric Quality (Response Bias) Flashcards
Psychometric Properties
➢ All measures/ tests will provide us with information & data
❖ However we need to determine what this information really means
➢ Psychometrics properties refer to internal structures & properties of a measure
❖ Determines how well the responses & scores on a measure reflect a construct
➢ Important for identifying/ defining critical aspects of an instrument:
❖ The adequacy & accuracy
❖ The suitability & relevance
❖ It’s use & meaningfulness in a specific circumstance/ context
➢ The development of a valid test is conditional on
the fact that it has been subjected to the relevant statistical analyses
❖ i.e., clarifying it has adequate psychometric properties
Psychometric Properties
Psychometrics Properties
(Consistency of Measures)
➢ The psychometric properties help detect changes/ differences in conditions
❖ i.e., the responsiveness of a measure
Responses Within The Measure
➢ Is a measure tapping into the relevant construct and dimensions?
❖ Factor analysis on items (e.g., structural validity)
❖ Correlations between responses on the items (internal consistency - reliability)
Scores On The Overall Test
➢ Are the overall scores on the test doing what is expected?
❖ Are scores correlated with expected/ similar constructs (e.g., criterion validity)
❖ Are scores consistent over multiple timepoints (e.g., test-retest reliability)
Challenges To Measurement
➢ Complexity Of Psychological Concepts:
❖ Some concepts are complex - need to fully understand the “true” characteristics
➢ Participant Reactivity:
❖ Tests are completed by individuals who may be aware of the test or answer unpredictably
➢ Observer Expectancy/ Bias:
❖ Some test-providers hope to see certain changes which may influence how they score tests
➢ Composite Scores:
❖ Tests typically involve a range of items/ questions that are averaged together
❖ Useful to do but important to consider if specific questions are problematic
➢ Score Sensitivity
❖ Some scales offer more sensitivity than others (e.g., yes-no questions only offer 2 options)
❖ Need to ensure a measure is capable of identifying meaningful differences in an attribute
➢ (Lack of) Knowledge Of psychometrics
❖ People often give out tests with little (or no) regard for psychometrics
Threats To Psychometric Quality
Threats To Psychometric Quality
Types of Bias
- Response Bias
-Test Bias
Types of Response Bias
➢ There are many types of potential response biases
❖ Very difficult to eliminate completely but important to be consider
❖ They may impact the validity and psychometric quality of a measure
1. Acquiescence
2. Extreme and moderate responding
3. Social desirability
4. Malingering
5. Careless or random responding
6. Guessing
➢ Appropriate methods used to detect the problem
➢ Successful methods that have been used to address the problem
- Acquiescence Bias
➢ When an individual agrees with statements without regard for the meaning
❖ e.g., “I enjoy my job”
❖ e.g., “I dislike my job”
➢ People engaging in acquiescence will agree/say “yes” to just about everything
❖ This is often prevalent when all questions are keyed in the same direction
o e.g., “agreement” responses (no reversed items)
➢ Makes it very difficult to effectively differentiate between participants responses
➢ Can also occur in the opposite direction (e.g., “Nay-saying”)
❖ Saying no to things to make a situation seem more positive than its is
➢ It is a threat to the psychometric quality of a test
❖ Creates correlations that are artificially more positive than they should be
Factors that increase acquiescence bias
➢ When respondents do not easily understand items
❖ Items are ambiguous or too complex
❖ If a survey is too advanced for the population (i.e., children)
➢ If there are a large number of items
❖ Lengthy items that take a while to read
➢ Participants are distracted (i.e., not focused on the items)
➢ Some situations/ people are naturally more susceptible to this bias
❖ The context that the survey is given (e.g., judgment evaluations)
Acquiescence Bias
(Solutions)
➢ Word the items in both directions
➢ Avoid too many or overly complex questions
❖ Ensure questions are easily understood and responded to
➢ Ensure participants know responses will be anonymous
❖ i.e., not identifiable from their responses
Extreme & Moderate Responding
➢ Many tests include statements/ questions that require people to respond in:
❖ Intensity (e.g., not at all – completely; not true – completely true)
❖ Endorsement (e.g., strongly disagree – strongly agree)
❖ Occurrence (e.g., never – always)
➢ Some answer options reflect extreme degrees while others reflect moderate degrees
➢ Individuals sometimes have tendencies to use or avoid extreme response options
❖ Some people more likely to endorse extreme responses (i.e., high positive or negative)
❖ Others may avoid extreme answers (even when they feel strongly)
➢ This can result in multiple problems:
❖ Artificial differences (based scoring preference – not feelings)
❖ Obscuring true differences
Extreme and Moderate Responding
➢ An extreme response is not in itself a problem
❖ Desirable for respondents to choose answers reflecting their TRUE construct levels
➢ However, responses can become problematic when:
❖ People with identical construct levels (i.e., true scores) differ in their tendency to
use moderate or extreme responses
❖ People with different construct levels do not differ in their willingness to endorse
moderate/extreme responses (i.e. all go with an average response)
Extreme and Moderate Responding
(Solutions)
➢ Ensure anonymity to participants
❖ Should reduce extreme responses
❖ More likely to respond extremely if not worried about being identified
➢ Have fewer response options (e.g., yes/ no)
❖ Helps reduce the extreme differences
❖ Try to reduce the total survey time (boredom may result in extremes)
➢ Some scales include “filler” questions
❖ Purposely bias some items and see how the person answers
❖ Checks if filler questions also answered at the extreme
- Social Desirability
➢ The tendency for a person to respond in a way that seems socially-appealing
❖ Regardless of his or her true characteristics
o e.g., faking being good
➢ Results in overestimating “good” (positive) behaviour
❖ And underestimating “bad” (negative) behaviour
➢ Example: “Have you ever stolen anything from your employer?”
❖ Tempted (likely) response would be to appeal to the employer
Social desirability can be affected by at least three sources:
- Test content
❖ Some constructs have greater implications for social appeal than others
❖ Intelligence or ability? Exercise behaviour? Honesty versus deceitfulness? - Testing context
❖ Likely to occur when participants can be identified & there may be consequences - Personality of respondents
❖ Some people are more likely to provide socially desirable responses
❖ The need for autonomy” often correlate negatively with socially desirability
❖ There are cross-cultural differences so individuals may differ in social desirability
- Social Desirability
Has implications for:
1. Applied psychology
❖ Psychological tests are often “transparent” in what is being measured (high face validity)
❖ Applied contexts may need to hide the real intent (i.e. job interview)
2. Behavioural research
❖ Lead to inflated correlations between variables (e.g. popularity and well-being)
Socially desirability can occur due to:
➢ Impression management
❖ Conscious process - test takers intentionally attempt to appear socially desirable
➢ Self-deception
❑ Unconscious process - people may actually believe they are better than they are
❑ Links with narcissism
- Malingering
➢ Involves respondents attempting to exaggerate their psychological problems
❖ Opposite issue to social desirability (i.e., “faking bad”)
➢ Compromises the quality of psychological assessment
❖ Failure to consider its impact can be costly
➢ Problematic in testing situations where individuals could benefit from “faking”
❖ Criminal competency hearings
❖ Disability evaluations
❖ Worker compensation claims
➢ More common within applied (rather than psychological) test evaluations
❖ General psychological evaluations = 7% - 27%
❖ Forensic evaluations = 31% - 45%
- Random (Careless) Responding
➢ Test takers provide responses that are truly or at least partially random
❖ Risk of producing scores that are meaningless
❖ Choose answers in a random fashion which are unrelated to item content
➢ This can be due to:
❖ Carelessness
❖ Lack of motivation to respond meaningfully (e.g. randomly circling numbers)
❖ The items are too complex to fully understand
❖ Survey is too long (participants just want to finish it)
➢Can be sometimes hard to detect but sometimes we see patterns
❖ Individual taking a test answering using a 5-point Likert scale
❑ Item 1: select 1 (strongly disagree)
❑ Item 2: select 2 (disagree)
❑ Item 3: select 3 (neutral)
❑ Item 4: select 4 (agree)
- Guessing
➢ Some tests are designed so that specific responses are correct and others incorrect
(e.g. Aptitude or achievement tests)
➢ In consequential situations, respondents may be motivated to guess
➢ Guessing can compromise the quality of test scores
❖ Particularly for tests with a limited set of response items
❖ Correct guesses can artificially inflate observed score relative to the true score
❖ Guessing produces test scores that are inconsistent with true score differences
❑ This can adversely impact reliability of a measure
(Some tests account for this effect in scoring procedures)
Coping With Response Bias
There are three general strategies to cope with response bias:
1. Managing the testing context
❖ Prevent response bias - using strategies to reduce the likelihood of their occurrence
2. Manage the test content/scoring
❖ Design tests in particular ways to help minimise response bias
3. Using “bias-check” tests
❖ Can use separate scales to directly measure different response biases
❑ Social desirability scales
❑ Extremity or acquiescence
These strategies have three main aims:
❖ Minimising the existence of response bias
❖ Minimise effects of response bias
❖ Detect bias and intervene
- Managing Test Context
Reducing situational factors that may lead to response bias:
➢ Assure participant responses are anonymous
❖ Socially desirability less likely if people feel they are responding anonymously
❖ More likely to feel comfortable and be honest
❖ More likely to admit undesirable attributes (about themselves, others, or an organisation)
➢ Try to minimise fatigue, stress, and frustration
❖ Tiredness and low motivation = increased biased & random responding
❖ Clear and concise instructions & questions helps reduce frustration
o Avoid complex or unambiguous items
❖ Ensuring participants can respond at a time suitable for them
o Online surveys can be saved and returned to (reducing frustration)
➢ Tell respondents that the validity of their responses can be evaluated
❖ Inform participants that false responding can be detected by the test
❖ Mere possibility of detection may convince some respondents to be honest/ unbiased
- Managing Test Content
Structure the content of the test to minimise response bias:
➢ “Balanced” scales (equal number of positively & negatively worded items)
❖ Minimise the effects of acquiescence bias (e.g., agreeing with everything)
➢ Include neutral phrases in the wording of items and response formats
❖ Instead of “I am a hostile person” - “I am sometimes less friendly than other people”
➢ Use of forced choice-formats:
❖ Present two characteristics and respondents only select one (eliminates extremes)
❖ Present two equally socially desirable characteristics (e.g. “friendly” and “assertive“)
❖ Then two equally undesirable characteristics (e.g. “timid” and “argumentative”)
➢ Use specialised scoring procedures to minimise/ discourage guessing
❖ Achievement tests (e.g. SAT) adjusts test scores to account for guessing
o Incorrect answers = point deducted
o An item left unanswered = no point deduction
- Managing Test Content
➢ Some measures incorporate so-called “validity scales”
Example: Millon Clinical Multiaxial Inventory (MMPI)
➢Used to measure personality traits and psychopathology
➢ Includes two validity scales within the measure:
1. L (Lie) Scale:
➢ 15-items that describe “minor” flaws and weaknesses
❖A high score = denial of these common characteristics (social desirability)
2. The F (Infrequency) scale:
➢ Consists of 64-items of traits endorsed by very few respondents
❖A high score = deviant/ random responding (malingering)
➢ The MMPI gives instructions on how to deal with response bias
❖ May totally exclude any interpretation of the scores
❖ May simply interpret results cautiously
❖ May use statistical procedures to “correct” the original scores
- Using “Bias” Checks
➢ There are tests dedicated to the establish the degree of socially desirability
➢ These scales exist independently of any other scale
❖Can be used in a variety of contexts (applied or research)
➢ Two of the common scales designed for this purpose:
1. Marlowe-Crowne Social Desirability Scale
2. Balanced Inventory for Desirable Responding
Using “Bias” Checks
➢ An alternative bias test is The Dot Counting Test
➢ Test specifically designed to detect malingered cognitive impairment
(i.e. “faking bad”)
➢ Consists of 12 cards:
❖ Half contain dots presented in a grouped fashion
❖ Half present dots in a random fashion
➢ The required cognitive competencies to perform this task are very basic
➢ Malingering suspected if it takes equal time to count the grouped & random dots
