Assessment Final Flashcards
Models of Test Construction
Empirical Reasoning
Relies on Data and Analysis rather than Logic and Reason
Deductive Reasoning
In general, uses logic and reasoining to determine what a test response/score means
Logical-Content Strategy
Test developer tries to use logic to determine what content that needs included in their measure
Distinguising Characteristic
Assumption that the test item is accurately describing the content you want
- is the question doing what you think it’s doing
Theoretical Strategy
starts with a theory about your characteristic and then logically dedces items that are in line with that theory
- all items are going to be consistent with this theory
- logical-content guided by theory
Criterion-Group Strategy
- Begins with administering items to the Criterion group and control group
- Look for items that distinguish (or differ) between the two groups
- we don’t care what the content of the items that are different are - Then do Cross Validation
- Do research to find out what it means when the item is continuously endorsed
- what characteristics are the same
Cross Validation
- replicate the study with an independent sample with two completely new groups
- If we get the same results, we can say its cross validated
Control Group
should be a representative sample that is statistically different than the criterion group
Criterion Group
Group of People who share a characteristic/criterion
- i.e schitzophrenia, BPD, etc
Factor Analytic Strategy
- Uses Factor analysis to creat the measure
- look at correlations between items to see how they hang in order to create the factors
- starts with inner-correlations
-how are the items related- trying to determine the minimum number of factors that account for all variability
- then you must label each factor
The MMPI Heart of the MMPI
Validity Scales- give us info about how a person is approaching a test (are they trying to fake good/bad)
Clinical Scales- Designed to assess psychological disorders
Content Scales- gives clinically useful information
History of the MMPI
Was originally developed to distinguish between normal and abnormal psychological functioning
How was the MMPI Constructed?
- originally had like 500 items
- 8 groups with 50 persons in each group
- Made no assumption about what a person’s response to any item means
T-Score Cutoffs of the MMPI
- Initial Interpretations:
- Raw scores are converted into T-scores.
- T = 50, SD = 10.
- Scores of 70 or higher indicate 2 standard deviations above the mean.
- Renormed to be 1.5 standard deviations, considered significantly elevated.
- Original Approach:
- Aimed to aid in diagnosis.
- Pattern Analysis by Meehl:
- Examines index scores and patterns in different scales.
- Utilizes two-point code or numerical code for interpretation.
These cutoffs help clinicians interpret MMPI results and identify significant elevations indicative of potential psychological disorders.
Strengths of the MMPI
-
Criterion-Group Strategy:
- Provides valuable information without assuming the meaning of responses.
- Data-driven approach enhances objectivity.
-
Empirical Basis:
- Utilizes True/False statements, offering self-referential insights.
- Incorporates validity scales to assess test-taking approaches and clinical scales to evaluate psychological disorders.
-
Original Development:
- Developed with a vast item pool to distinguish between normal and abnormal psychological functioning.
- Utilized eight criterion and control groups to ensure robustness.
-
Validity Scales:
- Includes scales like L, K, and F to detect response biases and omitted items.
Weaknesses of the MMPI
-
Cross Validation:
- Relies on a convenience sample, potentially limiting generalizability.
-
Validity Scale Criticisms:
- While the inclusion of validity scales is essential, they may not entirely mitigate response biases or omitted items.
-
Initial Interpretations:
- Conversion to T-scores may simplify interpretations but might overlook nuances in individual responses.
-
Pattern Analysis:
- While pattern analysis is conducted, determining the significance of certain patterns can be subjective and prone to interpretation biases.
-
Numerical Code:
- Reliance on numerical codes might overlook the qualitative aspects of responses, potentially missing important contextual information.
Validity Scales for the MMPI
-
L Scale (Lie Scale)
- Detects individuals favorably presenting themselves.
-
K Scale
- Empirical version of the L scale.
- Compares non-disturbed, healthy individuals with a healthy MMPI to disturbed individuals with a healthy MMPI.
-
F Scale (Infrequency Scale)
- Identifies individuals attempting to appear sick.
- Includes items rarely endorsed by healthy control individuals.
-
Cannot Say Scale
- Addresses omitted items.
- If someone skips 10% of the items, the entire profile is deemed invalid.
What is the CPI
The California Psychological Inventory
- Originally designed to tap into developing NORMAL psych functioning and development
- Sister to MMPI and mostly used in counseling centers or education settings
What is the 16 PF
- Stands for 16 Personality Factors.
- Developed by psychologist Raymond Cattell.
- Comprehensive model describing personality based on sixteen primary factors.
- Factors include warmth, reasoning, emotional stability, dominance, liveliness, rule-consciousness, social boldness, sensitivity, vigilance, abstractedness, apprehension, openness to change, self-reliance, perfectionism, tension, and privateness.
What is the Rogers Q Sort
- a diagram to find the things you most agree wit and the things you most disagree with
- combination strategies
- more modern - usually theoretical combined with empirical
What is the NEO
- NEO-PI-R: A comprehensive inventory to predict various aspects:
- Psychological well-being
- Physical well-being
- Coping styles
- Interests
- Most well-researched currently.
- Major 3 factors:
- Neuroticism
- Extroversion
- Openness
- Each factor has 6 facets, totaling 9 each.
- It clearly overlaps all theories of the Big Five personality traits.
- Taps into all dimensions of the Big Five.
- Provides a comprehensive assessment of personality traits.
Projective Hypothesis (KNOW THIS)
when individuals are presented with ambiguous or vague stimuli, their interpretations reveal underlying aspects of their personality and psychological functioning.
- typical or atypical responses convey significant information
What is the Barnum Effect
- Occurs when generalized statements are perceived as highly accurate descriptions of oneself.
- Characteristics:
- Based on vague or general statements.
- Individuals perceive them as personally relevant.
- Example:
- Horoscopes claiming personality traits that could apply to many.
- Psychological Mechanism:
- Individuals interpret statements in a way that fits their self-concept.
- Criticism:
- Lack of specificity and scientific validity.
- Usage:
- Common in astrology, fortune-telling, and personality assessments.
Basic History of the Rorschach
- Hermann Rorschach developed the Rorschach Inkblot Test in the early 20th century.
- Intended to identify psychological disorders by analyzing interpretations of inkblot images.
- The test involves free association and inquiry phases during administration.
- The Exner Comprehensive Scoring System attempted to address issues but didn’t resolve major problems.
- Major problems include:
- Norms were misrepresented.
- Scoring was unreliable and controversial.
- The “R” problem led to inflated scores with more responses.
- Over-pathologizing occurred, misidentifying disorders.
- Lack of relationship with diagnoses and incremental validity.
- Used in various contexts, including custody cases.
- Criticisms include lack of reliability, validity, and potential harm.
Development of the Rorschach
-
Clinical Validation:
- Blind Analysis: Worked due to the Barnum Effect.
- Barnum Effect: Generalized statements that many people identify with, like horoscopes.
- Confirmation Bias: Seeking evidence that supports inkblot responses while disregarding contrary evidence.
-
Exner Comprehensive Scoring System:
- Didn’t address major problems.
-
Major Problems:
- Norms: Original norms misrepresented.
- Unreliable Scoring: Messy and controversial, lacking validity.
- Problem of “R”: More talking leads to higher scores, related to intelligence and cooperation.
- Over-pathologizing: Incorrectly identifies disorders in nearly 50% of cases, especially in custody battles.
- Lack of Relationship with Diagnoses: No evidence supporting its predictive value.
- Lack of Incremental Validity: Claims of unique clinical insights are false; evidence suggests harm.
Incremental Validity
Gives us information we cannot get elsewhere that will help clinical because it helps us - which is false
Exner System of Scoring
- Scoring:
- Allows for norming but can be complicated and subjective.
- Psychometric Properties:
- Clinical Validation through “Blind Analysis” influenced by the Barnum Effect.
- Confirmation Bias: Seeking supporting evidence while disregarding contradictory evidence.
Problems with the Rorschach - all of them :)
- Exner’s system failed to address critical issues.
- Norms misrepresented, leading to unreliable scoring.
- Problem of “R” where more responses increase scores, correlated with intelligence.
- Over-pathologizing and lack of relationship with diagnoses.
- Lack of incremental validity, evidence suggests harm.
What theory is the TAT Based on
Psychdynamic theory
Murries Theory of Needs
- Proposed by Murray in 1938.
- Main needs include accomplishment, conformity, and attention.
- Edward developed a measure based on this theory.
- Social Desirability measured through forced-choice pairs.
- Individuals select which option is more like them, rather than rating individual items.
- Helps solve the balancing problem in measurement.
TAT Scoring Parameters
- Thematic Apperception Test (TAT) scoring involves several parameters:
- Content Analysis: Interpretation of the story’s themes, characters, and plot.
- Motive: Assessment of underlying needs, desires, and conflicts portrayed in the story.
- Imagery: Evaluation of the vividness and detail in the story’s imagery.
- Affect: Analysis of the emotional tone and intensity displayed in the story.
- Dynamic Components: Identification of interpersonal relationships, power dynamics, and resolution of conflicts depicted in the story.
- Defense Mechanisms: Recognition of psychological defenses employed by the storyteller.
- These parameters help assess the individual’s personality traits, unconscious processes, and interpersonal dynamics.
Holtzman
- Designed for psychometrics of the Rorschach.
- Standardized with detailed instructions and procedures.
- Includes alternate forms for assessing psychometric properties.
- Demonstrates good interrater and split-half reliability.
- Scores on 22 dimensions, some resembling Rorschach dimensions.
- Provides extra information through content domain scoring.
- Well-established norms based on a solid, representative sample.
- Problem: Lacks popularity compared to the Rorschach.
- Creator insisted on empirical evidence and avoided exaggerated claims.
- Validity remains unclear, with some skepticism from experts like Dr. Rhodes.
Drawing Tests
Finger Drawing Tests:
- Known for poor validity.
- Draw a Person Test:
- Typically used with children.
- Participants draw a person and may also tell a story about it.
- House-Tree-Person Test:
- Participants draw a house, tree, and person.
- Evaluates sizing and omissions.
- Kinetic Family Drawing Test:
- Used with children to assess family conflicts.
- Participants draw family members, looking for omissions and disproportions.
- Goodenough-Harris Test:
- Simple and practical.
- Assesses cognitive ability rather than serving as a projective test.
Word Completion Tests
Word Association Test:
- Participants respond with the first word that comes to mind.
- Aims to uncover areas of conflict, distress, etc.
- Originated in the 1800s with Jung.
- Still used today but minimally in clinical settings.
Sentence Completion:
- Patients complete sentence stems (e.g., “I am _____”).
- Seeks themes in responses.
- Rotter Incomplete Sentence Blank scored on a 7-point scale.
- Used more clinically; short, positive responses score higher.
Washington University Sentence Completion Test:
- Developed to measure ego development.
- Aims to assess autonomy and insight levels.
- Contains 36 deep stems.
Cognitive Behavioral Tests
How are C-B Assessments Different
-
Cognitive-Behavioral Rationale:
- Focuses on behaviors, thought processes, and physiological responses.
- Associates all these factors with a particular condition.
-
Specificity in Diagnosis:
- Targets specific behaviors or thought patterns rather than delving into deep-seated trauma.
- Identifies the specific pieces contributing to the diagnosis.
-
Behavioral Targets:
- Aims to change behaviors while addressing underlying causes.
- Focuses on modifying behaviors directly related to the condition.
-
Internal Dialogue:
- Interested in understanding the individual’s internal dialogue before, during, and after the unwanted behavior.
- Considers the cognitive processes influencing behavior.
Behavioral Excesses
Behaviors that are happening too much
Behavioral Deficits
Behaviors that are happening too little
Cognitive Functional Analysis
- Focuses on internal dialogue and how thoughts influence feelings and behavior.
- Recognizes that people don’t always experience positive emotions.
- Emphasizes self-monitoring to understand thought patterns and their impact on behavior.
Behavioral Functional Analysis
involves examining cognitive components, including scales and tests, to understand how thoughts, feelings, and behaviors are interconnected.
Dysfunctional Attitude Scale
Based on Beck’s cognitive model, it explores negative views about self, others, and the future.
Irrational Beliefs Test
- Irrational Beliefs Tests: Assess level of agreement/disagreement on irrational beliefs using a scale.
Irrational Beliefs Inventory
Indicating level of agreement/disagreement on a 5 pt scale
What were the problems with the Irrational Beliefs Test that were addressed with the IBI?
- Limited Scope: The IBT had a narrow focus and did not comprehensively cover all aspects of irrational beliefs.
- Lack of Standardization: The IBT lacked standardization in administration and scoring, leading to inconsistencies in results interpretation.
- Scoring Complexity: Scoring the IBT was complex and time-consuming, making it less practical for widespread use.
- Limited Utility: The IBT had limited utility due to its outdated format and lack of psychometric rigor.
The development of the Irrational Beliefs Inventory (IBI) aimed to address these issues by providing a more comprehensive, standardized, and user-friendly assessment tool for measuring irrational beliefs.
Interest Inventories
Assessments measure individual preferences in various domains like careers, hobbies, and activities through structured questionnaires or surveys. They aid in career exploration, academic guidance, and personal development.
Strong Interest Inventory
- Aimed to study vocational interests of professionals.
- Focused on likes and dislikes regarding career-related activities.
- Utilized the Strong Vocational Interest Blank.
- Originally had around 400 items.
- Separate forms for men and women.
- Offered 54 career options for men and 32 for women.
- Suggested stable interests over time.
- Criticized for gender bias, offering more options for men than for women.
Campbell Interest Invenory
- Improved version addressing shortcomings of the original Strong.
- Based on Holland’s theory of vocation.
- No gender bias present.
- Includes the Campbell Interest and Skill Survey.
- Revised to provide more comprehensive information, spanning approximately 10 pages.
Progression of Interest Inventories
Gender Stereotypes in Interest Inventories
Older ones had stereotypes that guided people a certain way
- Early inventories exhibited significant gender bias.
- Separate forms were used for men and women, often pushing women into stereotypical careers like nursing and teaching.
- Research indicates that women have unique considerations in career decision-making, such as family planning.
- Modern inventories aim to address these considerations and avoid reinforcing traditional gender stereotypes.
False Positive
The test says something is wrong when its actually true
False Negative
test says something is true when it is not
- test will make more errors
Cut Score
the cutoff that puts people into either category
Impact of a False Positive/Negative on Cut Scores
Positive: we will increase the cutscore to correct
Negative: if the cost of a false negative is high, we will lower the cut score
Purpose of Myers Briggs Personality Assessment
- Based on Carl Jung’s theory of personality.
- Identifies four main ways individuals experience the world: Sensing, Intuition, Feeling, and Thinking.
- Aims to assess temperament and personality preferences.
- Jung advocated for striving for balance between these methods.
- Recognizes that individuals rely more on certain methods over others.
- Considered relatively valid but not exceptionally robust in terms of psychometric properties.
What did the Halstead test for
Evaluates different cognitive functions to identify possible brain damage
Major Disadvantage of the Halstead
- multiple tests that takes like 8-12 hours
What did the Nebraska Test for
looked to see which areas of the brain are responsible for which behaviors
Major Disadvantage of the Nebraska
if there is injury, it crashes the whole system
CBLT
Tests for Specific Learning Errors
What is the STAI
The STAI, or State-Trait Anxiety Inventory, assesses two types of anxiety:
-
State Anxiety:
- Measures the level of anxiety that fluctuates from situation to situation.
- Reflects the immediate response to a specific event or circumstance.
-
Trait Anxiety:
- Assesses anxiety as a personality characteristic.
- Represents baseline functioning and indicates a tendency to be more anxious than average.
The STAI is renowned for its excellent psychometric properties, making it a reliable tool for measuring both state and trait anxiety.
What is the EMA
Ecological Momentary Assessments
- allows us to measure levels of stress in real time
- avoids retrospective recall and other biases
Fundamental Difference in Race and IQ
Some minority groups score lower on average on some tests
- most controversial differences show up in intelligence tests
- about 15%-20% of blacks score higher than the average white score
Main Controversies in Test Bias
The main controversies surrounding test bias revolve around various factors:
- Use in Decision Making: Tests are often used in employment decisions and educational placements, leading to concerns about fairness and providing accommodations.
- Accuracy of Predictions: The primary goal is to make accurate predictions, but biases in testing may lead to inaccurate assessments of individuals’ abilities and potential.
- Desirable Traits: Tests may inadvertently favor certain desirable traits, creating inequalities and highlighting differences between demographic groups.
- Group Differences: Some minority groups score lower on average on certain tests, particularly intelligence tests, leading to debates about the reasons behind these disparities.
- IQ Testing Controversy: Intelligence tests, in particular, have sparked controversy, with about 15%-20% of blacks scoring higher than the average white score. The controversy centers on understanding the underlying reasons for these differences.
- Genetic vs. Environmental Influences: If IQ were solely genetically determined, differences between groups would be relatively stable over time. However, the Flynn Effect, which shows increasing IQ scores over generations, suggests environmental factors play a significant role.
Evidence in Test Bias
Content of the item and the facts that disprove them
What is the Flynn Effect
Change in IQ scores that we see over the lase several generations
- IQs have slowly increased
- we think its environmental
- happens for all ethnic groups, but more rapidly for minorities - specifically blacks
Guidelines for Using Tests in Employee Selection
Guidelines for Using Tests in Employee Selection have evolved over time to ensure fairness and compliance with anti-discrimination laws:
-
1964 Civil Rights Act:
- Title VII of the Civil Rights Act established the Equal Employment Opportunity Commission (EEOC), which oversees workplace discrimination issues.
-
EEOC Guidelines:
- The EEOC published guidelines for employee selection based on psychological tests, emphasizing fairness and non-discrimination.
-
1978 Uniform Guidelines on Employee Selection Procedures:
- Established standards for evaluating the fairness and validity of employee selection procedures.
- Introduced the concept of adverse impact, where a selection procedure disproportionately excludes members of a protected group.
-
Adverse Impact:
- Guidelines mandate that any selection procedure resulting in adverse impact on protected groups must be justified by demonstrating its validity for the job in question.
-
Non-Discrimination:
- Guidelines prohibit discrimination based on race, gender, color, national origin, or other protected characteristics.
- Screening procedures that disproportionately reject minority candidates must be justified by demonstrating their validity and job relevance.
What is Differential Item Function Analysis
Definition: Differential Item Functioning (DIF) analysis is a method used to detect items in tests that exhibit bias against specific ethnic, racial, or gender groups.
Purpose: DIF analysis aims to ensure that tests provide fair and unbiased evaluations across diverse demographic groups, contributing to more equitable assessment practices.
Graphs with Lines and Slope
Literaturization of Test Bias
White Male Characterization has no Impact
BITCH
- Developed by R.L. Williams.
- Assesses knowledge of black culture.
- Participants define 100 vocabulary words relevant to the black community.
- Designed to tap into survival skills within black cultural context.
- Intended to challenge stereotypes and refute unfair labels of intellectual impairment.
- Comparisons with the WISC showed no change in categorization.
- Often used to assess a white person’s knowledge of the black community.
SOMPA
- Developed by Mercer in the 1980s.
- Addresses cultural diversity in assessment.
- Challenges the notion of a politically dominant group providing interpretations for society.
- Complex system with emphasis on validity and applicability across cultural groups.
- Assumes that all cultural groups have the same basic potential, with differences attributed to access to cultural experiences.
- Integration of Estimated Learning Potentials (ELPs) compares individuals to peers with similar experiences.
- Provides an IQ-like score reflecting learning potential.
Chitling Test
Argued that all animals are equally intelligent for the environment in which they live
- you cant compare kitties environments to a giraffes environment
- Our tests only support those in a white middle class environment
Quota
trying to match different quotas to represent the people
- explicitly recognizing race and gender
- opposite of unqualified individualism
- doesn’t predict outcomes
- in order for it to be fair, distribution must be representative
Thinking Differently (Last Slide of Ch 19)
- Encourages new interpretations of data.
- Recognizes the value of understanding diverse problem-solving approaches.
- Gardner’s 7 intelligences theory proposes seven distinct types of intelligence.
- Questions why certain abilities are considered intelligence while others are labeled as talents.
- Acknowledges individual variations in intelligence across different categories.
- Suggests developing alternative criteria for assessment, such as classroom grades.
- Sparks debate over the appropriateness and bias in grading systems, especially in middle and high school.
- Emphasizes the role of social environment in shaping educational outcomes.
- Highlights the importance of early intervention, particularly for minority students, to ensure test validity.
- Advocates for equal opportunities and educational access to reduce racial differences in test scores.
Societal Problem
Importance of Increased Experiences
more experience leads to more accurate testing results
What could all tests benefit fit from?
More Validity Evidence
Professional Issues Shaping Testing Field
- Involves discussions about the underlying theories of testing.
- Reliability is a key consideration in testing, ensuring consistency and stability.
- Despite efforts, even the best tests may not achieve perfect stability.
- Precision in testing may affect the level of stability attained.
Moral issues Shaping the Testing Field
- human rights
- labeling
- invasion of privacy
- divided loyalties responsibilities
Social Issues (and their subparts) Shaping the Testing Field
- dehumanizing
- more tech dehumanizes the testing situation
- usefulness of tests
- access to testing services
- it is very expensive
Promising Future
