Validity (Dimensionality & Factor Analysis) Flashcards
Validity
➢ A common definition:
❖ “Validity is the degree to which a test measures what it is supposed to measure”
➢ A more detailed definition is:
❖ “the degree to which evidence and theory support the interpretations of test scores
for the proposed uses of a test”
➢ Many authors state that “a test is valid” - that is not strictly true
❖ It is the interpretation of the test that we determine for validity
Key Aspects Of Validity
➢ This more sophisticated definition highlights three important implications:
1. A measure itself is not valid
❖ It is the validity of the interpretation of test scores for a specific construct
❖ Thus, a good understanding of the construct being measuring is needed to understand validity
2. Validity is a matter of degree (i.e., not “all-or-nothing”)
❖ Use terms like “high degree of validity” or “acceptable levels of validity”
❖ Not “this proves the measure is valid”
3. Validity of interpretation needs to be based on empirical evidence & theory
❖ We need to understand the definition of our construct & the theory that underpins it
❖ Data indicating that those with high scores are actually high in the concept
o We need to know precisely what we are trying to measure
“Validated” Measures
➢ Just because a measure is used a lot doesn’t make it valid
❖ Measures will be validated through scientific development and psychometric tests
❖ There is a wealth of various measures/ tests in existence…. not all have been validated
➢ A well-established measure will have:
❖ Strong empirical evidence underpinning its development
❖ Clear definitions of the construct and all its components (i.e., dimensions)
❖ Will have had psychometric tests conducted to check the degree of validity
➢ All sound psychological measures will have been “validated”
❖ By the authors who created the measure
❖ This is why it is important to cite the measure you use (and the authors)
➢ Any revisions made to a measure (changing items; shortening; simplifying for children)
❖ These revisions need to be examined for validity
❖ Any revisions of scales also need to be validated (sometimes by different authors)
Validity Vs Reliability
➢ Both concepts are fundamental to psychometrics but different conceptually
➢ Reliability
❖ How differences in scores reflect differences among people in a trait
o Reliability is considering consistency between scores
o But we don’t really need to know what the scores relate to (i.e. the trait)
o Reliability can be assessed by calculating a single score (e.g., correlation/ Cronbach Alpha)
➢ Validity
❖ Related to the interpretation of test scores in terms of their meaning
o Validity centres on our interpretation of what is being assessed
o We need to know what we are supposedly measuring
o Validity is assessed more on a continuum (i.e., by a series of considerations/ aspects)
➢ In fact, validity often requires reliability
❖ If scores are not consistent then we can’t know that tests are measuring the same thing
➢ In contrast, reliability can exist without validity
❖ We may have a test that produces excellent internal consistency/ test-retest scores…
❖ But does not actually measure what it intends to (i.e., cannot be interpreted as valid)
Validity: Importance
➢ Validity is a crucial issue in a test’s psychometric quality
❖ And the development of psychological tests/ measures
➢ When psychological measurements are created, they must have sufficient validity
for their intended purpose
❖ Without sufficient validity, the tests are scientifically meaningless
➢ Our ability to interpret behavioural research findings requires validity
❖ Especially if measures may be used for diagnosis purposes
❖ Poor validity could lead to inaccurate/ ineffective/ harmful decisions
➢ We want to be confident that a measure is assessing what we want
❖ So, we can accurately describe, predict, & explain different psychological attributes
Validity: Research & Individuals
➢ Measures are used for:
❖ Research studies (across a sample)
❖ Individuals
➢ Validity can therefore impact:
❖ Understanding of the world
o Scientific knowledge of concepts
❖ Quality of decisions
o Individuals, organisations, & society
➢ If our interpretations of tests is invalid:
❖ Harmful decisions
o Incorrect diagnosis
o Decisions that unfairly exclude people
Construct Validity
➢ Construct validity is the essential concept when thinking of validity
❖ “The degree to which scores can be interpreted as reflecting the psychological construct”
➢ There are five facets of overall construct validity
❖ Five types of evidence for establishing the validity of test interpretations
1. Test Content
2. Internal Structure
3. Response Process
4. Associations With Other Variables
5. Consequences of Use
Determining Validity
➢ Test Content (the content within the test itself)
❖ Face Validity
❖ Content Validity
➢ Internal Structure (dimensionality – factor analysis)
❖ Structural Validity (are the construct & components being measured accurately)
➢ Response Process (the process respondents take to complete the test)
❖ Response Process Validity
➢ Associations With Other Variables (how do test scores relate to other scores)
❖ Convergent & Discriminant Validity (links with related & unrelated constructs)
❖ Criterion Validity (links with important other variables)
o Concurrent Validity (relations between constructs at one timepoint)
o Predictive Validity (relations between constructs at future timepoints)
➢ Consequences of use (the use & impact of the test)
❖ Consequential Validity (intended & actual consequences of a test)
Test Content
➢ Understanding the test content is important for validity….
❖ Face Validity
❖ Content Validity
➢ Does the actual content of a test align with the content that it should be measuring
❖ Is the content the correct content to measure a particular construct
➢ If we a measuring a construct then the content of the measure should:
❖ Reflect all the essential components of that construct (e.g., does not ignore one)
❖ ONLY tap into the intended construct
Example: Empathy Measure
❖ Should include items that tap into cognitive empathy (understanding others’ thoughts/ views)
❖ Should include items that measure emotional empathy (understanding others’ emotions)
❖ Should NOT include items that measure happiness
Face Validity
➢ The degree to which a measure appears related to a specific construct
❖ the judgement of non-test experts (i.e., test takers)
➢ In other words, does the test make sense to respondents/ participants?
❖ Does the measurement appear “on its face” to measure the construct?
❖ Do the items/ questions make senses to what they are measuring?
➢ Face validity actually has no psychometric worth itself
❖ Non-expert opinion (test takers may not know the actual definition)
❖ No psychometric tests/ information required
➢ But face validity is important because if a test does not make sense:
❖ Test-takers may answer randomly or dishonestly
❖ Test-takers may misinterpret what is actually being measured
❖ They may lose motivation if they feel the test is meaningless
Content Validity
➢ The degree to which content truly reflects the intended construct
❖ Requires individuals who have a deep/ comprehensive understanding of the concept
❖ i.e. expert opinion
➢ Check if the content fully represents the relevant construct
❖ Do the items align with the conceptual definition of the construct and its components?
❖ Are all components of the construct tapped into?
o i.e., No key aspect is missing and no irrelevant content is included
➢ Assessed by experts who are familiar with the construct
❖ To critique/ judge the measure’s content
o How many items and their phrasing
o Confirm the items truly reflect the knowledge being assessed
o The response format
o How are responses coded and scored?
➢ This is often part of the “validation process” when creating a measure
❖ Often conducted by “expert teams”
Threats To Content Validity
➢ Content validity may seem somewhat simple…
❖ However, there are two key threats to content validity
1. If the test includes irrelevant content
❖ Construct-irrelevant content
❖ Content that is not directly related to the intended construct
o Decreases the validity of the test/ measure
2. If the test fails to measure the full scope of the construct
❖ Construct underrepresentation
❖ Should NOT EXCLUDE any fundamental component/ content
o Based on the theoretical literature
Internal Structure
➢ The second aspect of assessing construct validity is the internal structure
❖ The way in which parts of the test are related to each other
❖ Some tests include highly correlated items (may reflect specific components)
➢ This concerns structural validity
❖ “The match between the actual internal structure and the structure it should possess”
➢ For a test to be validly interpreted as a measure of a construct…
❖ …the ACTUAL structure should align with the THEORETICAL structure
➢ Important that the test measures all the relevant components (dimensions)
❖ Relates to the dimensionality of the construct
Dimensionality
➢ Dimensionality relates to the number of facets/ dimensions a construct includes
❖ This is based on the theoretical literature and empirical evidence
➢ Some constructs are unidimensional (i.e., they comprise a single construct)
❖ The construct only has one characteristic/ component
❖ All items on the test should explain that one component
o Unidimensional Example: Self-esteem
o Rosenberg Self-Esteem Scale has 10 items that all reflect that one dimension
➢ Some constructs are multidimensional (i.e., have different components within them)
❖ The construct explained by a number of different components
❖ Different items on the test may explain different components
o Multidimensional Example: Personality - HEXAC0 Inventory (6 dimensions)
o Openness; Honesty/Humility; Emotionality; Extroversion; Agreeableness; Conscientiousness
Dimensionality
➢ Dimensionality is important for valid interpretations of measurement scores
❖ It is also important for reliability (i.e., consistency between dimensions)
➢ There are two important considerations for dimensionality
1. We need to know the theoretical understanding of any dimensions
❖ How may dimensions actually exist within the construct? What are the dimensions?
➢ Even constructs that seem unidimensional may include multiple dimensions:
❖ Self-esteem can be measured as a unidimensional construct
❖ Some self-esteem measures may include dimensions of self-worth and social self-esteem
o In this second case, different items will reflect one of those dimensions
2. Important that the measure reflects all the relevant dimensions
❖ All dimensions are reflected in the questions/ items
❖ Items predominately only tap into one dimension each (limited overlap)
Types of Dimensionality
- Unidimensional (a single construct)
❖ All items relate to one dimension
o Average all items for one score - Multidimensional (With Correlated Dimensions)
❖ The different dimensions are correlated
❖ Explain a higher-order factor
o Average items for each dimension
o Can also compute a total score for higher-order factor - Multidimensional (With Uncorrelated Dimensions)
❖ The dimensions are not correlated (separate)
o Compute average score for each dimension
Testing Dimensionality: Factor Analysis
➢ To test the validity of the internal structure (dimensionality)
❖ Typically use factor analysis
➢ Factor analysis helps uncover how many factors exist within the scores
❖ Factors = how may dimensions are in the data
➢ There are two main types of factor analysis
❖ Exploratory Factor Analysis (EFA)
o No assumption about how many factors exist in the data
o The data indicates how many factors are present (i.e., explore how many)
❖ Confirmatory Factor Analysis (CFA)
o We assume certain items relate to different factors
o We try to confirm that items reflect different factors
➢ It can often be referred to as:
❖ Maximum likelihood factor analysis; Principal components analysis (PCA)
Exploratory Factor Analysis (EFA)
➢ EFA does not assume associations between different items
❖ Helps explore or clarify the relationships between items
❖ Identifies the total number of dimensions represented in a measure
o The number of dimensions should match the theoretical definition/ dimensions
➢ To identify the number of “factors” in the data
❖ We look at the eigenvalues (greater than 1)
➢ Example: Here two factors have eigenvalues > 1
❖ Both factors are higher than the others below
➢ Also consider the explained variance
❖ The first factors explain 36% variance each
❖ And 73% variance combined
o Including a third factors only explain 9% more
➢ Thus, two dimensions exist in this data
Scree Plot (Factors)
➢ You can also see the factors on a scree plot
❖ Eigenvalues on the y-axis (left-side)
❖ Factor Number on x-axis (bottom)
➢ Helps visually distinguish factors
Item-Factor Associations (Factor Loadings)
➢ Once we know how many factors (dimensions) exist
❖ We then look at which items are related to which factor
➢ We do this by looking at the item factor loadings
➢ Factor loadings indicate which items are associated with a factor
❖ & which items are not associated with a factor
➢ Factor loadings will range from -1 to 1
❖ They are interpreted as how much an item correlates with a factor
➢ Items load appropriately to a factor if higher than .40 (Stevens, 2012)
❖ Some authors suggest loadings greater than .30 are sometimes acceptable
o Items load very strongly if .70 or .80
➢ Items load poorly onto a factor if less than .40
❖ Not accurately measuring a factor
❖ May highlight a problematic item if not loading well
Response Process
➢ The third type of validity evidence is Response Process Validity
❖ The psychological process that respondents actually use compared to what they should use
➢ Psychologists assume that individuals go through a specific process during a test
❖ e.g., complete a task and then move to another
➢ If respondents do not follow this process, their scores may not indicate the construct
❖ i.e., they may cheat and move to the next task without completing the previous one
➢ Böckenholt (2012) propose a three step-process for how people respond to Likert scales
❖ Question/ Item = “Having lots of money is important to me”
➢ If they do not follow this process then…
❖ their scores may be interpreted incorrectly
Evaluating Response Process
➢ This type of validity is less common that others
❖ Mainly because the process to evaluate response process validity is lengthy
❖ Harder to accurately determine
Direct Evidence
➢ The most direct way is interview respondents
❖ Unpick their thought process
❖ Discuss a variety of issues relating to the test and construct
Indirect Evidence
➢ Use eye-tracking procedures
❖ Understand how respondents process stimuli when completing tests
❖ Gain a better understanding of the psychological processes that contribute to the scores
➢ Tracking “mouse-movements” and assessing response time
❖ If people take longer than expected to answer, are they distracted or confused?
