Part 2 (Final)

Question 1

What is a latent variable

Answer 1

latent variables are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables

Question 2

What are two things we are expected to measure in an observation

Answer 2

true score: the real/expected influences on our measurements
ideally, more true score than error. However, often not the case

error: undefined/unexpected influences on our measurements

Question 3

Why do we want enough questions in our questionnaire when it comes to error

Answer 3

if we have enough questions measuring the construct well, we can overcome error by cancelling out the randomness of overestimation vs underestimation

Question 4

How can you reduce error

Answer 4

Combining measures

like layering all pictures that had only 10% of the information we can get a clearer idea of true score

Question 5

What is reliability in the context of a questionnaire

Answer 5

In a world where our measurements are all error-prone, identifying patterns across multiple responses becomes critical. This is consistency

i.e. we need the latent variable to be influencing all responses to at least some extent producing a common pattern in the responses
the influence of the latent variable is the true score
any other influence is error

Question 6

What is the classical measurement model

Answer 6

The classical measurement model has 3 key assumptions
remember: an assumption is something we expect to be true

1. The individual items of a questionnaire each have error and true score. The amount of error in any given item varies randomly. The mean error across items is zero (given a sufficient N)
2. The error in one item is not correlated with the error in any other
3. The error in the items is not correlated with the true score

Question 7

What is the parallel test model

Answer 7

Extends the classical measurement model with 2 more assumptions to be more practical

1. The latent variable influences all items equally
   all item-construct correlations are the same

2.Each items has the same quantity of random error
the combined influences of all other factors are the same

Each item is true score + error, so if you reduce the amount of true score (latent variable influence) then you would increase the amount of error

Question 8

Name the 5 assumptions of the parallel test model

Answer 8

1. Only random errors
2. Errors are not correlated with each other
3. Errors are not correlated with the true score
4. The latent variable affects all items equally
5. The amount of random error for each item is equal

Question 9

What is the essentially tau-equivalent model

Answer 9

to avoid violating our assumptions it’s helpful to loosen them a bit

1. Only random errors
2. Errors are not correlated with each other
3. Errors are not correlated with true score
4. The latent variable affects all items equally only when standardized
   differences are due to constants
   e.g. all questions are turned into z-scores, we might have different response formats across our questions. If we do not convert them into z-scores, it might not look like they are all affected equally
5. The amount of random error for each item is not necessarily equal
   partly a consequence of not standardizing

Question 10

If you are using a mix of Likert-type, analog and dichotomous questions. Which model is better for you?

Answer 10

Essentially Tau-Equivalent Model because less likely to violate our assumptions

Question 11

What is the congeneric model

Answer 11

The congeneric model is much less strict

1. Random error is only preferred, but not necessary
2. Errors are preferably not correlated with each other, but can be
3. Errors are not correlated with the true score
4. The latent variable affects all items in some way
5. The amount of random error for each item is not necessarily equal

Question 12

Compare the models in strictness

Answer 12

1. Starting points: classical measurement model + parallel test
2. Common & somewhat strict: essentially tau-equivalent + congeneric
3. Much less strict: general factor (allows for multiple latent variables in each measure)

Question 13

What are additional assumptions that correlation analysis will add to our model

Answer 13

1. you have interval-level data
   probably not true, usually ordinal even if close to interval
2. your data follow a normal distribution
   not too well, without interval data
3. a straight line is the best way to represent the relation
   this is probably true

Question 14

Explain the assumption of linearity

Answer 14

If the line of best fit should be u-shaped, you have a big problem

A straight line fit to these data would be flat indicating no relation is present
luckily, for reliability, we’re talking about measuring the same construct across two different questions, so it’s quite unlikely for their relation not to follow a straight line

Question 15

Name the two deviations from normality

Answer 15

Skewness: the presence of a longer than normal tail

Kurtosis: the presence of a taller or wider than normal spread

Question 16

What does deviation from normality threatens

Answer 16

there are two main categories of deviation from normal (skewness and kurtosis), and they both threaten the validity of all these models

Question 17

Describe skewness

Answer 17

Either the left or the right tail could be pulled out
Skewness means you distribution has asymmetry

A negative skewness means the left tail is long
A positive skewness means the right tail is long

Question 18

Describe kurtosis

Answer 18

The peak can be pulled up or pushed down
There are two kinds of kurtosis as well

a negative kurtosis means the curve is falter
a positive kurtosis means the curve is taller

Assumes symetrical distribution, something is wrong on both sides of the distribution

Question 19

How do we assess whether kurtosis and skewness are too much

Answer 19

We assume a normal distribution, but don’t typically get it in practice
there will be some degree of skew
there will be some degree of kurtosis

If either score are +/- 3, then it indicates it might be too much of a problem

OR

Multiply the standard error (SE) by 3 and if the skewness and kurtosis are bigger than the number, then it’s too much
For that to make sense, you need a fairly small sample size, since as n increase, SE decrease, but overall assumptions are better

Question 20

How can large error be problematic for reliability

Answer 20

extremely large amounts of error will prevent you from observing any interesting associations (because it’s random)

Question 21

How do we determine whether we have too much error?

Answer 21

We estimate it by correlating a measure with itself

Internal consistency, whatever is not true score must be error

Question 22

Name the 5 form of internal consistency measures

Answer 22

1. Cronbach’s alpha
2. Split-half
3. Test-retest
4. Alternate Forms
5. Omega

Question 23

How common is Cronbach’s alpha

Answer 23

Most commonly reported

Easy to use with jamovi (and SPSS)

Question 24

Briefly describe Split-half

Answer 24

Less common than alpha
Easy to use too (but only manually)
Usually, split by odd and even numbers. Correlate points from even questions to odd questions

Question 25

Briefly describe Test-retest

Answer 25

Optimal, when possible
Require twice the resources
only really good for traits, not so good for states

Question 26

Briefly describe Alternate forms

Answer 26

Even less common
Requires two identical measures
make two different versions of the questionnaire that measure exactly the same thing. Alternate who gets A and B, and then assess the second version later and the higher the correlation, the more reliable

Question 27

Briefly describe omega

Answer 27

Newest form
No tau or error equivalent is required (true score + error)
based on congeneric model (loose model requiring a single latent variable)
harder to violate it’s assumption, but harder to calculate, which is fine with jamovi
more than one calculation for omega

Question 28

Describe the key assumptions of Cronbach’s alpha

Answer 28

Based on Essential Tau-equivalence model, depending on how it’s calculated, so is built on several key assumptions

1. There is a single latent variable
2. Error is always radom
   if not true, will affect how we believe our questionnaire to be reliable
3. True score influences items equivalently
4. Inter-item correlations are equivalent
5. All inter-items correlations would be equal in a large enough sample
   in an infinitely large sample
6. Items would have equal variability in a large enough sample
7. More items will estimate true score and error better

Question 29

How do you interpret a chronbach’s alpha

Answer 29

Like correlation coefficient that it is between -1 to 1, but you do not want to see negative

For research purposes, a good alpha is a = .70 or better

For other purposes, a good alpha is a = .80 r better (and it may need to be even higher for diagnosis)

We won’t hit 1 since we are going to violate the assumption to some extent

Can be thought of as the proportion of true score that made its way through the measure
if a = 0.8 → 80% true score is captured by the measure

Question 30

how can you reach a large alpha score

Answer 30

Either high inter-item correlation OR large correction (large sample size)

Alpha tends to increase when you add more items of equivalent value

Question 31

What will likely happen if you make your questionnaire too long

Answer 31

while it might seem like good reliability can be achieved simply by making very long questionnaires, this isn’t really true

Remember, when we make our questionnaires too long, people lose interest in our questions and that changes the average inter-item correlation
as k increase, rbar most likely will decrease

Question 32

What is a common mistake with reliability measures

Answer 32

You can never forget that reliability is NOT a property of a measure itself
How much true score is captured by a measure depends on the sample of people who completed it
some samples may interpret items differently, affecting error

It’s not sufficient to simply quote a previously published reliability statistic - you can start from there, but verify for your sample
generational understanding of the wording can affect you sample, people change across time thus affecting reliability

Question 33

How can you resolve problems with a low alpha

Answer 33

When alpha is lower than we would like, or even if it’s not, you can consider improving it by looking at whether to drop bad items
Whenever an item is not really ‘equivalent’ to the others, as alpha assumes, then dropping it will change alpha

if it has a low inter-item correlation, alpha goes up

if it has a high inter-item correlation, alpha goes down

But a questionnaire with more “bad” items can still be better overall than one with a small number of ‘good’ items

Question 34

Why is omega a better alternative than alpha

Answer 34

McDonald’s Omega is a better choice in almost every case (possibly all) and should be the go-to reliability statistic from now on

Base on the congeneric model; easier to meet restrictions

1. interpretation is the same as for alpha (easy replacement)
2. not all items need to be positively correlated (but still a good idea)
3. not all correlations need to be equivalent
4. when none of alpha’s assumptions are violated, it gives the exact same results; when they are, it is less of an underestimation

Question 35

What is validity

Answer 35

the state of being valid; the degree to which we are measuring whatever it was that we wanted to measure
like reliability, this isn’t simply a property of the measure itself

for a measure to have validity, we must consider whether it’s being used in the correct context
must be sure we are using it with the correct type of people

Question 36

What are the three main form of validity we should assess

Answer 36

Content validity: key issue is deciding what should be included

Criterion validity: key issue is deciding whether the measure ‘works’
usually applied in a predictive sense, can I predict the outcome

Construct validity: key issue is complex

Content and construct validity are always important, criterion validity only in certain context

Question 37

What is content validity

Answer 37

This form is easy to understand, but hard to be sure you’ve achieved it
to be sure you’ve represented the full range of possible content, you need to have an excellent understanding of what needs to be asked
what are all the key thoughts, behaviours, skills, etc.
have you represented their relative importance properly?
how much of each aspect goes into the total score for the measure?
Example: the midterms for this course try to address the full range of important topics discussed

Question 38

How can you assess content validity

Answer 38

Successful representation of all the critical aspects of the construct is usually determined by two things
adherence with prior literature review
expert review of your measure

Question 39

What is criterion validity

Answer 39

This form requires you to evaluate how well scores on your measure watch with an accepted ‘gold standard’ or tangible outcome assessment.

For example
delinquency assessment: correlate your measure with the number of offences committed

job suitability: correlate your measure with ratings of job performance

graduate records exam: correlate with grad school
GPA

Question 40

What is construct validity

Answer 40

Two separate definitions
An overall category that encompassess all other forms of validity

Assessing the relation of your measure with other theoretically-relevant measures

Question 41

What are the two sub categories of construct validity

Answer 41

convergent validity: do you have expected relations to theoretically related constructs?

discriminant validity: do you have the expected lack of relation to theoretically unrelated constructs?

Question 42

What is convergent validity

Answer 42

To properly establish convergent validity, you need to be sure the related constructs do not themselves contain facets of the construct you’re measuring.

For example:
Mindfulness should show a positive correlation with the Openness to Experience sub construct of the Big5 personality measure a a negative correlation with neuroticism

ADHD should show positive correlations with Memory Failures, and negative correlations with Mindfulness

convergent means a relationship: either positive or negative while discriminant should show NO relationship

Question 43

What is discriminant validity

Answer 43

to properly establish discriminant validity, you need to have good reasons for choosing the potentially unrelated (or very weakly related) constructs, For example

Intelligence is only weakly related to Artistic Ability - this is a theoretically relevant, not random choice

Memory Failures show no relation to Internal Locus of Control - this could be important as both LOC and MF are typically related to Depression

Question 44

What is face validity

Answer 44

The overlooked middle child (by psychometricians) of validity addresses the question of whether the measure seems to measure the right thing

It is critical to at least consider whether there are other reasonable interpretations

Outside of psychometrics, this form is frequently used as it’s the easiest to claim and not based on correlation
The basic idea is: any reasonable person would say you measured the right thing

Question 45

What is internal validity

Answer 45

Strongly related to the specific ways of establishing validity. It’s simply the extent to which you can account for other plausible explanations
You need to either rule out alternatives, or somehow argue they’re not plausible after all
Error, particularly biased responding, will again be concern as it increases the likelihood of threats to internal validity

Question 46

Name 6 types of systematic error

Answer 46

1. order effects: practice, fatigue, boredom, context effects, etc.
2. motivation: we could have incomplete data, change context effects
   can lead a to more response set
3. distraction: similar issues to a lack of motivation
4. sampling bias: when we haven’t sampled the right people
5. maturation: mainly problematic for predictive criterion validity
6. instrumentation: when our questions get less useful over time

Question 47

What does systematic error affect

Answer 47

some of the common sources of systematic error that affect only validity

Reliability is concerned with random error, while validity is also concerned with systematic error

Question 48

How should we mesure validity

Answer 48

Aside from face validity, we use correlations between our measure and another measure to quantify and demonstrate validity
We need to anticipate more modest correlations for validity than we would for reliability, however
the expected size should be based on prior literature and theory

an r up to .10 is good for showing discriminant validity

an r of about .60 is very good for showing convergent validity. between .2 and .7 are usually cutoffs

an r of about .85 or higher is usually very bad for showing convergent validity, too close to 1 and would be hard to say they are actually two different concepts

Question 49

If systemic error increases, what happens to validity measures

Answer 49

convergent validity
you are likely to over- or underestimate the correlation

discriminant validity
you are likely to get a higher correlation than you would want

Question 50

What are major flaws of Brown & Ryan’s questionnaire

Answer 50

all of the items are reverse scored

The MAAS items capture only one part of the full construct of mindfulness

Question 51

In practice how much error should expect

Answer 51

More often than not, we have much less true score than error

Question 52

What does it mean to have “enough signal to noise”

Answer 52

Ideally, you need enough true score (signal) to overcome the randomness of error (noise)

Question 53

What is another name for the Essentially Tau-equivalent Model

Answer 53

also called essentially equivalent to true score model

Question 54

How is alpha calculated

Answer 54

there are many ways to calculte it. Often uses essentially tau-equivalent model, but jamovi uses a mix with parallel test model

Question 55

In a negatively and positively skewed distribution where can you find the measured of central tendency

Answer 55

from left to right

Negatively skewed: mean median mode

Positively skewed: mode, median, mean

Question 56

Why is it bad to have too much skewness

Answer 56

High levels of skew is a source of error that will impact the accuracy of our correlations

Question 57

What is a caveat of the kurtosis measurement

Answer 57

It’s possible to calculate a big positive or negative kurtosis measure without a normally shaped distribution

Question 58

What is a platykurtic distribution

Answer 58

flat distribution

Question 59

What is a leptokurtic distribution

Answer 59

too skinny distribution

Question 60

The different threats of systematic error are likely to compromise which type of validity

Answer 60

internal validity