Part 2 (Final) Flashcards

1
Q

What is a latent variable

A

latent variables are variables that are not directly observed but are rather inferred (through a mathematical model) from other variables

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What are two things we are expected to measure in an observation

A

true score: the real/expected influences on our measurements
ideally, more true score than error. However, often not the case

error: undefined/unexpected influences on our measurements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Why do we want enough questions in our questionnaire when it comes to error

A

if we have enough questions measuring the construct well, we can overcome error by cancelling out the randomness of overestimation vs underestimation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

How can you reduce error

A

Combining measures

like layering all pictures that had only 10% of the information we can get a clearer idea of true score

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is reliability in the context of a questionnaire

A

In a world where our measurements are all error-prone, identifying patterns across multiple responses becomes critical. This is consistency

i.e. we need the latent variable to be influencing all responses to at least some extent producing a common pattern in the responses
the influence of the latent variable is the true score
any other influence is error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the classical measurement model

A

The classical measurement model has 3 key assumptions
remember: an assumption is something we expect to be true

  1. The individual items of a questionnaire each have error and true score. The amount of error in any given item varies randomly. The mean error across items is zero (given a sufficient N)
  2. The error in one item is not correlated with the error in any other
  3. The error in the items is not correlated with the true score
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is the parallel test model

A

Extends the classical measurement model with 2 more assumptions to be more practical

  1. The latent variable influences all items equally
    all item-construct correlations are the same

2.Each items has the same quantity of random error
the combined influences of all other factors are the same

Each item is true score + error, so if you reduce the amount of true score (latent variable influence) then you would increase the amount of error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Name the 5 assumptions of the parallel test model

A
  1. Only random errors
  2. Errors are not correlated with each other
  3. Errors are not correlated with the true score
  4. The latent variable affects all items equally
  5. The amount of random error for each item is equal
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What is the essentially tau-equivalent model

A

to avoid violating our assumptions it’s helpful to loosen them a bit

  1. Only random errors
  2. Errors are not correlated with each other
  3. Errors are not correlated with true score
  4. The latent variable affects all items equally only when standardized
    differences are due to constants
    e.g. all questions are turned into z-scores, we might have different response formats across our questions. If we do not convert them into z-scores, it might not look like they are all affected equally
  5. The amount of random error for each item is not necessarily equal
    partly a consequence of not standardizing
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

If you are using a mix of Likert-type, analog and dichotomous questions. Which model is better for you?

A

Essentially Tau-Equivalent Model because less likely to violate our assumptions

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What is the congeneric model

A

The congeneric model is much less strict

  1. Random error is only preferred, but not necessary
  2. Errors are preferably not correlated with each other, but can be
  3. Errors are not correlated with the true score
  4. The latent variable affects all items in some way
  5. The amount of random error for each item is not necessarily equal
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Compare the models in strictness

A
  1. Starting points: classical measurement model + parallel test
  2. Common & somewhat strict: essentially tau-equivalent + congeneric
  3. Much less strict: general factor (allows for multiple latent variables in each measure)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What are additional assumptions that correlation analysis will add to our model

A
  1. you have interval-level data
    probably not true, usually ordinal even if close to interval
  2. your data follow a normal distribution
    not too well, without interval data
  3. a straight line is the best way to represent the relation
    this is probably true
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Explain the assumption of linearity

A

If the line of best fit should be u-shaped, you have a big problem

A straight line fit to these data would be flat indicating no relation is present
luckily, for reliability, we’re talking about measuring the same construct across two different questions, so it’s quite unlikely for their relation not to follow a straight line

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Name the two deviations from normality

A

Skewness: the presence of a longer than normal tail

Kurtosis: the presence of a taller or wider than normal spread

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What does deviation from normality threatens

A

there are two main categories of deviation from normal (skewness and kurtosis), and they both threaten the validity of all these models

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Describe skewness

A

Either the left or the right tail could be pulled out
Skewness means you distribution has asymmetry

A negative skewness means the left tail is long
A positive skewness means the right tail is long

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Describe kurtosis

A

The peak can be pulled up or pushed down
There are two kinds of kurtosis as well

a negative kurtosis means the curve is falter
a positive kurtosis means the curve is taller

Assumes symetrical distribution, something is wrong on both sides of the distribution

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

How do we assess whether kurtosis and skewness are too much

A

We assume a normal distribution, but don’t typically get it in practice
there will be some degree of skew
there will be some degree of kurtosis

If either score are +/- 3, then it indicates it might be too much of a problem

OR

Multiply the standard error (SE) by 3 and if the skewness and kurtosis are bigger than the number, then it’s too much
For that to make sense, you need a fairly small sample size, since as n increase, SE decrease, but overall assumptions are better

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

How can large error be problematic for reliability

A

extremely large amounts of error will prevent you from observing any interesting associations (because it’s random)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

How do we determine whether we have too much error?

A

We estimate it by correlating a measure with itself

Internal consistency, whatever is not true score must be error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Name the 5 form of internal consistency measures

A
  1. Cronbach’s alpha
  2. Split-half
  3. Test-retest
  4. Alternate Forms
  5. Omega
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

How common is Cronbach’s alpha

A

Most commonly reported

Easy to use with jamovi (and SPSS)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Briefly describe Split-half

A

Less common than alpha
Easy to use too (but only manually)
Usually, split by odd and even numbers. Correlate points from even questions to odd questions

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Briefly describe Test-retest
Optimal, when possible Require twice the resources only really good for traits, not so good for states
26
Briefly describe Alternate forms
Even less common Requires two identical measures make two different versions of the questionnaire that measure exactly the same thing. Alternate who gets A and B, and then assess the second version later and the higher the correlation, the more reliable
27
Briefly describe omega
Newest form No tau or error equivalent is required (true score + error) based on congeneric model (loose model requiring a single latent variable) harder to violate it’s assumption, but harder to calculate, which is fine with jamovi more than one calculation for omega
28
Describe the key assumptions of Cronbach's alpha
Based on Essential Tau-equivalence model, depending on how it’s calculated, so is built on several key assumptions 1. There is a single latent variable 2. Error is always radom if not true, will affect how we believe our questionnaire to be reliable 3. True score influences items equivalently 4. Inter-item correlations are equivalent 5. All inter-items correlations would be equal in a large enough sample in an infinitely large sample 6. Items would have equal variability in a large enough sample 7. More items will estimate true score and error better
29
How do you interpret a chronbach's alpha
Like correlation coefficient that it is between -1 to 1, but you do not want to see negative For research purposes, a good alpha is a = .70 or better For other purposes, a good alpha is a = .80 r better (and it may need to be even higher for diagnosis) We won’t hit 1 since we are going to violate the assumption to some extent Can be thought of as the proportion of true score that made its way through the measure if a = 0.8 → 80% true score is captured by the measure
30
how can you reach a large alpha score
Either high inter-item correlation OR large correction (large sample size) Alpha tends to increase when you add more items of equivalent value
31
What will likely happen if you make your questionnaire too long
while it might seem like good reliability can be achieved simply by making very long questionnaires, this isn’t really true Remember, when we make our questionnaires too long, people lose interest in our questions and that changes the average inter-item correlation as k increase, rbar most likely will decrease
32
What is a common mistake with reliability measures
You can never forget that reliability is NOT a property of a measure itself How much true score is captured by a measure depends on the sample of people who completed it some samples may interpret items differently, affecting error It’s not sufficient to simply quote a previously published reliability statistic - you can start from there, but verify for your sample generational understanding of the wording can affect you sample, people change across time thus affecting reliability
33
How can you resolve problems with a low alpha
When alpha is lower than we would like, or even if it’s not, you can consider improving it by looking at whether to drop bad items Whenever an item is not really ‘equivalent’ to the others, as alpha assumes, then dropping it will change alpha if it has a low inter-item correlation, alpha goes up if it has a high inter-item correlation, alpha goes down But a questionnaire with more “bad” items can still be better overall than one with a small number of ‘good’ items
34
Why is omega a better alternative than alpha
McDonald’s Omega is a better choice in almost every case (possibly all) and should be the go-to reliability statistic from now on Base on the congeneric model; easier to meet restrictions 1. interpretation is the same as for alpha (easy replacement) 2. not all items need to be positively correlated (but still a good idea) 3. not all correlations need to be equivalent 4. when none of alpha’s assumptions are violated, it gives the exact same results; when they are, it is less of an underestimation
35
What is validity
the state of being valid; the degree to which we are measuring whatever it was that we wanted to measure like reliability, this isn’t simply a property of the measure itself for a measure to have validity, we must consider whether it’s being used in the correct context must be sure we are using it with the correct type of people
36
What are the three main form of validity we should assess
Content validity: key issue is deciding what should be included Criterion validity: key issue is deciding whether the measure ‘works’ usually applied in a predictive sense, can I predict the outcome Construct validity: key issue is complex Content and construct validity are always important, criterion validity only in certain context
37
What is content validity
This form is easy to understand, but hard to be sure you’ve achieved it to be sure you’ve represented the full range of possible content, you need to have an excellent understanding of what needs to be asked what are all the key thoughts, behaviours, skills, etc. have you represented their relative importance properly? how much of each aspect goes into the total score for the measure? Example: the midterms for this course try to address the full range of important topics discussed
38
How can you assess content validity
Successful representation of all the critical aspects of the construct is usually determined by two things adherence with prior literature review expert review of your measure
39
What is criterion validity
This form requires you to evaluate how well scores on your measure watch with an accepted ‘gold standard’ or tangible outcome assessment. For example delinquency assessment: correlate your measure with the number of offences committed job suitability: correlate your measure with ratings of job performance graduate records exam: correlate with grad school GPA
40
What is construct validity
Two separate definitions An overall category that encompassess all other forms of validity Assessing the relation of your measure with other theoretically-relevant measures
41
What are the two sub categories of construct validity
convergent validity: do you have expected relations to theoretically related constructs? discriminant validity: do you have the expected lack of relation to theoretically unrelated constructs?
42
What is convergent validity
To properly establish convergent validity, you need to be sure the related constructs do not themselves contain facets of the construct you’re measuring. For example: Mindfulness should show a positive correlation with the Openness to Experience sub construct of the Big5 personality measure a a negative correlation with neuroticism ADHD should show positive correlations with Memory Failures, and negative correlations with Mindfulness convergent means a relationship: either positive or negative while discriminant should show NO relationship
43
What is discriminant validity
to properly establish discriminant validity, you need to have good reasons for choosing the potentially unrelated (or very weakly related) constructs, For example Intelligence is only weakly related to Artistic Ability - this is a theoretically relevant, not random choice Memory Failures show no relation to Internal Locus of Control - this could be important as both LOC and MF are typically related to Depression
44
What is face validity
The overlooked middle child (by psychometricians) of validity addresses the question of whether the measure seems to measure the right thing It is critical to at least consider whether there are other reasonable interpretations Outside of psychometrics, this form is frequently used as it’s the easiest to claim and not based on correlation The basic idea is: any reasonable person would say you measured the right thing
45
What is internal validity
Strongly related to the specific ways of establishing validity. It’s simply the extent to which you can account for other plausible explanations You need to either rule out alternatives, or somehow argue they’re not plausible after all Error, particularly biased responding, will again be concern as it increases the likelihood of threats to internal validity
46
Name 6 types of systematic error
1. order effects: practice, fatigue, boredom, context effects, etc. 2. motivation: we could have incomplete data, change context effects can lead a to more response set 3. distraction: similar issues to a lack of motivation 4. sampling bias: when we haven’t sampled the right people 5. maturation: mainly problematic for predictive criterion validity 6. instrumentation: when our questions get less useful over time
47
What does systematic error affect
some of the common sources of systematic error that affect only validity Reliability is concerned with random error, while validity is also concerned with systematic error
48
How should we mesure validity
Aside from face validity, we use correlations between our measure and another measure to quantify and demonstrate validity We need to anticipate more modest correlations for validity than we would for reliability, however the expected size should be based on prior literature and theory an r up to .10 is good for showing discriminant validity an r of about .60 is very good for showing convergent validity. between .2 and .7 are usually cutoffs an r of about .85 or higher is usually very bad for showing convergent validity, too close to 1 and would be hard to say they are actually two different concepts
49
If systemic error increases, what happens to validity measures
convergent validity you are likely to over- or underestimate the correlation discriminant validity you are likely to get a higher correlation than you would want
50
What are major flaws of Brown & Ryan's questionnaire
all of the items are reverse scored The MAAS items capture only one part of the full construct of mindfulness
51
In practice how much error should expect
More often than not, we have much less true score than error
52
What does it mean to have "enough signal to noise"
Ideally, you need enough true score (signal) to overcome the randomness of error (noise)
53
What is another name for the Essentially Tau-equivalent Model
also called essentially equivalent to true score model
54
How is alpha calculated
there are many ways to calculte it. Often uses essentially tau-equivalent model, but jamovi uses a mix with parallel test model
55
In a negatively and positively skewed distribution where can you find the measured of central tendency
from left to right Negatively skewed: mean median mode Positively skewed: mode, median, mean
56
Why is it bad to have too much skewness
High levels of skew is a source of error that will impact the accuracy of our correlations
57
What is a caveat of the kurtosis measurement
It’s possible to calculate a big positive or negative kurtosis measure without a normally shaped distribution
58
What is a platykurtic distribution
flat distribution
59
What is a leptokurtic distribution
too skinny distribution
60
The different threats of systematic error are likely to compromise which type of validity
internal validity