Part 3 (Final)

Question 1

Name 4 areas where it is important to follow ethical guidelines in psychological assessment

Answer 1

test/questionnaire selection
scoring & interpretation
participant recruitment
treatment of participants

Question 2

What does treatment of participant ethical guideline entail

Answer 2

Responsible treatment of participants
Even decisions made prior to seeing any participants are part of their ethical treatment
e.g. understand your measures, follow appropriate procedures, ensure participant wellbeing

Question 3

What is included in informed consent

Answer 3

It includes things like estimating the time commitment and informing them about their right to withdraw, and your goals but not hypotheses.

Question 4

Name some elements of ethical guideline “basics of wellbeing”

Answer 4

1. informed consent
2. assured confidentiality, as much as possible
3. deliver feedback appropriately

Question 5

What is included in confidentiality

Answer 5

this includes you practicing responsible handling of data

open-access data must be anonymous, people must be aware of the broad distribution of data, must plan in advance

Question 6

What is included in deliver feedback appropriately

Answer 6

ideally, there should be some benefit to the participants

usually, we cannot give them any scores if we do not understand the measure already

Question 7

What are some regulations around children in research

Answer 7

Children 14-18 participating in research must be towards a project that directly benefits them or children like them

Children under 14 must have parental approval, parents and children can withdraw at any time

Question 8

Define Responsible Data Handling

Answer 8

No unauthorized access to your data, this includes the government

Safe retention of data
anonymity goes a long way here - don’t collect personally identifying information if it’s not needed
traditionally, keep for 5 years then destroy (not necessarily anymore)

In testing, ensure the results won’t be interpreted inappropriately

Question 9

Define Responsible Data Removal

Answer 9

Data removal is allowed, but needs to be done responsibly as well

Balancing need to respect participants’ contribution with the need to reach accurate conclusions

Participants have the right to try to derail our studies, but not the right to succeed in doing so

Remember, our statistics have assumptions we should meet many statistics require complete data sets

Question 10

What are the assumptions of the factor analysis

Answer 10

Based on general factor model, factor analysis shares some important assumptions with more classical measurement models

1. Errors are random and not correlated with the latent variable
2. Correlations among items exist because they share a common latent variable(s)

Question 11

What is a factor

Answer 11

Factor is another way we could refer to a latent variable

Question 12

What is a component

Answer 12

Component is another way we could refer to a latent variable

Question 13

Why would we use the words “factor” or “component” instead of “latent variable”?

Answer 13

The main reason to use these terms instead of latent variable is to better acknowledge that the unobserved influence was derived empirically

Question 14

How can you identify factors

Answer 14

Factor analysis is a process of trying to identify the latent variable(s) that influenced our measurements

Items that have stronger associations (correlations) with each other but weaker associations with other items will form identifiable clusters

we’re capitalizing on similarities and differences in correlations across items

if all items correlate strongly, there will be only one factor identified

Question 15

What is the main element to have a good factor analysis

Answer 15

We need quite large data sets for factor analysis to give accurate results: larger than what we need for good internal consistency reliability or validity analyses
we should have 50% more data for factor analysis than for reliability/validity analyses (around 300-400) to have a meaningful factor analysis

Question 16

What should you keep in mind as you run factor analyses (4)

Answer 16

1. What you put into the analysis dictates what you get out of it
2. Every item has the potential to create a factor, and influence the creation of other factors
3. Adding or dropping even one item will change the outcome
4. Factor analysis should be viewed as a process requiring many iterations, thus it is time consuming

Question 17

When should a cluster be called a factor

Answer 17

Technically, we always find more than one ‘cluster’ or pattern of responses in a questionnaire - only the important ones get called factors

Researchers typically select as factors any components with an eigenvalue > 1.0

The eigenvalue is a measure of the amount of information captured by an item

An eigenvalue of 1 is generally seen as indicating the factor capture as much information as one typical (good) item

Question 18

What is a parallel analysis

Answer 18

Parallel analysis created random data with the same number of variables and observations as your data. A correlation matrix is created with the random data and then eigenvalues are calculated

When the eigenvalues from the random data are larger than the eigenvalues for your real data, you know the variables in the factors are not correlated better than random noise

Question 19

How do you interpret a scree plot based on eigenvalues against a parallel analysis

Answer 19

All the eigenvalues are plotted, and so are the stimulated eigenvalues
Here the simulated eigenvalues comes from the randomly generated Parallel Analysis data set
What jamovi will call a factor is any blue dot that is before the first time a yellow dot rises above the blue dots
Factors are just numbered sequentially, after sorting the eigenvalues largest-to-smallest

Question 20

How do you interpret a traditional scree plot

Answer 20

The same eigenvalues are plotted, it’s just the comparison line that is different
Looking at the scree plot, where do you think we change mountain to ‘scree’
The number of factors identified to the left of where we think the screen starts is how many we should keep

Question 21

What are some changes you can do to an exploratory analysis that will make a difference

Answer 21

changes worth considering: dropping one or more item(s), adding a new item, using rotation

Question 22

What are some changes you can do to an exploratory analysis that will not make a difference

Answer 22

changes that won’t impact anything: removing participants with missing data, reverse-scoring

Question 23

Briefly describe the exploratory factor analysis

Answer 23

Exploratory factor analysis is iterative, you repeat until a good solution arises
Identifying the number of factors is based on eigenvalue, using whatever method makes sense

Question 24

Explain the different rotations for EFA

Answer 24

Rotation allows us to spread the variability among our factors more evenly
There are two main forms of rotation worth considering

Orthogonal: this maximizes the squared variance in the factor loadings - clusters are as different as possible, unrelated

Oblique: this maintains some relationship between the factors-clusters are different but related things

Question 25

What is the varimax rotation for EFA

Answer 25

tried to maximize the differences between your clusters by rotating the factor axes
This is called varimax or quartimax rotation in jamovi
Varimax rotation is very commonly used. It makes it easier to identify the differences between clusters of items (or which items best represent which cluster)

Question 26

What is the oblique rotation for EFA

Answer 26

Tries to maintain some of the association between your clusters (and measure it) while rotating the factor axes
This is called oblimin or promax rotation in jamovi

Oblimin rotation is used to minimize the squared loading covariance of the factors, while allowing them to be correlated
as different as possible while still being related

This oblimin rotation is, obviously, not an orthogonal rotation

This means the horizontal and vertical axis (for 2 factors) aren’t at 90 degrees
The strength of the correlation dictates the extent of the departure from a 90-degree angle

The results obtained in jamovi would look the same as with a varimax rotation, or even an unrotated solution, but it will have different factor loadings

Question 27

Where do we expect the variance to come from for EFA

Answer 27

Multiple sources of variance

In Exploratory Factor Analysis we expect the latent variable(s) to be only part of the forces acting on our measurements. The other influences are probably just random error, though.

That means the eigenvalues we calculate are affected by our conception of how to attribute the sources of variance

Question 28

What is the principle component analysis

Answer 28

In Principle Component Analysis we expect all variance to come from common sources among the items, through not necessarily to an equal extent in each

This means there is no choice of extraction method, as these options under EFA are based on different ideas of the sources of variance

It’s quite possible for some of the common variance to still be error, though; accordingly, we still want to discard any ‘factors’ that seem too trivial - on the basis of their eigenvalues

Question 29

When should you use PCA or EFA

Answer 29

Technically, PCA is not considered factor analysis. You should use it if you want to maximize the amount of explained variance

If someone asks you to do a factor analysis, you should do EFA

Question 30

Describe the EFA and the CFA

Answer 30

Exploratory Factor Analysis (EFA) allows very little control over the latent variables, theories are introduced through rotation or forcing a set number of factors

Confirmatory Factor Analysis (CFA) requires you to exercise control over which items theoretically go with which latent variable; with the right software, not jamovi (yet), you could even specify expectations for how strong the relations should be

Question 31

What should be the process for CFA

Answer 31

If you go back and look at the process for EFA, it has a very similar structure
The main difference here is that we need an a priori theory and we need to check that theory
The examination of the model fit dictates whether we are done or need to revise our model or our theory

Question 32

What statistics should you look at in a CFA

Answer 32

use standardized estimates, preferable, recognizing that each of the factor loadings should be significant (p < .05)

examine the chi squared as well as the CFL, TLI & RMSEA statistics

you want the chi squared to be non-significant, though that’s unlikely

you want the CFI & TLI to be above .95 (below .90 is very bad)

you want the RMSEA to be below .05 (not more than .08 upper CI)

Question 33

Give the definitions for standardize (3)

Answer 33

1. turning into z-scores
2. norming
3. an expected set of procedures, or that the items on a questionnaire have been previously established and we didn’t change them, that a measure has established normative information available, or some combination of all these. Unstandardized means one or more of these is missing

Question 34

What is a proprietary measure, which are standardized vs unstandardized

Answer 34

Another big difference between many standardized and unstandardized ones are proprietary. That is, you have to pay to use them.

Standardized: Controlled administration conditions, normative information available, interpretation guidelines available, instruction manual available, carefully vetted (?), and probably costly

Unstandardized: May not require controlled conditions, typically lacks normative information, less interpretable for giving feedback, mainly useful for research purposes

Question 35

Are proprietary measures of personality better?

Answer 35

for measuring personality, free tests are better: more reliable and efficient

Question 36

What is a hierarchical factor analysis

Answer 36

You may recall I mentioned that it’s possible to theorize and (with different software) test multiple levels of latent variables. There’s a name for this.

Hierarchical Factor Analysis is one where you have proposed one or more latent variable(s) causing changes in one or more other latent variable(s)

The idea here is you have a primary latent variable that influences the other latent variables

Question 37

Why use a hierarchical factor analysis

Answer 37

We’re often focused on measuring just one factor, but this isn’t always what we want.

A test for a course may not capture just one construct
An IQ test captures G (overall intelligence), but it also has multiple components within it

A personality “test” assesses many aspects of personality; the Big 5: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism
they aren’t really tests, they are questionnaire

Question 38

What inter-item correlations should we expect between latent variables in a hierarchical factor analysis

Answer 38

above .3, no more than .8

Question 39

Why use experiments

Answer 39

In experimental research we are usually trying to manipulate something, and to observe the effect of that manipulation on an outcome.

Many studies use performance on some task as their outcome

Often, they just compare two different groups of people as the “manipulation”
Very rarely are psychometrics evaluated in these contexts

Question 40

Why should we evaluate the psychometric properties of experiments

Answer 40

If we don’t, we may:

Making less-than-optimal design decisions (e.g., power, task selected)
Failure to find predicted results
Incorrectly interpreting results

Question 41

Define discrimination, reliability and validity in a cognitive task

Answer 41

Discrimination: Scores aren’t entirely random (e.g., high/low performance by some individuals)
similar to good distribution and range from questionnaire, can be thought of as criterion validity

Reliability: Scores are a precise estimate of one’s ability

Validity: Scores reflect the right cognitive faculty

Question 42

What is the risk of assuming the same reliability for control and treatment groups

Answer 42

The control group may not have the same reliability, because of response biases

Assuming reliability would be equal could lead to incorrect conclusions

Question 43

What is the difference between tests and questionnaires

Answer 43

For tests, items have right and wrong answers, whereas for questionnaires they do not (except with respect to the truth of the response for that individual).
validity, reliability but also suitability of purpose is important for tests

We are still interested in psychometric qualities for tests, but not all concepts apply equally well:
Internal consistency reliability may not make much sense

This depends on how many factors we expect to capture using the test, and how strongly correlated the factors are

Question 44

Define discrimination and difficulty for tests

Answer 44

For proper tests, there are two important statistics to consider – in addition to reliability and validity.

Discrimination: The ability to accurately separate individuals into high/low performers
Discrimination is based on a relatively simple formula, especially compared with everything else we’ve seen.
Discrimination = Ph/Nh - Pl/Nl

Difficulty: The likelihood of getting a question correct

Ph = People responding correctly (H) 
Pl = People responding correctly (L) 
Nh = Number of people in group (H) 
Nl = Number of people in group (L)

Question 45

What is the goal of discrimination in tests

Answer 45

Remember, the goal of discrimination is good categorization of individual cases into high and low performers.

That means we ideally have approximately half of people getting items wrong, and we want to be able to predict in which half an individual falls with relatively good accuracy – for each item

So, there is typically a connection between discrimination ability and item difficulty

Question 46

How do you calculate difficulty in tests

Answer 46

The item’s discrimination score should be interpreted in light of the item’s difficulty score.
Difficulty is 1 minus the proportion of correct responses from all individuals
Difficulty = 1 - Correct/N

Question 47

How would you optimize a test for research

Answer 47

If you are designing a test for research purposes, you want to maximize the variability in the scores (think, full range of possible responses).

That means you should aim for most items to have a difficulty score near the optimal .50
This gives you equal room above and below, across participants

With optimal difficulty, you have optimized the probability of getting a high discrimination score – though it is by no means certain

This means your item can separate good performers from poor performers

Question 48

What are some psychological considerations when creating tests

Answer 48

People are naturally inclined to want to do “well” on their tests. For research, this may or may not be a key concern – it depends on the test.

Getting just over 50% on a test doesn’t fit most peoples’ definition of “doing well”, even if it is above average
Pretty much only psychometricians would think this way

So, sometimes you need to sacrifice some psychometric quality to accommodate psychological considerations
Considering a test as a whole, there may be utility in having some very difficult items even if that means they have poor discrimination (this would depend on you choice of split, of course)

Question 49

How can you get around psychological considerations of tests

Answer 49

If it’s an option, standardization can help resolve the contrast between good psychometrics and psychological considerations.

In this case, standardization means taking the raw scores, which are likely very low, and converting them into a new score that looks more acceptable

Percentiles would be one way of doing it
IQ scores typically use standardization via Z scores, where a Z of 0 becomes 100 (a psychologically pleasing number) and each Z difference of 1 adds or subtracts 15 points

Question 50

If you have multiple factors in your factor analyses, how should you assess reliability?

Answer 50

Run one reliability analysis per factor (e.g. 2 factors = 2 reliability analysis)

Question 51

Contrast differences between PCA and EFA

Answer 51

For PCA, covariance matrix, not correlation matrix and there is no real definition of random error

PCA tend to have higher correlations because what EFA would attribute to error (unexplained variance), PCA will attribute to covariance (explained variance)

Question 52

What is the advantage of doing factor analysis rather than just averaging out correlations

Answer 52

Factor analyses can account for error and use “true score” better, allowing us to have more precise models

Question 53

What do we need for good measurements

Answer 53

We need good theories
We need to capture all our dimensions well
We need reliability
We need convergent validity (and ideally also discriminant validity)

Question 54

Describe the issue of content validity with the depression scales

Answer 54

There are 280 different measures of depression.
The most cited scale for depression is the CES-D
The CES-D has 20 items and 1/3 of them do not appear in any other commonly used measure of depression
Therefore, all 279 other research teams did not agree that a third of the questions should be measuring the construct of depression
this could led us to believe this scale measures depression and another component
Could lead to p-hacking, where one chooses the scale of depression that fits their hypothesis and reject their ones that don’t to demonstrate significance

Question 55

Describe the issue of reliability with the depression scales

Answer 55

The situation isn’t any better when looking at reliability for depression.
Inter-rater reliability of major depression diagnoses was .28 (presumably this is an r)
Cronbach’s alpha has many limitations, but in most cases is the only statistic ever used to demonstrate reliability
At least 20% of studies don’t even report alpha
Some report other people’s alpha, not in relation to their own sample

Disconnect between construct validity and reliability, while they should go together

Question 56

How is removing items from a scale bad for validity

Answer 56

Adding or removing items is a common practice, without providing a clear reason for doing so
Changing items isn’t necessarily a bad thing; the lack of justification is
Many studies don’t try to demonstrate validity, and many that do simply provide a citation to some other study