Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Prelims > METHODS > Flashcards

METHODS Flashcards

Method definitions, theories, assumptions, limitations, procedures, tips and tricks, etc.

1
Q

Demand Characteristics

A

When the participant changes their true response due to how they believe the experimenter wants it to go.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Expectancy Effects

A

When the experimenter’s biases cause them (consciously or unconsciously) to sway the experiment to match how they expect it to go.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Regression Discontinuity Design

A

A type of quasi-experimental design where you create an artificial control group by looking at people who fall close to a particular cutoff score, given that the cutoff score elicits the treatment. (ie, study outcomes of students with a grade of 79% compared to those at 80% who received a scholarship)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What are the 5 types of Quasi-experimental design?

A

Single group without a control Multigroup without a pretest Multigroup with control and pretest Time series designs Regression discontinuity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What are the 3 ingredients necessary for a randomized experimental design?

A

Random assignment Manipulates the independent variables Measurement of dependent variables

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Grounded Theory of Causal Generalization

A

Shadish, Cook, and Campbell, 2002 1. Surface Similarity 2. Ruling Out Irrelevancies 3. Making Discriminations 4. Interpolation & Extrapolation 5. Causal Explanation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Multicollinearity

A

When there is a very high (or even perfect) correlation between independent variables. It’s a good indication that the data is not reliable. (Edwards 2008)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Construct Validity

A

the degree of correspondence between the constructs (i.e., latent traits) referenced by a researcher and their empirical realization (i.e., operationalization/measure); in other words, is it measuring what it claims to be measuring?(Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Bias

A

systematic variance in an operational deginnition that is unrelated to the focal construct (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Unreliability

A

nonsystematic (random) variance in an operational definition (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Deficienct

A

when the constructs empirical realization (operationalization/measure) does not fully capture the essence of the construct (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Special Purpose vs. Non-Special Purpose Settings

A

More appropriate than a field vs lab distinction SP- created for the specific purpose of doing research (e.g., university labs, simulated work settings in an org); designed to allow for the effective (unconfounded) manipulation of one or more independent variables NSP- created for the purposes other than reserach (e.g., organizations, classrooms, churches, etc. )

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Probability sampling

A

workers are selected from a population (with N members) in a way that ensures that the probability of selection a sample of a given sze (e.g., n=10) is equal to the probbilty of selection any other sample of the same size… I.E., all people in the population have an equal chance at being selected to participate– rarely occurs in IO for many obvious reasons

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Startified random sampling

A

a researcher specifies strate of employees within an org (e.g., managerial, nonmanagerial) and then randomly select a specific number of workers from each stratum of interest

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

multistage cluster sampling

A

a researcher a) selects a random sample of units in one or moer clusters (e.g., private vs. public-sector, profit vs. non profit) and then b) randomly selects clusters of orgs from each larger cluster

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

nonprobabilty sampling

A

uses non random strategies for selecting sample members from a target population e.g., convenience sampling, purposive sampling of heterogeneous instances, systematic sampling, quota sampling

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

convenience sampling

A

selecting a nonrandom sample based on their availability to to participate in a study (i.e., intro to psych students)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

purposive sampling of heterogeneous instances

A

selecting a nonrandom sample of members on the basis that the researcher believes that they are diverse in terms of characteristics that might influence a causal relation between variables (Shadish et al., 2002)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

systematic sampling

A

select sample members in a methodical way from lists of assumed population members (e.g., every 10th person in the SIOP directory)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

quota sampling

A

selecting specific numbers of sample members of different types to produce a sample that is roughly representative of the target population

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Critiques of IO rigorous reserach

A

a) has little to no practical utility b) does not inspire confidence in the work of scientiests c) distorts important aspects of reality in orgs d) fails to consider phenomenon that are not readily measurable e) has low releance to the criteria that are important to practitioners f) doesn’t offer much more than common sense/common knowledge g) it is artificial/ irreleant/trivial j) has low operational validity (aka variables that can’t be controlled by practitioners realistically or ethically k) focuses on “proving” the truth of hypotheses and not problem solving l) untimely- deals with issues that change more rapidly than the knowledge it produces m) involves participants who are not representative of the populations to which generalizations are to be made (Stone-Romero, 2011; Thomas & Tymon, 1982)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Critique of NSP Causal Claims

A
  • harder to control extraneous and confounding variables even using random experiments or quasi-experiments - the claim that there is more mundane realism and thus more external validity doesn’t always hold up - typically involce nonrepresentative samples of subjects, settings, and operational definitions of manipulations/measures - Locke (1986) showed evidence that typically relations found in SP settins hold up in NSP settings.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Mundane Realism

A

Is it consistent with the every day realities (e.g., adding health benefits, vacation time, etc. to simulated orgs even if it is unrelated to the study); if a study lacks mundance realism, an effect found in SP settings may not generalize to NSP settings (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Experimental Realism

A

the manipulations/measures are operationally defined such that they have a suffidient impact on study participants, it is noticable, it is legitimate enough to show an effect; if it lacks this, models may fail to show support for causal connections between variables, even if there is one i.e., why manipulation checks are important (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

3 Major Attributes of Randomized Experimental Designs

A
  1. Manipulation of IVs (gives confidence about temporal precedence- causes preceding effects 2. Random assignment of units (individuals, teams, orgs) to study conditions (gives confidence that results are not due to confounds; that conditions are equivalent) 3. Measurement of DVs (outcome measures must be reliable or effects may not be found)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

Limitations of Randomized Experimental Designs

A
  1. You just can’t manipulate some variables (e.g., sex) 2. Some variables that can be manipuated, shouldn’t be on ethical grounds (e.g., mental health) 3. in NSP settings, may not be able to randomly assign units 4. in NSP it may not be possible to isolate units (spatially), so you can’t be positive about treatment effects 5. participants may refuse to be assigned to experimental conditions on a random basis 6. it may be unethical to withold beneficial treatments from units (i.e., control group receiving no training/treatment they should really get)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

Threats to experimental design internal validity

A

Differential attrition across conditions, history, testing, resentful remodalization, and the interaction of two or more threats (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

3 Attributes of Quasi-Experimental Designs

A
  1. Manipulation of IVs 2. Nonrandom assignment of units to conditions (why it doesn’t hold as much power to make causal inferences) 3. Measurement of assumed DVs ( assumed because the units are not randomly assigned to we can’t be sure the outcomes are caused by the IV, only assume so)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
29
Q

Nonexperimental Designs

A

quantitative & qualitative research

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
30
Q

Correlational study

A

studies that consider relations between (among) variables shouldn’t be referred to as correlational studies because corrrelation is a statisical technique; not an experimental design

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
31
Q

Nonexperimental Design Attributes

A
  1. Measurement of assumed IVs (not manipulating makes them be assumed &hard to prove temporal precedence) 2. Nonrandom assignment of units to assumed conditions (collecting data from participants with an assumed level of the IV that could have stemmed from various causes) 3. Measurement of assumed DVs (not randomly assigned so we can’t be sure the DVs are caused by the IVs)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
32
Q

Limitations of Nonexperimental Designs

A

You really can’t make causal claims/inferences because of the lack of internal validity and the inabilty to prove temporal precedence, control confounds (only able to confidently demonstrate correlation/covariation)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
33
Q

Design vs. Method

A

Research Design and Statistical Methods are independent; Studies of each type of design can use a variety of statistical methods to analyze the data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
34
Q

Prediction vs. Causal Claims

A

Prediction is a statistical term meaning inforation on a set of predictor variables can be used to perdict the value of a ctierion of interest This does not equate to evidence of a cause and effect relationship.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
35
Q

Causal claims

A

Need to be able to prove temporal precedence, covariation, and rule out any alternative explanations (control for confounds) Internal validity is driven by experimental design and cannot be compensated with external validity. The use of causal models (e.g., regression, MLM, SEM) does not justify the use of causal claims/inferences

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
36
Q

Most frequent IO designs

A

Nonexperimental studies (esp. quantitative studies) are most common and NSP (field) settings being most common (Stone-Romero, 2011) Quasi-Experimental designs seem to be the least common

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
37
Q

“Ranking” of Validity

A

If a study does not have internal, construct, and statistical conclusion validity; the external validity is irrelevant.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
38
Q

4 types of Validity

A

Internal validity ( relating to the existense of cause-effect relationships between variables) Construct validity (correspondence of constructs with operationalizations) Statistical Conclusion validity (the correctness/confidence/trueth of statistical estimates derived from the study) External validity (the extent to which relations generalize across different settings, populations) (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
39
Q

Stone-Romero (2011) Suggestions for Minimizing Common Method Variance

A

1) obtain predictor and criterion data from different sources 2) separate the time at which predictor and criterion variables are measured 3) counterbalance the order in which variables are measured

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
40
Q

Threats to Statistical Conclusion Validity

A

a) low statistical power (i.e., small sample sizes) b) failing to mett the assumptions of the stats test c) conducting a large # of stats tests using a Type I error rate tha is lower than the actual (effective) Type I error rate

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
41
Q

Statistical significance & practical importance

A

They are not mutually excluse There are “non-sig” effects that are practically important There are sig-effects that have effect size so small they make them not practically important See (Wasserstein et al., 2019) for more on why p-values/statistical significance is problematic.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
42
Q

Why is MTMM problematic?

A

1) Assumes traits and methods are uncorrelated 2) No set “rules” for what is enough in terms of satisfying rules (Stone-Romero, 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
43
Q

Detecting Method Variance

A

1) MTMM 2) CFA

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
44
Q

4 Attributes of Qualitative Research

A
  1. Occurs in natural settings (most of the time) 2. data is derived from the participants perspective 3. should be reflexible (i.e., flexible, dynamic) 4. qualitative instruments, observation methods, and modes of analysis are not standardized; the researcher serves as the main research instrument
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
45
Q

Typical Purpose of Qualitative Research

A

Theory generation or Theory elaboration; not as suitable for theory testing (Lee et al., 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
46
Q

3 most common qualitative designs in IO

A

Case Studies Ethnographies (observations) in-depth interviews

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
47
Q

Grounded Theory approach to Qualitative Research

A

on a regular, iterative basis it requires a) hypothesis generation from data b) testing hypotheses on new data that may elicit revisions in OG hypotheses c) testing revised hypotheses with new data continue this until there’s theoretical saturation or no new learnings occur (Lee et al., 2011)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
48
Q

Qualitiatve vs. Quantitative questions

A
  1. What is occuring? How is it occuring? What constructs should I use to explain it? 2. What is the prevalence of this? Does this generalize?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
49
Q

Qualitative limitations

A
  1. Very time intensive 2. Can be professionally risky (takes away from other research methods, takes so long, harder to publish) 3. some org scholars claim we have too much theory, but it is possible to test theory through qualitative means as well (Lee et al., 2011)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
50
Q

MTMM Rule for Showing Convergent Validity

A

Validity Diagonal must not = 0 and they should be sufficiently large (Campbell & Friske, 1959)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
51
Q

MTMM Rules for Discriminant Validity

A
  1. The validity diagonal > adjacent rows/columns in the hetero-trait/hetero-method triangles 2. The validity diagonal> corresponding heterotrait/monomethod triangles 3. the same pattern of trait interrelationships should be shown in all heterotrait triangles in both the monomethod blocks and heteromethod blocks (Campbell & Friske, 1959)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
52
Q

Calculating Method Variance with MTMM

A

heterotrait monomethod values- heterotrait heteromethod values = method variance (Campbell & Friske, 1959)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
53
Q

MTMM Tips and Tricks

A
  1. You can view heterotrait monomethod triangles as method factors– if its the highest correlation, it means method is mostly driving it
  2. the smaller the values in the heterotrait/heteromethod triangles, the more indepedence there is (whether it is the traits or the methods)
  3. if heterotrait/hetero method triangles are close to equivalent with validity diagonal that implies there is no discriminant validity
  4. if heterotrait/ monomethod traingles are about the same or close to equivalent to reliability values, that implies there is no discriminant validity

(Campbell & Friske, 1959)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
54
Q

MTMM logic summary

A

Measures of the same trait should correlate higher with each other than measures of different traits with different methods Ideally, the validity values should also be higher than the correlations among different traits measured by the same method ** these conditions are rarely met and therefore it is a subjective judgement in the end about what is “valid enough” (Campbell & Friske, 1959)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
55
Q

MTMM Assumptions

A
  1. Traits and Methods are independent/uncorrelated 2. You are interpreting validites with the reliabilty of methods in mind 3. Unreliable methods makes the whole procedure a bit pointless it is concerned with the adequacy of tests as measures of a construct, rather than the adequacy of a construct (Campbell & Friske, 1959)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
56
Q

Reliability and Validity in relation to agreement

A

Reliability is agreement between two efforts to measure the same trait through maximally simlar methods Validity is represented in the agreement between two attempts to measure the same trait through maximally different methods (Campbell & Friske, 1959)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
57
Q

Topics/Trends of 1917-1925

A
  • Objectivity and methods to solve important problems
  • Cognitive ability tests
  • Interest in the exceptional
  • beginning of statistical significane and prediction models
  • sources of rating errors (e.g., halo (Thurston, 1920) and guessing)
  • focus on psychometric properties, classification, and test equivalence (Cortina et al., 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
58
Q

Topics/Trends of 1925-1945 (The Depression and WWII)

A
  • Test scoring methods, test form equivalence, and cross-validation
  • assessment of non-cognitive testing (e.g., social/emotional intelligence, values, interests)
  • limitations of self report measures
  • Shift to a hypothetico-deductive method (i.e., hypothesis testing and theory driven studies)
  • properties of distributions (e.g., growth curves, Poisson)
  • Reflections on the past: too much survey work, need to expand samples of interest (Cortina et al. 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
59
Q

Topics/Trends 1946-1969

A
  • Predicive power of personality measures (esp. MMPI)
  • Social desirability and faking
  • development of many new measures (OPQ, PAQ)
  • literature reviews become common
  • science of job attitudes
  • statistical significance over effect size (lol a start of the issue)
  • smaller font for method section (showing the shift to focus of theory at the expense of methods)
  • Reflections on the past: we stopped studying the exceptional, decline in practitioner authorship, shortened time perspective for the field (Cortina et al. 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
60
Q

Topic Trends 1970-1989

A
  • Focus on measure development (job characteristics, work values, job involvement)
  • IRT
  • self vs. supervisor reports,
  • # of points and anchors in a rating scale
  • increased focus on methods-specific papers
  • data-analytic innovations of meta-analysis and SEM (Cortina et al. 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
61
Q

Topic/Trends of 1990-2014

A
  • Methodological plurality (no single approach no matter its potency is the right answer for all questions/research)
    • Broadening of methodological choices
  • -introduction of methods-oriented journals -
  • publication of reactive and prescriptive reviews of methodological practices
  • realization that data analytic approaches do not occer a pancea to solve research issues -
  • subgroup differences -
  • level of analysis, aggregation, and MLM -
  • importance of effect sizes
    • restoration of method section font size (Cortina et al. 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
62
Q

Suggestions for 2017 and Beyond

A
  • Replication of research -understanding the exceptional, both good and bad -identifying inappropriate methods in the review process (method experts review each paper; pre-review before starting) - better research design and measurement - more specific theories that are testable rather than offerly vague ones that could never by fully validated (Cortina et al. 2017)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
63
Q

Critique of Maximum Likelihood Estimation for Validity

A
  • Only uses sample derived information in making inferences/decisions about test validity; ignoring all relevant info accumulated in the past (Schmidt & Hunter, 1977)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
64
Q

Bayesian Approach for Validity Generalization

A
  • utilizes sample data and prior knowledge in estimating validity & weighs each in proportion to its information value - prior distribution of variances is combined with sample info to create a posterior distribution - the mean (or mode) of the posterior distribution is taken as the estiate of the parameter of interest (i.e., test validity) (Hunter & Schmidt, 1977)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
65
Q

Why is Bayesian for Validity a good idea?

A
  1. bayesian priors used are based on data (i.e. not subjective) and incorporates all past relevant info
  2. the assumptiosn made about between study variance in criterion reliability and range restriction are conservative
  3. certain sources of error variance in the obtained distribution are not corrected for, making it even more conservative
  4. corrections made to the mean of the prior for average range restriction effects are “probably” conservative
  5. procedures provides a parsimonious, sophisticaed and tech sound solution for the overarching problem of validity generalization
  6. the model can be extended to provide an improved method of data analysis and decision making in criterion-related validity studies
  7. when slightly modiffied, it provides a tool that can lead to the establsihment of general principles about trait-performance relationships (Hunter & Schmidt, 1977)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
66
Q

Correcting for error variance due to sample size

A

Find the average sample size across published studies then take 1(N-3) to get the estimates variance due to the sample size (Hunter & Schmidt, 1977)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
67
Q

Notion for correcting for error variance (i.e., reliabilty and range restriction) in validity generalization

A

If you take a bunch of validity coefficients, standardized them into Z scores, and then subtract the error variance and the variance of the distribution of the validiy coefficient is 0- then essentially it shows there is no true variation in validation; therefore, if we correct these in the beginning we are more likely to find the true validity ((Hunter & Schmidt, 1977)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
68
Q

Correction Warning

A

corrected variance tends to overestimate rather than underestimate (Hunter & Schmidt, 1977)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
69
Q

Common Method Variance

A

variance that is attributed to the measurement model rather than to the constructs the measures represent (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
70
Q

Effects of Common Method Variance

A

Method effects inflate the observed relationship when the correlation between the methods is higher than the observed correlation between methods -method effects; it deflates the observed relationships when the method correlations are lower than the observed correlation- method effects (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
71
Q

Common method biases causes

A

common rater common measurement context common item context characteristics of the items themselves (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
72
Q

Stages of the Response Process

A

Comprehension Retrieval Judgement Response Selection Response Reporting (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
73
Q

Comprehension Stage Potential Method Bias

A

Item ambiguity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
74
Q

Retrieval Stage Potential Method Bias

A

measurement context, question context, item embeddedness, item intermixing, scale size, priming effects, transient mood states, and item social desirability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
75
Q

Judgement state potential method biases

A

consistency motif ( in attempt to increase accuracy in the face of uncertainty), implicit theories, priming effects, item demand characteristics, and item context-induced mood states

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
76
Q

Response selection potential method biases

A

common scale anchors and formats; item context-induced anchoring effects

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
77
Q

Response reporting potential method biases

A

consistency motif (in an attempt to appear rational), leniancy bias, acquiescence bias, demand characteristics, and social desirability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
78
Q

Two main ways to control for method biases

A

a) Study design b) statistical controls never rely on one or the other; a combo of both is best

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
79
Q

Common Rater Effects

A

Consistency Motif Implicit theories (& illusory correlations) social desirability leniency bias acquiescence biases (yea-saying and nay-saying) mood states (i.e., PANAS) transient mood states

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
80
Q

Item Characteristic Effects

A

Item social desirability item demand characteristics item ambiquity common scale formats common scale anchors positive and negative item wording

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
81
Q

Item Context Effects

A
  • Item priming effects
  • item embeddedness
  • context-induced moods
  • scale length (shorter is better for fatigue/boredom, but may enhance the likelihood of previous items influencing responses)
  • Intermixing (or grouping) of items or constructs on the questionnaire
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
82
Q

Measurement Context Effects

A

Predictor and criterion variables measured at the same point in time predictor and criterion variables measured in the same location predictor and crierion variables measured using the same medium

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
83
Q

Crique of Harman’s Single Factor Test

A

The use of EFA/CFA to see if the most variance loads on one factor is problematic because it assumes if there is method variance it would account for most/all the covariance which is a large and inappropriate assumption to make; it doesn’t control for the method variance; just identifies it * yet it is still one of the most widely used methods for detecting method bias (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
84
Q

Partial Correlation Procedures for Method Bias

A

Partialling out social desirability as a surrogate of method variance Partialling out an unrelated “marker” variable that is unrelated to the construct of interest Partialling out a general methods factor compares the structural parameters with and without these factors to determine potential effects (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
85
Q

Controlling for a measured latent methods factor

A

items are allowed to load on their theoretical constructs and a latent method factor (with its own measurement component) and the sigficance of the structural parameters are examined with and without the latent factor; the latent factor is assessed by a surrogate variable (e.g., social desirability) assumed to represent CMV (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
86
Q

Controlling for an unmeasured latent methods factor

A

Items are allowed to load on their theoretical constructs and a latent common methods variance factor’ structural parameters are compared with and without the methods factor’ the variance of the responses is partitioned into trait, method, and random error (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
87
Q

CFA x MTMM

A

Combines both CFA and MTMM to partiion variance into trait, method, and random permitting researchers to control both for method variance and random error when examining relationships between predictor and criterion variables (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
88
Q

Correlated Uniqueness Model

A

Each observed variable is caused byonle one trait factor and a measurement error term; there are no methods factors; the model accounts for method effects by allowing the error terms of constructs measured by the same method to be correlated (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
89
Q

Direct Product Model

A

this model assumes the trait measures interact multiplicativel with the methods of measurement to influence each observed variable. The model assumes stronger the correlation between the traits, the more intercorrelation between the traits will be infuenced by shared method biases (trait and method components can’t be separated in analysis can’t control for method factors while testing for relationships; does not test for interactions after first controlling for main effects) (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
90
Q

Partial Correlation Approaches Disadvantages

A

fails to distinguish between method bias at the measurement level from method bias at the construct level ignored measurement error in the method factor only control for single source of method bias at a time ignores method x trait interaction - besides Harmon’s single factor solution, these solutions are the worst (Podsakoff et al., 2003)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
91
Q

Single Method Scale Score Approaches disadvantages

A

-ignores measurement error in the method factor - only controls for a single source of method bias at a time - ignores method x trait interaction (Podsakoff et al., 2003)

92
Q

Single method factor approaches disadvantages

A
  • only controls for a single source of method bias at a time - ignores method and trait interactions (Podsakoff et al., 2003)
93
Q

Multiple-method factor approaches disadvantages

A

Ignores method x trait interactions (i.e., your best bet because these approaches leverage a MTMM approach in a way) (Podsakoff et al., 2003)

94
Q

4 Questions to ask when determining the right CMV remedy

A
  1. Can the predictor and criterion variables be obtained from different sources? 2. Can the predictor and criterion variables be measured in different contexts? 3. Can the sources of method bias be identified? 4. can the method bias(es) be validly measured?
95
Q

Mediator

A

an intervening variable that transmits the effects of an antecedent variable on to a dependent variable X->M->Y

96
Q

Moderator

A

a variable that causes the relationship between X and Y to vary; whether is it the direction or the magnitude (also a statistical interaction)

97
Q

A moderated mediation

A

when the magnitude or direction of the mediation effect is dependent on the values or level of a moderator (e.g., motivation -> job performance-> salary, but gender moderates job performance such that the job performance-> salary relationship is weaker for women than men) ** completely made up not based on a study

98
Q

A mediated moderation

A

a mediated interaction; basically when the X x Z interaction is causing the M (mediator) which is then causing the Y e.g., SJT test format x race -> face validity (reading comprehension -> test performance (Chan & Schmitt, 1997)

99
Q

Alpha Change

A

changes in absolute levels given a constant conceptual domain and a constant measuring instrument (Golembiewski et al., 1976; Chan, 2011)

100
Q

Beta Change

A

changes in absolute levels that are driven by an alteration of the respondents scaling; a mental recalibration of the scale (Golembiewski et al., 1976; Chan, 2011)

101
Q

Gamma Change

A

changes in the conceptual domain (i.e., changes in the meaning or conceptualization of what the construct is ((Golembiewski et al., 1976; Chan, 2011)

102
Q

Baron & Kenny (1986) Mediation Test

A
  1. use regression/correlation to test for the seperate significant relationships between a) X and Y b) X and M c) M and Y controlled for X 2. if one or more of the above relationships is non-significant–> no mediation 3. if significant; one should test the significance of a) M and Y controlling for X b) X and Y controlling for M 4. if a) is significant and b) is non-significant–> mediation!
103
Q

Baron & Kenny Mediation criticisms

A
  1. very low ability to detect mediated effects 2. does not provide a numerical esaimte of the strength of the meditation (i.e., indirect effect) 3. requires the overall X and Y is significant, however, technically a causal chain between X ->M->Y where X –> Y don’t have a significant relationship could surely exist OR it could exist at the population level but the sample level doesn’t detect it because of small effect size/low power
104
Q

OLS Regression Assumptions

A
  1. errors are assumed to be normally distributed with a mean of 0 (normality assumption) 2. error variances are constant across different values of X (homoscedasticity assumption) 3. the errors from each observation are not correlated with the errors from the other sample observations (i.e., independence assumption violating the independence assumption is the most impactful violation you can make (Chan, 2011)
105
Q

What types of methods/tests are used for testing Interactions?

A

When X and Z are categorical -> ANOVA When either X or Z are continuous -> multiple regression or hierarchical multiple regression

106
Q

Proving causal claims in quasi-experimental designs

A

Designs without control groups can yield strong causal inferences ONLY by reducing the plausibility of alternative explanations for the treatment effect. ○ To reduce the plausability of alternative explanations in quasi-experiments is: - Identification and study of plausible threats to internal validity - Primacy of control of design □ e.g., adding elements such as observation at more pretest time points, additional control groups - Coherent pattern matching □ A complex prediction is made about a given causal hypothesis that few alternative explanations can match (e.g., nonequivalent dependent variables) (Shadish et al., 2002)

107
Q

Ιdentify the relevant components in the ΜΤΜΜ matrix

draw conclusions on the validity

A

Τhere is convergent validity

108
Q

Identify the relevant MTMM components and make appropriate validity inferences

A
109
Q

Time Series Design

A
  • A large series of observations made on the same variables consecutively over time
  • Key is knowing the specific point in the series at which a treatment occurred.
     If the treatment had an effect, the causal hypothesis is that the observations after treatment will have a different slope or level from those prior to treatment;
     A series should show an interruption to the prior situation at the time at which treatment was delivered
110
Q

Time Series Design Effects

A

Types of effects:
- Should be a sharp discontinuity at the point of the intervention (called a change in
level or intercept)
- Should be a change in slopes of the series at the point of interruption (called a drift,
trend, or slope)
- Continuous effect does not decay over time
- Discontinuous effect does not persist over time (common if intervention is
introduced and then removed)
- Immediate or delayed (more difficult to interpret) effects -
 Form
Describe the effect on all three of the above dimensions
of the effect (level, slope), its permanence (continuous vs discontinuous) and its immediacy
(immediate or delayed)

111
Q

We must design studies that MAXMINCON

(Shadish et al. 2002)

A

1) MAXimize systematic variance
i. Collect information on variables that may affect the treatment
ii. Variance you can account for is systematic; variance unaccounted for becomes error
2) MINimize error variance
i. Use highly reliable measures
ii. Use multiple measures of the same construct
3) CONtrol extraneous variance
i. Control any influences that many impact our outcome

112
Q

Surface similarity

A

i. Scientists generalize by judging the apparent similarities between what’s being studied and the target of the generalization
1) E.g., the effects of secondhand smoke in the workplace seems more similar to public settings at issue in policy debates than do studies of smoking in private residences

113
Q

Ruling out irrelevancies

(shadish et al., 2002)

A

Identifying those attributes of persons, settings, treatments, and outcomes that are irrelevant because they do not change a generalization
E.g., Size is irrelevant to the membership of the bird category

114
Q

Making discriminations

Shadish et al 2002

A

Scientists generalize by making discrimination that limit generalization. 1) Child psychotherapy works in a lab but might not work in the clinic

115
Q

Interpolation and extrapolation

A

Scientists generalize by interpolating to unsampled values within the range of the sampled persons, settings, treatments, and outcomes, and much more difficult, by extrapolating beyond the sampled range
1) The effects of therapies for cancer increase as the dose increases until the point at which they would cause the patient to die (interpolation) or the effects of toxic chemicals on small animals will generalize to humans (extrapolation)

116
Q

Causal explanations

A

Scientists generalize by developing and testing explanatory theories about the.target of generalization
1) A theory applies to both scratching in primates and rearing in rats because the biological mechanisms underlying both these behaviors is the same.

117
Q

Shadish et al 2002 take on validity

A
  • “The approximate truth of an inference”
  • When we say something is valid, we make a judgment about the extent to which relevant evidence supports that inference as being true or correct
  • It is a property of inferences, not a property of design or methods
118
Q

Statistical conclusion validity

A
  • “The validity of inferences about the correlation (covariation) between treatment and outcome”
  • Refers to the appropriate use of statistics to infer whether the presumed independent and dependent variables covary
119
Q

Construct validity

shadish et al. 2002

A

the validity of inferences about the higher order constructs that represent sampling particulars

120
Q

Internal validity

shadish et al., 2002

A

○ The validity of inference about whether or not the covariation resulted from a causal relationship

121
Q

External validity

A

The validity of inferences about whether the cause-effect relationship holds over variation in person, places, setting, treatments, and measurement variables.

122
Q

Three major aspects of construct validation

Nunnally, 1978

A

○ Specifying the domain of the construct
○ Empirically determining the extent to which items measure that domain
○ Examining the extent to which measure produce results that are predictable from theoretical measures

123
Q

Factor Analysis vs. Principle Components Analysis

Differences in analysis

A

1) The differences lie in the variance that is analyzed
 PCA:
□ ALL the variance in the observed variables is analyzed…which is why there is 1.0 in the diagonal of the correlation matrix– assumes each item is perfectly correlated with itself

 FA:
□ Only SHARED variance is analyzed
□ Uses an estimate of the variance in each items shared with other items as the diagonal of the correlation matrix
 Called a ‘communality estimate’

124
Q

Factor analysis vs. Principle Component Analysis

Theory

Tabachnick & Fidell, 2001

A

Theoretical differences in the reason that the variables are associated with a factor or component

  • *a. FA: reflexive structure**
    a) Factors cause the variables
  • *b. PCA: formative structure**
    a) Variables produce the component

There is NO underlying theory about which variables should be associated with which factors (more about data reduction and categorization not identifying underlying latent traits)

125
Q

EFA vs CFA

A

EFA
 Theory Development purposes
□ One seeks to describe and summarize data by grouping together variables that are correlated (used in early stages of research)
○ Confirmatory
 Theory testing purposes
□ Used to tests a theory about latent processes □ Often performed through SEM

126
Q

Eigenvalue

A

an index of the strength of the factor; Put differently, the amount of variance for which the factor accounts

  • we retain factors with eigenvalue greater than 1
127
Q

Rotations in Factor Analysis

A

process by which the solution is made more interpretable without changing underlying mathematical properties

  • Orthogonal: factors are uncorrelated
    • Varimax most common
  • Oblique: factors are correlated
    • Promax rotation most common
128
Q

Determining the number of factors in EFA

Hinkin (1998)

A

1) Kaiser criterion: keep factors with eigenvalues greater than 1

  • This criterion isn’t always useful
  • Some authors have found that Kaiser’s approach can lead to over- factoring
  • It is based on assumptions pertaining to PCA

2) Scree test: look for the dip.
3) Also, percent of total and common variance accounted for by the factor
4) And interpretability (are all factors retained interpretable?)

129
Q

What items to retain in scale development with EFA

Hinkin 1998

A

1) Factor Loadings: only items that clearly load on a single factor (grater than factor loading of .40 and/or loading twice as strong on the appropriate factor than any other factor)

  • a. .4 or .45 are common cutoffs
  • b. More than .5 is really good
  • c. Large and positive loadings for the “correct” measures

2) Communality stats: this helps determine the proportion of variance in the variable explained by each of the items; retain items with higher communalities
3) Percentage of total item variance: larger the better ( > 60% is good)

130
Q

Effects sizes are linked to

A
  • The sample size on which the effect size is based
  • Alpha- level: The probability at which we accept an effect as being
  • Statistical power: The ability of a test to detect and effect of that size
131
Q

Item generation approaches

Hinkin 1998

A

2 approaches to generate preliminary items

1) Deductive
a) Review literature and theory to acquire a full understanding of the construct (e.g., finding the definition of expert power) and find further examples of behaviors that fit the definition

2) Inductive
a) Simply ask people for examples of behavior
i) Useful when conceptual basis of construct may not be easily identifiable and exploratory research
ii) E.g., asking respondents to describe “how your manager
communications with you”

132
Q

Item development recommendations

Hinkin 1998

A

1) Simple, short statements
2) Familiar language
3) Avoid double-barreled items

133
Q

Number of items for scales

Hinkin 1998

A

1) Short measures are effective at minimizing response bias caused by fatigue and boredom
2) At least four items are needed to test the homogeneity of items within a latent construct (by item intercorrelations) (Billings & Nilan, 1985)
a) Scales can have high internal consistency even with low item intercorrelations
3) Final goal will be to retain about four to six items for most constructs, but the final determination must be made only with accumulated evidence to support construct validity (Schriesheim et al., 1993)
4) Always generate twice as many items as you may need

134
Q

Scale development - internal consistency assessment

Hinkin 1998

A

i. Reliability: the accuracy of precision of a measuring instrument and is a necessary condition [but not sufficient] for [construct] validity
1) Establish reliability after unidimensionality has been established
ii. Can calculate a number of ways but internal consistency using Cronbach’s alpha is the most commonly accepted measure (Price & Mueller, 1986)
1) A large coefficient alpha (minimum of .70 for exploratory measures; Nunnally, 1978) provides and indication of strong item covariance and suggests that the sampling domain has been captured adequately (Churchill, 1979)
2) May want to eliminate items that reduce reliability, especially if these additional items lower interitem correlations
3) Alpha is very sensitive to the number of items (Cortina, 1993)

135
Q

EFA vs PCA

A

PCA is more for data reduction, minimizes the values, the goal is to produce a set of components that can be used to reproduce the data matrix

EFA- derive a set or factors that can be used to produce the covariance matric; infer what we can’t see from what we can see; identify the underlying commonalities driving the factors

136
Q

Path Analysis vs SEM

A

Path analysis- no measurement errors included

SEM- the measurement error is included

137
Q

Key Difference between Moderation and MLM

A
  • It’s like a MANOVA and multiple ANOVAs
  • at a practical concrete level is the error rating and having more statisical power
138
Q

Rwg

(James et al., 1984)

A
  • The extent to which the group agreed
  • the ratior of variance of the group to variance of complete disagreement
  • i.e., the ratio of observed disagreement to possible total disagreement
  • possible disagreement usually modeled as a null distribution (unified distribution) but at times it is a skewed distribution and not rectangular
    • arbitrary rule of .7 or greater to justify aggregating
139
Q

Interrater agreement vs. Interrater reliabiltiy

A

Agreement: the extent to which the members of the group giver interchangable responses

  • about the absolute value
  • shown in terms of distance apart on graphs
  • this is what is needed to justify agreement/groups/aggregation

Reliability: the extent to which the group ratings are orrelated with each other

  • about the ranking of ratings
  • shown in terms of shape on graphs
  • aggregation based on this can get you aggregation bias because it can be very high with no interrater agreement present
140
Q

Intraclass Correlations

ICC(1) vs. ICC (2)

A

ICC (1)- the variance within groups compared to the variance between groups

  • the lower the number is the less variance there is in between groups than within groups; which shows agreement
  • proven to be wrong for justifying aggregation- Rwg is a better option

ICC (2) - the reliabilty of the group means or the amount of error in the means

  • you want this one to be larger
141
Q

Parceling in SEM

A

when you have more than 4-7 items loading on a factor, it becomes impossible to get a good fit

  • you take the first three average them = parcel, and so on
  • your indicators are then little composites of items (parcels) not items
142
Q

Ways to replace missing data

A
  1. Mean substitution ( worst option)
  2. imputation
    1. the process of predicting what the person would have said to that item if they answered based on their responses to all the items they did answer
  3. E.M. Algorithm ( Expectations maximization)
    1. considers all possible configurations of values in all the empty slots and comes out with the configuration that best optimizes some sort of criterion
143
Q

Spector (2010) CWB research

A

Basically showed common method variance is essentially a myth and doesn’t need to be made into as big of a deal

  • conclused we should use frequency ratings, not agreement
  • should use self-report not supervisor
  • don’t use antithetical items
144
Q

Reflective indicators (Podsakoff, 2003)

A

(a) the indicators were conceived of as manifestations of the construct, (b) changes in the indicator
were not expected to cause changes in the construct, (c) changes in the construct were expected to cause changes in the indicators, (d) the indicators shared a common theme, (e) eliminating an indicator was not expected to alter the conceptual domain of the construct, (f) a change in the value of one of the indicators was expected to be associated with a
corresponding change in all of the other indicators, and (g) the indicators were expected to have the same antecedents and consequences.

145
Q

Formative indicators

Posdakoff 2003

A

(a) the indicators were viewed as defining characteristics of the construct
(b) changes in the indicators were expected to cause changes in the construct,
(c) changes in the construct were not expected to cause changes in the indicators,
(d) the indicators did not necessarily share a common theme,

(e) eliminating an indicator might alter the conceptual
domain of the construct,

(f) a change in the value of one of the indicators was not necessarily expected to be associated with a change in all of the other indicators,
(g) the indicators were not expected to have the same antecedents and consequences.

146
Q

CTT Limitations

A

1) Item and person statistics are interdependent

  • a. Can’t compare items from different samples
  • b. Can’t compare persons who take different tests

2) Reliability is flawed and you always get lower-bound estimates
3) Standard error of measurement same for all persons

  • (despite how well that persons ability matches up with item difficulty)
  • Is based on reliability which is a test property that doesn’t consider the person or persons in question

4) Is test oriented, not item oriented
* a.Goal is to design test. We know nothing about probability of responses to individual items

5) If test scores are not equally precise across individuals (which they are not, given that
items contain more information when matched with the respondents ability), how can we
compare scores across persons within a sample?

147
Q

Why use Cohen’s D?

A
  • Used as a counterpoint to significance tests where this effect size gives an indication of how big or small a significant difference is.
    • This difference can then be compared to Cohen’s (1988) estimates of what is typical of a small (e.g., .2), medium (e.g., .5), or large (e.g., .8) effect. Typically share the same range as standard deviation (-3.0 to 3.0)
  • To provide a common metric on which to compare effects for meta-analysis or what not when outcome variables may be measured on different scales.
148
Q

SEM

(covariance structure analysis)

A

A hypothesized causal model that specifies directional paths linking all of the study variables is posited based on theoretical arguments by the researcher and fitted to the
observed data.

149
Q

MultiLevel Modeling (MLM; HLM)

A
  • Evaluates relationships at multiple levels of analysis
  • Models variances among variables at these different levels
  • Well suited to tests cross-level interactions
  • Allows researchers to partition the variance in outcome variables
  • Provides method for analyzing longitudinal relationships
150
Q

When to use MLM?

A
151
Q

Why is Ordinary Least Squares (OLS) Regression not appropriate for nested/multilevel data?

A

1) Errors in OLS are assumed to be independent
* In multilevel data, independence assumption is almost always violated
2) size for determining statistical significance of b2 is the lower level observations which is incorrect, instead of using the group-level sample size
* Using the wrong (inflated) sample size, the standard errors are wrong and it is easier to obtain statistical significance for the effect of z (increasing type 1 error)

152
Q

Why/When to use MLM

A
  • Observations nested within a larger grouping
  • Occasions within ppl (repeated measures), Kids within schools, employees within companies, etc
  • Need to account for non-independence of observations (i.e., not assigned at random)
  • Balanced data are not required (i.e., groups don’t have to be equal in size)
  • Analyses complete data, not just complete data
  • Lots of missing data with repeated measures and longitudinal data; takes that into account
153
Q

Hinkin 1998 on MTMM

A

Original MTMM guidelines criticized for use of unrealistic assumptions and reliance on a qualitative assessment of comparisons of correlations.

154
Q

An SEM model is typically composed of what two parts?

A
  • A measurement model – describes how latent variables are measured by their manifest indicators (i.e., test scores, survey observation)
  • ·A structural model – describes the relationship between the latent variables and shows the amount of unaccounted for variance.
155
Q

Measurement Concerns/Challenges

A
  1. The Criterion Problem
  2. Violating Assumptions
  3. Multicollinearity
  4. Common Method Variance
  5. Common Method Bias
  6. Measurement Invariance
  7. Inconsistencies in studies/literature
    1. typically indicates a missing moderator
156
Q

When should we dichotomize?

(Cohen, 1983)

A
  • to Eases data analysis,
  • variable in question is a crude measure
  • dichotomy represents true state of affairs
157
Q

Reverse Coded Items

Spector et al. 1997

A
  • Factor analyses of scales that contain items written in opposite directions sometimes show two factors, each of which contains items written in only one direction.
158
Q

Schmitt 1996

Issues with how people use alpha

A
  • Alpha is not an appropriate index of unidimensionality to assess homogeneity; it assess internal consistency– those are different
  • In correcting for attenuation due to unreliability, use of alpha as an estimate of reliability is based on the notion that the measures involved are unidimensional.
    • if it isn’t unidimensional it will lead to overcorrection
  • There is no sacred level of acceptable or unacceptable level of alpha. In some cases, measures with low levels of alpha may still be quite useful.
  • Presenting only alpha when discussing the relationships of multiple measures is not sufficient. Intercorrelations and corrected intercorrelations must be presented as well
159
Q

Nunally 1984

A
  • Says reliability should be a minimum or .70 for research
  • Should be at least .90 and preferably .95 for making important decisions about individuals.
160
Q

The Various Descriptions of Alpha (Cortina, 1993)

A
  • Alpha is the mean of all split-half reliabilities
  • Alpha is the lower bound of reliability of a test
161
Q

Internal consistency vs. homeogeneity

(Cortina, 1993; Schmidt, 1996)

A

internal consistency- refers to the degree of interrelatedness among items

homogeneity- refers to unidimensionality

  • internal consistency is a necessary but not sufficient condition for homogeneity
162
Q

Scale length and Alpha

A
  • the higher the # of items, the higher the alpha– even if the scale is not unimdensional
  • alpha is essentially a function of the number of items in the scale and it shoul dbe interpreted with the number of items in mind
  • if a scale has more than 14 items, its very hard to have an alpha lower than .7

(Cortina, 1993)

163
Q

What are the assumptions of Classical Test Theory?

A

CTT assumes ZERO. 1) The mean of the error scores is zero. 2) The correlation between True Score and Error Score is zero. 3) The correlations between error scores is zero.

164
Q

What is our goal when we use CTT to assess reliability?

A

To separate 1) the stable characteristics of the individual (true score) from 2) the inconsistent factors (error)

165
Q

Reliability (CTT)

A

2 interchangeable definitions: The consistency of the measure. How much the measure is free of error.

166
Q

In CTT, if a source of variance is the same across test-takers, it is…

A

Systematic variance, which is not treated as error.

167
Q

3 Types of Reliability Methods (CTT)

A

1) Test Retest (same test, taken twice) 2) Alternate forms (2 tests) 3) Internal Consistency (split half or Cronbach’s alpha)

168
Q

Reliability Index (CTT)

A

Correlation between observed and true score

169
Q

Reliability Coefficient (CTT)

A

Correlation between parallel measures

170
Q

Lumsden (1976)

A

Lays out all the issues/criticisms with Classical Test Theory (remember the flogging wall?) - Tests are unreliable - All other things equal, longer tests are better up until boredom or fatigue - Higher item covariances are better, up until attenuation paradox could occur

171
Q

What are the 6 categories of sources of variance in CTT?

A

1) Lasting & general (reading comprehension) 2) Lasting & specific (ability level) 3) Temporary & general (health) 4) Temporary & specific (test taking strategy) 5) Systematic issues (time limits) 6) Random events (lucky guesses)

172
Q

Cronbach’s Alpha

A

The mean of all split-half correlations

173
Q

Attenuation Paradox

A

In CTT, reliability and validity increase together, but after a certain point any further increases in reliability will decrease validity

174
Q

Sampling distribution

A

The distribution of a statistic gathered from all possible samples of the population. From the central limit theorem, we know that the sampling distribution will always be normal.

175
Q

Sampling error

A

The difference between the sample statistic and the population statistic, caused by sample characteristics differing from the population. Can be corrected with a larger sample size or a more representative sample.

176
Q

Content Validity

A

The degree to which the measure covers a representative sample of the content. General steps: 1. define content domain; 2. Select SME’s; 3. Provide framework for rating or matching items to the content domain; 4. Collect & summarize data

177
Q

Criterion-related validity

A

Correlation (Validity Coefficienct; r xy ) between predictor (IV) and criterion (DV). Can be Concurrent (IV & DV collected simultaneously) or Predictive (DV collected after IV).

178
Q

Construct validity

A

Accuracy of inferences (convergent & discriminant).

179
Q

Multitrain-miultimethod matrix (MTMM)

A

A matrix of multiple meathods used to measure multiple traits; Same construct, diff. method correlations (Monotrait-heteromethod; Convergent) should be higher than diff. constructs, same methods (Heterotrait-monomethod)

180
Q

Restriction of Range

A

Removal of portion of sample (e.g. cutoffs) often results in attenuation.

181
Q

Attenuation

A

decrease in validity coefficient; Caused by unreliability and restriction of range; Statistical corrections possible

182
Q

Validity coefficient

A

predictor-criterion correlation

183
Q

Common problems in Criterion-related validation

A

Small sample size; Low reliability (pred. & crit.); Restriction of range; The criterion problem

184
Q

The Criterion Problem

A

Contamination or deficiency in our criterion.

185
Q

Validity

A

Validity is a representation of the scientific confidence in the inferences and information from a test.

186
Q

Cohen (1983)

A

Talks about the cost of dichotomization, which is that is reduces the correlation (r), which then reduces 1) effect size 2) the value of the t-test 3) statistical power

187
Q

Cox (1980)

A

This is the paper that tried to find the optimal # of scale response options. He found that a) 7 +/-2 is best for keeping the responses meaningful to the respondent, and b) 3 to 9 response options are best for determining differences among people.

188
Q

Tourangeau et al (2000)

A

The authors line out the psychology of survey-takers, the “Response Process Model”, saying that responses come from the process of 1) comprehension 2) retrieval 3) judgement 4) response

189
Q

Standard Error of Measurment (CTT)

A

the standard deviation for an person’s expected score across a large number of equivalent tests

190
Q

MacCallum et al (1999)

A

Says that there is not rule of thumb for sample sizes of EFA’s. It needs to be determined by 1) Sample size 2) Communality of the items 3) Overdetermination of the factors The lower your communality, the higher the sample size needs to be.

191
Q

Preacher & MacCallum (2003)

A

The article that warns people not to use “little jiffy” when doing an EFA analysis because it uses a PCA (bad: items load on factors; assumes correlation) and a Varimax rotation (bad: assumes variables are unrelated).

192
Q

Communality of the Item

A

How much the item varies (%) due to latent factors. More communality means less random error. Less error means you don’t need a big sample size to estimate factor loadings.

193
Q

Overfactor

A

Retaining too many factors in an EFA, as opposed to underfactoring (retaining too few). Better to overfactor than underfactor.

194
Q

Unified Framework of Vaidity (definition)

A

A visualization of the important inferences in validation that describe relationships between predictor and criterion, measure and construct domains.

195
Q

Binning & Barrett (1989)

A

Point 1: Validity is confidence in inferences and information from a test and not a characteristic of the test itself, and validation is a reflection of the process of developing scientific theories. Point 2: Provide unified framework of validity

196
Q

Method Effects

A

The method is causing higher correlations; detected in the difference in heterotrait-monomethod and corresponding heterotrait-heteromethod matrices.

197
Q

Campbell & Fiske’s (1959)

A

4 Rules: 1) Validity diagonal correlations should be statistically significant & practically meaningful; 2) Validity diagonal element should be larger than others in the same row and column in the heterotrait-heteromethod triangles. 3) Validity diagonal element should be larger than others involving that trait in the heterotrait-monomethod triangles. 4) Pattern of correlations among traits should be the same in all heterotrait triangles

198
Q

Measurement

A

Application of qunatitaive values to behavioral samples Be refined enough to allow maximal transmission of information, but not too nuanced as to encourage measurement error.

199
Q

Internal Validity

A

The degree of correctness (truth, value, legitimacy) of inferences about the causal connections between focal constructs (the inference that x causes y).

200
Q

Item Difficulty in CTT

A

The proportion of test takers who endorse/get the item correct for dichotomous items, or the item’s mean for items that are polytomous.

201
Q

Item Discrimination in CTT

A

How well an item does at differentiating among test takers who have different levels of the trait being measured For CTT, Calculated by correlating the score on a particular item with total score on the scale (positive and .3 or greater seems to be the common heuristic)

202
Q

Cohen’s D

A

The amount of difference between 2 groups in standard deviation units

203
Q

Do’s and Don’t of Field Experiments in Orgs

A
204
Q

SEM Model Example

A
205
Q

Construct Validity

A
206
Q

Endogenous vs. exogenous variables

A
207
Q

Zickar 2020 Main Points

A
208
Q

Measure Development Steps

according to Zickar 2020

A
209
Q

Advantages and Disadvantages of

CTT

EFA

CFA

IRT

Zickar (2020)

A
210
Q
A
211
Q

Unified Validity

THE figure

Binning & Barrett 1989

A
212
Q

Questionable Research and Analyses Practices

(Campbell & Wilmont, 2018)

A
213
Q

Why null hypothesis significance testing is problematic

Campbell & Wilmont (2018)

A
214
Q

Positives of Theory

Campbell & Wilmont, 2018

A
215
Q

Steps of Theory Construction

A
216
Q

What theory is not

A
217
Q

4 Key ingredients to a theory

(Campbell & Wilmont, 2018)

A
218
Q

Meditation-product of coefficients approach

A

Uses the coefficients of the a and b paths to estimate the indirect path

  • Sobel test to assess significance
  • Preacher/Hayes Macro using bootstrapping- less conversative

(MacKinnon et al., 2010)

219
Q

Mediation- difference in coefficients approach

A

Estimates the effect by taking c-c’

aka the direct effect without mediators-direct effect with mediators to estimate the side

takes that and divides by standard error to determine significance

220
Q

Moderated meditation testing

A

Run regressions or SEMs for the two groups and see if there is a difference in parameters/fit.

(MacKinnon et al., 2007)

221
Q

3 step process to moderation

(Chan, 2011)

A
  1. Enter X and Z as predictors in block 1
  2. Enter XZ as predictor in block 2
  3. See whether there is a statistically significant increase in variance accounted for in the criterion
    1. if significant, plot the simple slopes to see the nature of the interaction
222
Q

Benefits of ESM

(Beal, 2015)

A

Reduction of memory biases

reduction of method biases

Ease of collection/implementation

223
Q

Key attributes of ESM

(Beal, 2015)

A
224
Q

Goal of ESM

A

To capture a representative range of experiences as they occur in their natural setting/environment (Beal, 2015)

225
Q

Issues with ESM

A
  1. Participant burnout or fatigue- careless responding, reduced response rates
  2. Responders can anticipate the next survey questions, which may influence their responses
  3. measurement equivalence- is the measure equivalent over each repeated measure
  4. missing data is inevitable
  5. internal validity issues– because of the repeated measures it can be hard to tell which caused which

Prelims (7 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions