ResearchDesignFlaws

Question 1

What are the 10 big design flaws

Answer 1

• no control group
• using the wrong statistical test
• misuse of correlation
• misuse of ANCOVA
• Misuse of Standard Error of Measurement (SEM)
• Regression Toward the Mean
• Sampling Problems
• Using Non-Equivalent Groups
• Confusing Discriminant and Convergent Validity
• Poor Reliability of Measurement/Confusing Reliability with Validity

Question 2

What threats are present if you don’t use a control group

Answer 2

• history
• maturation
• testing (if a pre-test design)
• instrumentation (if measurement could be changed from pre- to post- test)
• threats to internal validity present (resulting low generalizability)

Question 3

Statistical tests that use categorical variables

Answer 3

• chi-square
• logistical regression

Question 4

Statistical tests that use continuous variables

Answer 4

• t-tests
• ANOVA
• regression

Question 5

What are the two ways you could misuse correlation

Answer 5

• confusing correlation with causation
• wrongly concluding there is no correlation between two variables

Question 6

What type of relationship do you have when two variables are associated only because they are influenced by a third variable?

Answer 6

Spurious relationship

Question 7

You should only use ANCOVA to establish equivalency among groups if you used _____ assignment

Answer 7

random

Question 8

Standard error of Measurement is a measure of _____

Answer 8

precision

Question 9

Standard error of Measurement is NOT a measure of _____

Answer 9

validity

Question 10

You want Standard Error of Measurement to be small/large

Answer 10

small

Question 11

What is regression towards the mean

Answer 11

If you only look at extreme scores, when you re-test the scores are likely to be less extreme

Question 12

What are two sampling problems

Answer 12

• trying to generalize a sample of convenience to the entire population
• trying to generalize a specific population to a much larger population

Question 13

If you must use non-equivalent groups, you should at least use a ____ for comparison

Answer 13

pre-test

Question 14

What is Convergent Validity

Answer 14

measure of the same trait by different methods

should be correlated

Question 15

What is Discriminant Validity

Answer 15

the correlation between different traits measured by the same or different methods (should be smaller than convergent validity)

Question 16

When using a measure, we want a Cronbach’s Alpha of at least

Answer 16

.70, or at least .80 if there are 9+ items

Question 17

How can you increase reliability of a scale

Answer 17

Add additional items

Question 18

When you add additional items to a scale, you increase ___ but not ____

Answer 18

reliability, validity

Question 19

How should you examine rates of attrition?

Answer 19

Look to see if drop-out rates differ in some (systematic) way

Question 20

What has happened when you have poor operationalization of variables

Answer 20

your operationalization does not seem to really capture the actual construct or it capturs other things as well as the construct

Question 21

What do you have when the experimental manipulation does not feel real the the subjects

Answer 21

low experimental realism

Question 22

What do you have when the experimental manipulation does not mirror the real world

Answer 22

low mundane realism

Question 23

What words describe badly written measure items

Answer 23

• ambiguous
• double-barreled
• leading
• emotionally loaded
• over-demanding
• over-specific
• confusing (e.g., reverse-worded)

Question 24

It is difficult to generalize results if your scores have high ___ and ____

Answer 24

• variance,
• standard deviation

(i.e., it is hard to generalize results if items are spread out and there is little consistency)