Lecture 2

Question 1

Beta-

Answer 1

The probability of accepting the null when it is false (type 2 error)

Question 2

Power

Answer 2

The probability of rejecting the null when it is false or the likelihood of finding differences between conditions when, in fact, the conditions are truly different

Question 3

Effect size-

Answer 3

A way of expressing the difference between conditions in terms of common metric (across measures and studies). This refers to the magnitude of the difference between two or more conditions. Obtain the difference between the group means and divide by the common (pooled) standard deviation.

Question 4

History-

Answer 4

Events in addition to the independent variable to which subjects are exposed that could influence their performance on the dependent variable/ Typically these events occur between repeated measurement. This could include events such as news reports, weather or common experience

Question 5

Maturation-

Answer 5

observed changes as a result of ongoing, naturally occurring processes rather than independent variable. The threat is produced by internal, biological, physical or psychological changes in subjects. This could be physical changes, normal growth, typical developmental progression or typical social and general knowledge gains

Question 6

Testing-

Answer 6

Subjects performance on a test is affected (or altered) as a result of prior measurement or assessment instead of the independent variable. The before study measures impact the after study measures, rather than the independent variable. Improved performance on the measures may be due to learning, memory and/or practice effects rather than intervention effects.

Question 7

Instrumentations

Answer 7

Changes in measurement tools or measurement procedures may affect the outcome of the study (does not refer to changes in the study participants. Changes in instruments may occur when outcome variable are not measured in the same way at pre and posttest. Change in the research personnel can occur to the study personnel through experience, become more adept at measurement. In observational studies it may be wise to assess observer drift to see if raters are adhering to the original definitions in the observational protocol

Question 8

Alpha

Answer 8

The probability of rejecting the null hypothesis when it is true (type 1 error

Question 9

Beta

Answer 9

The probability of accepting the null when it is false (type 2 error)

Question 10

Power

Answer 10

The probability of rejecting the null when it is false or the likelihood of finding differences between conditions when, in fact, the conditions are truly different.

Question 11

Effect size

Answer 11

A way of expressing the difference between conditions in terms of common metric (across measures and studies). This refers to the magnitude of the difference between two or more conditions. Obtain the difference between the group means and divide by the common (pooled) SD.

Question 12

History

Answer 12

Events in addition to the IV to which subjects are exposed that could influence their performance on the DV. Typically, these events occur between repeated measurement. this could include events such as news reports, weather or common exp.

Question 13

Maturation

Answer 13

Observed changes as a result of ongoing naturally occurring processes rather than IV. The threat is produced by internal, biological, physical or psychological changes in subjects. This could be physical changes, normal growth, typical dx progression or typical social and general knowledge gains.

Question 14

Testing

Answer 14

Subjects performance on a test is affected (or altered) as a result of prior measurement or assessment instead of the IV. The before study measures impact the after study measures, rather than the IV. Improved performance on the measures may be due to learning, memory, and/or practice effects.

Question 15

Instrumentations

Answer 15

Changes in measurement tools or measurement procedures may affect the outcome of the study (does not refer to changes in the study participants. Changes in instruments may occur when outcome variables are not measured in the same way at pre and posttest. Changes in the research personnel can occur to the study personnel through experience - become more adept at measurement. In observational studies it may be wise to ssess observer drift to see if raters are adhering to the original definitions in the observational protocol.

Question 16

Statistical regression

Answer 16

The changes over time expected in the performance of subjects that occur for statistical reasons, but might incorrectly be attributed to the IV (e.g., treatment). On repeated testing, there is a statistical tendnecy for the scores of any given sample to regress to the mean on any given measure. Thus increases or decreases in score may be a result of statistical regression rather than indicating change resulting from the interventions

Question 17

Selection biases

Answer 17

participant characteristics may interact with the IV. Age, gender, socioeconomic status, ed. level and ethnicity may play important roles in the response to many interventions and treatments. This threat is very plausible when intact groups (quasi exp). or self-selected participants are used. It is important that during the design phase to determine what characteristics may impact the outcome and exert control accordingly.

Question 18

Combination of selection and other threats

Answer 18

this may occur when threats to internal validity vary for the different groups within the study. Selection methods may interact with the other threats to internal validity further biasing the results (history, maturation, testing, instrumentation, attrition). Selection-history threat is plausible when one group is exposed to a particular event that a comparison group did not.

Question 19

Mortality/attrition

Answer 19

differential drop out across conditions at one or more time points that may be responsible for outcomes rather than the intervention itself. The longer and more involved a study is, the higher rate of participant’s attrition. This may make before and after samples non-comparable. Is there an attrition bias such that subjects later in the research process are no longer representative of the larger initial group?

Question 20

Special Problems involving control groups: Treatment imitation or diffusion

Answer 20

When those in the control group learn of the exp. arrangements or are accidentally exposed to the intervention. Thus, control group members may benefit from info. given to the treatment group. As a result, similar performance on outcome variables occurs between tx and control groups, obscuring any treatment effects.

Question 21

Special Problems involving control groups: Compensatory rivalry

Answer 21

Occurs when one group, typically the comparison or control group, learns of differences with the other group, typically the experimental or treatment group. Awareness of differences may create feelings of competition, which may bias attributions made to intervention effects. My lead to competitive attitudes where the control group may work harder to compete with the other group.

Question 22

Special Problems involving control groups: Resentful demoralization

Answer 22

When one group, typically the comparison or control group, learns of differences with the other group, typically the exp or tx group. Awareness of differences may create feelings of deprivation, which may bias attributions made to intervention effects. Control group participants may become discouraged/ angry leading to psych or actual withdrawal from the study. They may also “not try hard” in light of the known differences

Question 23

Minimize history/maturation by

Answer 23

the use of a control group, selected from the same pop. as the exp. group can help rule out the effects of hx and maturation. Also, the shorter the duration of an exp., the less likely hx and maturation will be a threat.

Question 24

Minimize testing threat by:

Answer 24

The use of a research design that does not include a pretest caneliminate testing as a potential threat to internal validity. It may also help to use different but equivalent forms of a test for pre and posttest.

Question 25

Minimize instrumentation threat by:

Answer 25

careful specification and control of the measurement procedures can eliminate most instrumentation threat. Standardized instruments, administration or data collection procedures and the training of study personnel are among the procedures that help control the instrumentation threat.

Question 26

Minimize stat. regression threat by:

Answer 26

Typically caused by bias associated with extreme scorers. Avoid assigning subjects to groups based on their extreme scores. While random assignment should theoretically result in a similar distribution of scores this should still be evaluate in each group separately (i.e., levels of the IV)

Question 27

Minimize attrition threat by:

Answer 27

recruiting larger groups of participants or more than needed for stat. analyses. Include incentives and compensation as appropriate.

Question 28

Minimize the selection threat by:

Answer 28

Utilizing random selection and random assignment of subjects. If random selection and assignment are not possible, then use certain statistical techniques (e.g., ANOVA) that can adjust for group differences.

Question 29

Statistical conclusion validity

Answer 29

Refers to the facets of the quantitative evaluation that influence the conclusions reached about a cause and effect relationship. It is impacted by sampling procedures, statistical tests, and measurement tool/procedures. conclusions regarding a casual relationship rely on hypothesis testing. The nul hypothesis (H0) would specify that there are no differences between groups (i.e., exp. vs. control conditions). Are the differences we obtain reliable that would allow us to reject the null hypothesis? Specify a probability level a priori.

Question 30

Low statistical power (threat to statistical conclusion validity)

Answer 30

Statistical power refers to the extent to which we can detect difference between groups when differences exist within the population. This is the most common threat. When power is weak, the likelihood of a type 2 error is increased. The most common way to increase power is by increasing the sample size.

Question 31

Variability in the procedures (threat to statistical conclusion validity)

Answer 31

Lack of consistency in the execution of the exp. procedure will affect the statistical validity. By minimizing any procedural variability, the likelihood of detecting a true difference between groups is increased. We must minimize sources of variability in the study design and in its implementation (utilizing protocols and administering treatment in a standardized fashion).

Question 32

subject heterogeneity (threat to statistical conclusion validity)

Answer 32

The more variability in subject characteristics=greater the variability in the subjects reactions to the measures and the intervention.This can make less likely that one will be able to detect difference between conditions. This is especially true when the subjects are heterogeneous on a characteristic that is correlated with the independent variable. This threat could be addressed either by..choosing a more homogenous sample or using a heterogeneous sample and evaluate the effect of the relevant characteristic.

Question 33

Unreliability of measures (threat to statistical conclusion validity)

Answer 33

As measurement error increases reliability of the measurement decreases. Thus, a greater portion of the subjects score will be due to random variation. Thus, if one uses unreliable measures, the obtained effect size is likely to be lower.

Question 34

multiple comparisons and error rates (threat to statistical conclusion validity)

Answer 34

A potential threat to the statistical validity deals with the number of statistical tests that will be completed. The more tests the more likely a chance difference will be found. Note that 1 relationship in 20 (e.g. pairwise comparisons) will be found to be statistically significant just by chance alone. By definition of a probability value (p-value or alpha) .05 significance. Thus as the number of tests increase so does the experiment-wise error rate.