Lecture 2 Flashcards
Beta-
The probability of accepting the null when it is false (type 2 error)
Power
The probability of rejecting the null when it is false or the likelihood of finding differences between conditions when, in fact, the conditions are truly different
Effect size-
A way of expressing the difference between conditions in terms of common metric (across measures and studies). This refers to the magnitude of the difference between two or more conditions. Obtain the difference between the group means and divide by the common (pooled) standard deviation.
History-
Events in addition to the independent variable to which subjects are exposed that could influence their performance on the dependent variable/ Typically these events occur between repeated measurement. This could include events such as news reports, weather or common experience
Maturation-
observed changes as a result of ongoing, naturally occurring processes rather than independent variable. The threat is produced by internal, biological, physical or psychological changes in subjects. This could be physical changes, normal growth, typical developmental progression or typical social and general knowledge gains
Testing-
Subjects performance on a test is affected (or altered) as a result of prior measurement or assessment instead of the independent variable. The before study measures impact the after study measures, rather than the independent variable. Improved performance on the measures may be due to learning, memory and/or practice effects rather than intervention effects.
Instrumentations
Changes in measurement tools or measurement procedures may affect the outcome of the study (does not refer to changes in the study participants. Changes in instruments may occur when outcome variable are not measured in the same way at pre and posttest. Change in the research personnel can occur to the study personnel through experience, become more adept at measurement. In observational studies it may be wise to assess observer drift to see if raters are adhering to the original definitions in the observational protocol
Alpha
The probability of rejecting the null hypothesis when it is true (type 1 error
Beta
The probability of accepting the null when it is false (type 2 error)
Power
The probability of rejecting the null when it is false or the likelihood of finding differences between conditions when, in fact, the conditions are truly different.
Effect size
A way of expressing the difference between conditions in terms of common metric (across measures and studies). This refers to the magnitude of the difference between two or more conditions. Obtain the difference between the group means and divide by the common (pooled) SD.
History
Events in addition to the IV to which subjects are exposed that could influence their performance on the DV. Typically, these events occur between repeated measurement. this could include events such as news reports, weather or common exp.
Maturation
Observed changes as a result of ongoing naturally occurring processes rather than IV. The threat is produced by internal, biological, physical or psychological changes in subjects. This could be physical changes, normal growth, typical dx progression or typical social and general knowledge gains.
Testing
Subjects performance on a test is affected (or altered) as a result of prior measurement or assessment instead of the IV. The before study measures impact the after study measures, rather than the IV. Improved performance on the measures may be due to learning, memory, and/or practice effects.
Instrumentations
Changes in measurement tools or measurement procedures may affect the outcome of the study (does not refer to changes in the study participants. Changes in instruments may occur when outcome variables are not measured in the same way at pre and posttest. Changes in the research personnel can occur to the study personnel through experience - become more adept at measurement. In observational studies it may be wise to ssess observer drift to see if raters are adhering to the original definitions in the observational protocol.
Statistical regression
The changes over time expected in the performance of subjects that occur for statistical reasons, but might incorrectly be attributed to the IV (e.g., treatment). On repeated testing, there is a statistical tendnecy for the scores of any given sample to regress to the mean on any given measure. Thus increases or decreases in score may be a result of statistical regression rather than indicating change resulting from the interventions
Selection biases
participant characteristics may interact with the IV. Age, gender, socioeconomic status, ed. level and ethnicity may play important roles in the response to many interventions and treatments. This threat is very plausible when intact groups (quasi exp). or self-selected participants are used. It is important that during the design phase to determine what characteristics may impact the outcome and exert control accordingly.
Combination of selection and other threats
this may occur when threats to internal validity vary for the different groups within the study. Selection methods may interact with the other threats to internal validity further biasing the results (history, maturation, testing, instrumentation, attrition). Selection-history threat is plausible when one group is exposed to a particular event that a comparison group did not.
Mortality/attrition
differential drop out across conditions at one or more time points that may be responsible for outcomes rather than the intervention itself. The longer and more involved a study is, the higher rate of participant’s attrition. This may make before and after samples non-comparable. Is there an attrition bias such that subjects later in the research process are no longer representative of the larger initial group?
Special Problems involving control groups: Treatment imitation or diffusion
When those in the control group learn of the exp. arrangements or are accidentally exposed to the intervention. Thus, control group members may benefit from info. given to the treatment group. As a result, similar performance on outcome variables occurs between tx and control groups, obscuring any treatment effects.
Special Problems involving control groups: Compensatory rivalry
Occurs when one group, typically the comparison or control group, learns of differences with the other group, typically the experimental or treatment group. Awareness of differences may create feelings of competition, which may bias attributions made to intervention effects. My lead to competitive attitudes where the control group may work harder to compete with the other group.
Special Problems involving control groups: Resentful demoralization
When one group, typically the comparison or control group, learns of differences with the other group, typically the exp or tx group. Awareness of differences may create feelings of deprivation, which may bias attributions made to intervention effects. Control group participants may become discouraged/ angry leading to psych or actual withdrawal from the study. They may also “not try hard” in light of the known differences
Minimize history/maturation by
the use of a control group, selected from the same pop. as the exp. group can help rule out the effects of hx and maturation. Also, the shorter the duration of an exp., the less likely hx and maturation will be a threat.
Minimize testing threat by:
The use of a research design that does not include a pretest caneliminate testing as a potential threat to internal validity. It may also help to use different but equivalent forms of a test for pre and posttest.
Minimize instrumentation threat by:
careful specification and control of the measurement procedures can eliminate most instrumentation threat. Standardized instruments, administration or data collection procedures and the training of study personnel are among the procedures that help control the instrumentation threat.
Minimize stat. regression threat by:
Typically caused by bias associated with extreme scorers. Avoid assigning subjects to groups based on their extreme scores. While random assignment should theoretically result in a similar distribution of scores this should still be evaluate in each group separately (i.e., levels of the IV)
Minimize attrition threat by:
recruiting larger groups of participants or more than needed for stat. analyses. Include incentives and compensation as appropriate.
Minimize the selection threat by:
Utilizing random selection and random assignment of subjects. If random selection and assignment are not possible, then use certain statistical techniques (e.g., ANOVA) that can adjust for group differences.
Statistical conclusion validity
Refers to the facets of the quantitative evaluation that influence the conclusions reached about a cause and effect relationship. It is impacted by sampling procedures, statistical tests, and measurement tool/procedures. conclusions regarding a casual relationship rely on hypothesis testing. The nul hypothesis (H0) would specify that there are no differences between groups (i.e., exp. vs. control conditions). Are the differences we obtain reliable that would allow us to reject the null hypothesis? Specify a probability level a priori.
Low statistical power (threat to statistical conclusion validity)
Statistical power refers to the extent to which we can detect difference between groups when differences exist within the population. This is the most common threat. When power is weak, the likelihood of a type 2 error is increased. The most common way to increase power is by increasing the sample size.
Variability in the procedures (threat to statistical conclusion validity)
Lack of consistency in the execution of the exp. procedure will affect the statistical validity. By minimizing any procedural variability, the likelihood of detecting a true difference between groups is increased. We must minimize sources of variability in the study design and in its implementation (utilizing protocols and administering treatment in a standardized fashion).
subject heterogeneity (threat to statistical conclusion validity)
The more variability in subject characteristics=greater the variability in the subjects reactions to the measures and the intervention.This can make less likely that one will be able to detect difference between conditions. This is especially true when the subjects are heterogeneous on a characteristic that is correlated with the independent variable. This threat could be addressed either by..choosing a more homogenous sample or using a heterogeneous sample and evaluate the effect of the relevant characteristic.
Unreliability of measures (threat to statistical conclusion validity)
As measurement error increases reliability of the measurement decreases. Thus, a greater portion of the subjects score will be due to random variation. Thus, if one uses unreliable measures, the obtained effect size is likely to be lower.
multiple comparisons and error rates (threat to statistical conclusion validity)
A potential threat to the statistical validity deals with the number of statistical tests that will be completed. The more tests the more likely a chance difference will be found. Note that 1 relationship in 20 (e.g. pairwise comparisons) will be found to be statistically significant just by chance alone. By definition of a probability value (p-value or alpha) .05 significance. Thus as the number of tests increase so does the experiment-wise error rate.
