7: Sampling and Ethics

Question 1

Regarding sampling, define:

• Universe
• Population
• Sample

Provide examples for each.

Answer 1

Universe: broad population to which you want to eventually generalize your findings to (e.g., all people with anorexia nervosa).

Population: defined group within the universe from which the sample will be selected (e.g., AN patients at 10 specific hospitals).

Sample: participants who actually take part in the study (e.g., 50 participants from those hospitals).

Question 2

Define a statistic versus a population parameter.

Answer 2

Statistic: quantitative measurement from a sample, what we actually get in research.

Population parameter: estimate of the value in the population. What we want to know, almost never know with certainty.

Question 3

Any single study can only generalize to the _____ of interest.

Answer 3

Population.

Question 4

Generalizing to the whole universe is sometimes done if mechanism is plausible, but it is a _____, not statistical inference.

Answer 4

Logical.

Question 5

All of the statistics that use p-values and confidence intervals depend upon a random sample. Why is it a problem when you don’t have a random sample?

Answer 5

The estimates from the statistics are liable to be biased away from those who do not participate.

Question 6

Population is defined using _____ and _____ criteria

Answer 6

Inclusion; exclusion.

Question 7

Define homogeneous and heterogeneous sampling.

Answer 7

Homogeneous: very strict selection criteria.

Heterogeneous: varied population with very few selection criteria.

Question 8

List one pro and two cons to homogeneous sampling.

Answer 8

Pro: reduces error, and thus increases statistical power.

Cons: reduced generalizability to the universe; creates difficulty in finding eligible participants.

Question 9

What are the two types of sampling in qualitative research?

Answer 9

Purposive sampling: selecting according to criteria important to research question; may be inclusion/exclusion criteria OR deliberately selecting broad range of demographics.

Theoretical sampling: in grounded theory, participants who are sampled may change as the theory develops during research. Data collection continues until theoretical saturation is reached (no new data).

Question 10

How can sample characteristics pose a threat to external validity? What is a solution?

Answer 10

Differences between study sample and other samples you wish to generalize the finding to (e.g. age, gender, etc.); many of these demographics could act as moderators (e.g., sample of women can’t generalize to men).

Solution: get a representative sample, or don’t generalize to people not represented by your sample.

Question 11

What is ecological validity and what is a solution to potential threats?

Answer 11

The methods, materials and setting of the study must approximate the real-life situation under investigation.

Solution: Try to make research approximate real-life OR test to make sure lab experiments are applicable to real-life decisions.

Question 12

How can reactivity of experimental arrangements pose a threat to external validity? What is a solution?

Answer 12

Awareness of being in a study may affect behavior or elicit reactions, may have theories about what is transpiring, may try to act as expected (to make study a success) or try to violate expectations (to make study fail).

Solution: don’t let participants know hypotheses ahead of time; use unobtrusive or implicit measures; use placebo treatment.

Question 13

What is multiple-treatment interference and what is a solution for it?

Answer 13

If researcher were to apply several treatments, difficult to determine how well each of the treatments would work individually.

Solution: if multiple treatments, use Treatment 1, Treatment 2, control, and “both” groups.

Question 14

Define novelty and disruption effects and provide a solution.

Answer 14

Novelty effect: treatment appears to work in the short term because it’s “new” and people are enthusiastic and have high expectations.

Disruption effect: treatment appears to fail in the short term because it changes daily routine, and takes some time to properly implement.

Solution: wait a while after a program has been implemented before evaluating effectiveness.

Question 15

What is test sensitivity? Provide an example. What is a solution?

Answer 15

Whether pre-testing or test itself alters subject experience and responses.

E.g., what effect does rating mood prior to a mood induction technique have on how you evaluate mood after manipulation?

Solution: avoid within-subject design, or multiple conditions varying the order of stimuli presentation.

Question 16

Describe two sample selection biases.

Answer 16

Convenience Sampling: Admitting anyone you can find into the study, usually from a population that is easy to recruit (e.g., students).

WEIRD research: Western Educated Industrialized Rich Democratic

Question 17

Regarding WEIRD research, meta-analysis between 2003-2007 of 6 top psychology journals found what two statistics?

Answer 17

80% of participants were psychology undergrads.

96% of samples come from countries that make up only 12% of the population.

Question 18

How many participants you need to have a successful experiment relies on what three pieces of information?

Answer 18

Effect size.

Alpha (chance for Type I error).

Beta (chance for Type II error).

Question 19

Effect size addresses what two questions?

Answer 19

How large is the effect in the population?

What is the smallest effect that is clinically relevant?

Question 20

List the two common standardized metrics for effect size and what is considered small, medium, and large.

Answer 20

Pearson’s r, for correlations; .10 = small; .30 = medium; .50 = large.

Cohen’s d, (Mean Difference) / Pooled Std. Dev.; .20 = small; .50 = medium; .80 = large.

Question 21

What does alpha mean? What is it usually set at?

Answer 21

Probability of finding an effect when none exists, also known as “Type I error” or “false positive.” Value set based on how confident want to be with results.

.05 (a 5% chance of making a Type I error).

Question 22

What does beta mean and how does it relate to statistical power? What does the value depend on and what is the conventional value?

Answer 22

Probability of missing an effect that is in fact present, also known as “Type II error” or “false negative.”

Statistical power is defined as 1 minus Beta.

Depends on sample size, population parameter for effect size - larger for both = more statistical power.

Conventional value for statistical power is .80 (20% chance of Type II error).

Question 23

Who were the participants in the Monster Study (1939) and what were the two conditions?

Answer 23

Participants: 22 orphan children.

Positive praise for children with stuttering to try and improve their condition.

Induce stuttering condition, where the researchers attempt to induce stuttering in healthy children.

Question 24

What were the consequences of the Monster Study (1939)?

Answer 24

University made public apology in 2001, six of the 10 people in the induce stuttering condition sued in 2003.

None developed stuttering, many had serious emotional scars - one child ran away from the orphanage to escape the “intervention.”

Question 25

Informed consent includes the idea of full information. Describe it. What are caveats to this?

Answer 25

Must tell participants everything they need to know to make a rational decision about participating in study, usually includes signed informed consent form.

Caveats: participants’ understanding needed; competence to provide consent; problems with deception.

Question 26

Describe freedom of choice, with regards to informed consent, and two potential issues.

Answer 26

Participation must be voluntary, without coercion.

Implicit coercion from clinicians; captive populations.

Question 27

Research should not harm participants. If harm is unavoidable, what is a prerequisite?

Answer 27

Potential gain to humanity must outweigh the harm done.

Question 28

Harm should be minimized in all ways possible. What is one way to do this?

Answer 28

Using as small a sample as possible.

Question 29

Describe five ethical considerations in which benefits can be withheld in clinical trials.

Answer 29

Control groups: harmed by withholding treatment?

Inflexible treatments: clinician cannot adapt using intuition or judgement.

Randomization: lack of choice of treatments.

Narrow inclusion criteria: often excludes co-morbid patients.

Referrals at termination: good research would dictate participants don’t seek additional treatment.

Question 30

With regards to ethics in research, define privacy and confidentiality.

Answer 30

Privacy: participant’s right not to provide information to the researcher if they feel it’s intrusive.

Confidentiality: data should be anonymous and even anonymous data should be accessible by only a few people.

Question 31

What are the three levels of ethics review?

Answer 31

Exempt: (a) Publicly available data; (b) research-like activities like quality assurance; (c) educational activities.

Expedited: usually reserved for modifications to an existing study.

Full Review: everything else; usually takes quite a while.

Question 32

In science, what is the difference between error and fraud?

Answer 32

Error is an honest mistake, fraud is intentional effort to deceive and misrepresent.

Question 33

Describe, briefly, the Burt Affair.

Answer 33

Controversy against Sir Cyril Burt, who was said to have published false data to support theory that intelligence is largely inherited.

Question 34

List six questionable research practices.

Answer 34

Changing hypotheses to fit data.

Selective reporting of favourable results.

Selective citation from (of) other studies.

Citing from secondary sources.

Incorrectly citing sources.

Failure to correct errors detected after publication.

Question 35

The Reproducibility Project found what two things?

Answer 35

Lower means, higher standard deviations.

Effect sizes in pre-registered studies were about half the size.