Chapter 10- Introduction to simple experiments

Question 1

Mueller and Oppenheimer 2014

Answer 1

Students were randomly assigned to take notes on a lecture using either a notebook or a laptop. They then took a test on what they learned in the lecture. Both groups scored equally on factual information, but the longhand group scored higher on conceptual information. Two other studies had the same finding

Question 2

Leonard 2017

Answer 2

Babies were assigned to the effort or no effort condition. Babies in the effort condition watched a model attempting various strategies to get a plastic frog out of a box and took multiple tries to get a toy off a carabiner. In the no effort condition, the model solved each problem 3 times, with no additional strategies to try to solve it. The babies were then given a toy that played music, and attempted to press a button that looked like it started the music, but was actually inert. The babies in the effort group pressed the button 11 more times than the babies in the no effort group.

Question 3

Experiment

Answer 3

When researchers manipulated at least one variable and measured another

Question 4

Manipulated variable

Answer 4

A variable that was controlled, like when researchers assign participants to a particular level of a variable. Mueller study- note taking was a variable because it had 2 levels, and was manipulated by flipping a coin to assign participants to one level.

Question 5

Measured variable

Answer 5

Take the form of records of behavior or attitudes, like self reports, behavioral observations, or physiological measures. In the Mueller study, researchers measured student performance on test questions.

Question 6

Independent variable

Answer 6

In an experiment, the independent variable is the manipulated (causal) variable. Conditions- levels of the independent variable, like effort or no effort

Question 7

Dependent variable

Answer 7

The measured variable or outcome variable. How a person acts on the measured variable depends on the level of the independent variable. A dependent variable is not the same as its levels, either. In the Leonard study, the dependent variable was the number of button presses (not “25 presses”).

Question 8

How many of each variable does an experiment need to have?

Answer 8

Experiments must have at least one independent and one dependent variable, but they often have more than one dependent variable. In the notetaking study, the dependent variables were performance on factual and conceptual questions.

Question 9

Control variable

Answer 9

A variable that a researcher holds constant on purpose. In the note taking study, participants answered the same questions, watched the same videos in the same room, and had the same experimenter. Control variables aren’t really variables at all because they don’t vary.

Question 10

What is the purpose of a control variable?

Answer 10

When researchers manipulate independent variables, they need to make sure they’re only varying one thing- the potential causal force. Therefore, they also control potential third variables in their study by holding all other factors constant between the levels of the independent variable. Control variables are important for establishing internal validity.

Question 11

How do independent variables in experiments establish covariance?

Answer 11

True experiments manipulate an independent variable, which has at least 2 levels. Because independent variables vary and offer a comparison group, we are able to establish covariance. However, results also matter- if there was no difference in how babies behaved in the 2 conditions, the study would have found no covariance, and researchers could not conclude a causal relationship.

Question 12

Control group

Answer 12

A control group is a level of an independent variable that is intended to represent “no treatment” or a neutral condition. Not every experiment has or needs a control group. In the note taking study, there were two levels to the independent variable, but neither was a control group because there was no “no note taking” condition.

Question 13

Treatment group

Answer 13

When a study has a control group, the other level(s) of the independent variable are usually called the treatment groups.
In an experiment testing the effectiveness of a new medication, the researchers might assign some participants to take the medication (treatment group) and some to take a sugar pill (control group).

Question 14

How do experiments establish temporal precedence?

Answer 14

By manipulating the independent variable, researchers are able to establish temporal precedence. In the Leonard study, the independent variable was manipulated and babies were able to observe the model’s behavior. Then, the babies’ behavior was observed. This ensures that the cause comes before the effect (outcome). Experiments unfold over time, in contrast to a correlational study where all variables are measured at once.

Question 15

Internal validity

Answer 15

To be internally valid, the experiment must establish that the causal variable, and not other factors, are responsible for the change in the outcome variable.

Question 16

How can internal validity be interrogated in an experiment?

Answer 16

You can interrogate internal validity by exploring potential alternative explanations. For example, were students in one note taking group given more difficult questions? Since both groups got the same test, the difficulty of the test was a control variable and didn’t influence the outcome.

Question 17

Confounds

Answer 17

There can be several possible alternative explanations for a research question, which can be threats to internal validity. When a study has a confound, you are confused about what’s causing the change in the dependent variable. Is it the intended causal variable, or something else?

Question 18

Design confound

Answer 18

An experimenter’s mistake in designing the independent variable- it occurs when a second variable happens to vary systematically along with the intended independent variable. If adult models in the baby study have accidentally exhibited more cheerful attitudes in the effort than in the no-effort group, the study would have a design confound because cheerfulness would vary along with effort (the independent variable).

Question 19

Which validity do design confounds threaten?

Answer 19

The accidental second variable is an alternative explanation for the results and threatens internal validity.

Question 20

How do we know if a potentially problematic variable is a design confound?

Answer 20

Not every potentially problematic variable is a confound. It might be the case that some of the adult models were cheerful and others were more reserved. Every individual has differences, but the variability of emotional expression is only a problem for internal validity if it shows systematic variability with the independent variable (if the cheerful models only worked in the effort condition). If there was unsystematic variability (random or haphazard) in expression across groups, it would not be a confound.

Question 21

Selection effects

Answer 21

When the kinds of participants in one level of the independent variable are systematically different from those in the other. This can happen when the experimenters let participants choose which group they want to be in, or if one group of people (only women, or only people who sign up early in the semester) to one condition, and the other type of people (only men) to the other.

Question 22

How can selection effects be prevented?

Answer 22

Well designed experiments often use random assignment to avoid selection effects, like flipping a coin to assign people to each group.

Question 23

Random assignment

Answer 23

This means that each participant has an equal chance of being in one condition or the other. Using deliberately unsystematic means of assigning groups splits up one type of participant. For example, if 12 babies were more interested in the study, probabilistically 6 babies would be put in one group and 6 in the other. This makes experimental groups virtually equal before the independent variable is applied. After random assignment, researchers should be able to test the groups for intelligence, motivation, etc and get similar results between groups.

Question 24

Matched groups

Answer 24

Researchers would measure participants on a certain variable that would matter to the dependent variable, like GPA could matter for note taking. The researchers would then match up the participants with the two highest GPAs in pairs, and then assign one of each pair to each group, and go down the list until the lowest GPAs had been matched and assigned to each group.

Question 25

When are matched groups used?

Answer 25

For certain studies, researchers might want to be absolutely sure that the experimental groups are as equal as possible. In that case, they would use matched groups.

Question 26

What is the advantage of matched groups?

Answer 26

Matching has the advantage of randomness- each member of the matched pair is randomly assigned, so this technique prevents selection effects. It also ensures that groups are equal on an important variable, like GPA.

Question 27

What is the disadvantage of matched groups?

Answer 27

The disadvantage is that the matching process requires an extra step, like finding out people’s GPA. It requires extra resources.

Question 28

Independent groups design

Answer 28

Separate groups of participants are placed into different levels of the independent variable. Also called between subjects design or between groups design. In both the note taking and baby study, participants were assigned to one of two levels of the independent variable.

Question 29

Within groups design/within subjects design

Answer 29

Each person is presented with all levels of the independent variable. In the note taking study, an independent groups design was used. However, it would have been within groups if researchers asked each participant to take notes on two videos, using a laptop for one and a notebook for the other.

Question 30

2 forms of independent groups design

Answer 30

Posttest-only design and the pretest/posttest design

Question 31

Posttest only design

Answer 31

Randomly assigns participants to independent variable groups and are tested on the dependent variable once. The notetaking study is an example, with only two independent variable levels. Participants were assigned to one condition and were only tested once on the video they watched.

Question 32

How do posttest only designs satisfy all 3 criteria for causation?

Answer 32

1. Covariance- detects difference in the dependent variable since there are 2 groups.
2. Temporal precedence- the independent variable comes first in time
3. When conducted well, they establish internal validity. When appropriate control variables are used, there are no design confounds, and random assignment gets rid of selection effects.

Question 33

Pretest/posttest design

Answer 33

Participants are randomly assigned to at least two groups and are tested on the key dependent variable twice- once before and once after the exposure to the independent variable.

Question 34

Example of pretest/posttest design

Answer 34

One study on the effects of mindfulness assigned students randomly to participate in either a 2 week mindfulness or nutrition class. One week before and one week after the class, students completed one verbal reasoning section of the GRE of the same difficulty. The mindfulness group improved significantly better at the posttest than the pretest.

Question 35

Advantage of pretest/posttest design

Answer 35

Pretest/posttest design helps researchers to make sure there was no selection effect (from seeing pre test scores were very similar). It also allows researchers to track people’s progress over time.

Question 36

Is pretest/posttest or posttest only design better?

Answer 36

In some cases, pretest/posttest can’t be used. For the baby study, babies would likely get frustrated and stop trying if the study went on for too long.

Question 37

2 basic types of within groups designs

Answer 37

1. Repeated measures design

2. Concurrent-measures design

Question 38

Repeated measures design

Answer 38

When participants are measured on a dependent variable more than once, after exposure to each level of the independent variable.

Question 39

Example of repeated measures design

Answer 39

Boothby et al., 2014- researchers hypothesized that sharing a good experience with another person makes it even better than it would have been if experienced alone, and that sharing a bad experience would make it even worse. The participants ate chocolate both with a confederate and by themselves. People liked the chocolate more when the confederate was also tasting it. The independent variable had two levels: sharing and not sharing an experience. Participants experienced both levels, making it a within-groups design. The dependent variable was participants’ rating of the chocolate. It was a repeated-measures design because each participant rated the chocolate twice (repeatedly).

Question 40

Concurrent-measures design

Answer 40

Participants are exposed to all the levels of an independent variable at roughly the same time, and a single attitudinal or behavioral preference is the dependent variable.

Question 41

Example of concurrent-measures design

Answer 41

In one study, researchers showed infants a male and female face at the same time, and recorded which one looked at each face the longest. The independent variable was the gender of the face, and babies experienced both levels (male and female) at the same time. The dependent variable was the baby’s looking preference

Question 42

Advantages of within-groups designs

Answer 42

1. Ensures the participants in the two groups will be equivalent (since they are the same participants)- the only difference between the two conditions can be attributed to the independent variable.
2. Allows researchers to make more precise estimates of the differences between conditions. When unsystematic variability in various characteristics is held constant across conditions, there is less extraneous error when measuring the effect of the manipulation of the independent variable.
3. Generally requires fewer participants overall, since the same group is experiencing all levels of the independent variable

Question 43

Why can matched group designs be treated as within groups designs?

Answer 43

The idea of treating each participant as “their own control” means that matched group designs can be treated as within groups designs, since matched participants are more similar to each other than traditional independent groups design.

Question 44

How do within groups designs establish covariance?

Answer 44

Within groups designs allow researchers to manipulate an independent variable and incorporate comparison conditions, they provide an opportunity for establishing covariance.

Question 45

How do within groups designs establish temporal precedence?

Answer 45

The experimenter controls the independent variable and ensures that it comes first. The infants saw the faces first, and then looking time was measured.

Question 46

For within groups designs, what do researchers have to be concerned with when it comes to internal validity?

Answer 46

For internal validity, researchers don’t need to worry about selection validity since the participants are the same for both conditions.
They do need to avoid design confounds. For example, in the chocolate tasting study, participants tasted chocolate that was exactly the same.

Question 47

Order effects

Answer 47

Sometimes, being exposed to a particular condition first changes how participants respond to the later condition. This is called order effects- it occurs when exposure to one level of the independent variable influences responses to the next level. When it occurs in a within groups design, it’s considered a confound

Question 48

Examples of order effects (2)

Answer 48

1. Practice effects

2. Carryover effects

Question 49

Practice effects

Answer 49

When a long sequence might lead participants to get better at the task or to get tired or bored by the end.

Question 50

Carryover effects

Answer 50

When some form of contamination carries over from one condition to the next. Drinking orange juice right after brushing your teeth would contaminate the experience.

Question 51

Counterbalancing

Answer 51

When researchers present the levels of the independent variable to participants in different sequences. The sequences are determined by researchers, and participants are randomly assigned to each. Example- chocolate tasting experiment. One group sequence is: taste chocolate alone, rate chocolate, taste chocolate with confederate, rate chocolate. The other group switches the chocolate tasting steps to tasting chocolate with the confederate comes first.

Question 52

What is the purpose of counterbalancing?

Answer 52

This avoids order effects- order effects should cancel each other out when the data is combined.

Question 53

2 methods for counterbalancing an experiment

Answer 53

1. Full counterbalancing

2. Partial counterbalancing

Question 54

Full counterbalancing

Answer 54

When there are two or three levels of the independent variable, all possible condition orders are represented. With 2 levels, there are only 2 possible orders. With 3 levels, there are 6.

Question 55

Partial counterbalancing

Answer 55

Only some of the possible condition orders are represented, when there are 4 or more conditions. One way to do this is present the conditions in a randomized order for every subject. Also Latin square- a formal system to ensure that every condition appears in each position at least once.

Question 56

3 main disadvantages of within groups design

Answer 56

1. Potential for order effects in repeated-measures designs
2. Within-groups design might not be possible or practical. For example, in a study teaching children methods for riding a bike, you can’t go back to baseline to teach them both methods.
3. Demand characteristic

Question 57

Demand characteristic

Answer 57

A cue that leads participants to guess an experiment’s hypothesis. This creates an alternative explanation for the study’s results. For example, if people have to rate the attractiveness of one white person and one black person, for example, they might guess that the study is about prejudice.

Question 58

Is pretest/posttest a repeated-measures design?

Answer 58

No. In a pretest/posttest design, participants only see one level of the independent variable, not all levels.

Question 59

How is the construct validity of dependent variables in an experiment interrogated?

Answer 59

You would ask how well the researchers measured their dependent variables (in note taking study- factual and conceptual knowledge). The questions had good face validity and high interrater reliability when the answers were graded.

Question 60

How is the construct validity of independent variables in an experiment interrogated?

Answer 60

You would ask how well the researchers manipulated or operationalized the variables. In the notetaking study, participants received either a laptop or a notebook, so the variable was clearly operationalized.

Question 61

Manipulation check

Answer 61

An extra dependent variable that researchers can insert into the experiment to convince them that their experimental manipulation worked. Manipulation checks will likely be used when the intention is to make participants think or feel a certain way. They can manipulate feelings of anxiety by telling participants they want them to give a public speech.

Question 62

Pilot study

Answer 62

A study using a separate group of participants, that is completed before or after the study of primary interest to confirm the effectiveness of the manipulations.

Question 63

How does construct validity relate to theory testing?

Answer 63

Experiments are designed to test theories. Interrogating the construct validity of an experiment requires you to evaluate how well the measures and manipulations researchers used in their study capture the conceptual variables in their theory. Study results either support or don’t support a theory.

Question 64

How do we determine a causal claim can be generalized to other people?

Answer 64

When interrogating external validity, you ask about random sampling- randomly gathering a sample from the population. If there was random sampling, the results can typically be generalized, at least to the population the participants came from.

Question 65

How do we determine a causal claim can be generalized to other situations?

Answer 65

The note taking study might generalize to other ted talk lectures, but not necessarily to other situations, like live college lectures

Question 66

Which validity is generally prioritized in experiments?

Answer 66

In an experiment, researchers usually prioritize internal validity- it can be difficult to ensure both internal and external validity.

Question 67

Which questions do we ask to determine statistical validity? (2)

Answer 67

1. How large is the effect?

2. How precise is the estimate?

Question 68

Why do we want to know effect size?

Answer 68

How large is the difference between the groups of the independent variable? Asking this question helps to evaluate covariance. In general, the larger the difference, the stronger the causal effect.

Question 69

2 ways to express effect size

Answer 69

1. To use the original units

2. To use a standardized effect size. When there are two groups in the experiment, we use a d value.

Question 70

d value

Answer 70

This takes into account both the difference between means and the spread of scores within each group (the standard deviation). When d is large, it means the independent variable caused a large change in the dependent variable, relative to how spread out the scores are.
When d is small, it means that the scores of participants in the two experimental groups overlap more

Question 71

Which d values are considered small, medium, and large?

Answer 71

Generally, a d of .2 is small, .5 is medium, and .8 is large

Question 72

How do we determine the precision of the estimate?

Answer 72

We can compute the 95% confidence interval. The confidence interval will be more precise when the sample is larger and has less variability. If the CI doesn’t include zero, it’s statistically significant

Question 73

Replication

Answer 73

Another step in estimating the population effect is to conduct the study again and find multiple estimates. When studies have been replicated, you can combine their estimates to get an even better estimate of the true population value.

Question 74

How do we know if a causal claim is appropriate?

Answer 74

If the internal validity is sound, you can assume the causal claim is appropriate. If there is a confound, the causal claim should be demoted to an association claim.

Question 75

3 internal validity questions to ask of any experiment

Answer 75

1. Did the experimental design ensure that there were no design confounds, or did some other variable accidentally covary along with the intended independent variable?
2. If the experimenters used an independent groups design, did they control for selection effects by using random assignment or matching?
3. If the experimenters used a within groups design, did they control for order effects by counterbalancing?