module 9 Flashcards
Experiment
in an experiment researchers manipulate at least one variable and measure at least one other variable
Lab experiment
Experiment in artificial environment
Field experiments
Experiment in a natural setting
Everyday real life environment of participant
–> setting participants manipulators and outcome measures are authentic
example difference field experimental
Suppose you would like to know whether running a large ad on Google is more effective than running a small ad. Put differently, is it worth the money to run a large ad? You could opt for a lab experiment and invite people to the lab, where you ask them to imagine that they are searching Google. You show half of them a Google page (fabricated by you) with a small ad, and the other half of them a Google page with a large ad (also fabricated by you). Afterwards, you ask all participants to fill out a small questionnaire comprising, amongst others, a question on whether or not they would be inclined to click on the ad they just saw.
Alternatively, you could run a field experiment in which you manipulate ad size on an actual Google page, with half of the Google page visitors seeing the large ad and the other half the small ad. You could then measure the number of people that click on the ads.
The field experiment has:
an authentic context: an actual Google search (rather than having people imagine that they are searching Google);
authentic participants: people who genuinely search on Google (rather than people who may never search Google but are now instructed to do so)
authentic treatments: real ads (rather than fabricated ads)
authentic outcome measures: actual click rates (rather than asking people whether they would be inclined to click on the ad if they were in the real world)
Experiments (lab or field) are particularly suitable when
The number of independent (and moderator) variables is limited and
At least one independent variable can be manipulated
When is a lab experiment preferred over a field experiment
When researchers want maximum control over the research environment to rule out alternative explanations
When is a field experiment preferable over a lab experiment
When it is essential to measure real-world behavior in real-world situations i.e. when high external validity is crucial. After running a lab experiment, researchers can only speculate to what extent their findings would apply to a real-world setting. After running a field experiment, they know their findings apply to a real world setting
The levels of the manipulated variable are referred to as
Conditions
control group
A level of the independent variable that represents a neutral condition. When a study has acontrol group, the other levels of the variables are called the treatment groups
between subject experiment design
Different groups of subjects are assigned to different levels of the independent variable
within subject experiment design
Each subject (participant) is presented with all levels of the independent variable
Posttest only experiment design
Simplest between subject design
Subjects are randomly assigned to the levels of the independent variable
The dependent variable is then measured once
Pretest/posttest experiment design
Subjects are randomly assigned to the levels of an independent variable
The dependent variable is measured twice: once before and once after exposure to the independent variable
Pretest/posttest experiment design
Subjects are randomly assigned to the levels of an independent variable
The dependent variable is measured twice: once before and once after exposure to the independent variable
Why would researchers use a posttest only design, why not always use a pretest/posttest design?
Sometimes it may be problematic to measure the dependent variable beforehand, as it may influence the second measurement. IF the pretest makes participants change their subsequent behavior/reaction, a pretest should be avoided
Interaction or moderator effect
Whether the effect of one independent variable depends on the level of the other independent variable
Factorial research:
Combine the two independent variables: they study each possible combination of the independent variables
Factorial dsign
design with more than one IV
Test for interaction / moderation effects
Illustration: factorial design:
Hypothesis:
Tag ons increase recall, but have a negative effect on attitude (boredom and or irritation)
The negative effect of tag-ons can be prevented by varying the tag-on
Experimental design:
IV1: tag-ons
IV2: variation
DV: Attitude towards ad
In a within subjects factorial design, both independent variables are manipulated as
Within subjects. Therefore if the design is a 2x2 design, there is only one group of subjects in the experiment but they participate in all four cells (or combinations) of the design
In mixed factorial designs, one independent variable is manipulated as
Between subjects and one independent variable is manipulated as within subjects
The notation for a factorial design with two independent variables is “a x b”, where:
a indicates the number of levels of the first independent variable
B indicates the number of levels of the second independent variable
An example:
A 3x4 factorial design has two independent variables, one with three levels and one with four levels. It results in 12 cells (3x4 = 12).
Sometimes, research studies have three independent variables. Such a design is called a three-way design. For example, in a 2x2x2 factorial design, there are two levels of the first independent variable, two levels of the second independent variable, and two levels of the third independent variable.
This leads to how many cells or condition in the experiment?
eight cells or conditions in the experiment (2x2x2 = 8).
Three way factorial designs are complex to interpret. A three-way interaction means that the two-way interaction between two of the independent variables depends on the level of the third independent variable
Concrete measure of dependent variable
E.g. the number of M&Ms eaten
abstract measurement of dependent variable
E.g. the perceived tastiness of an M&M
Level at which chronbachs alpha is acceptable
> 0.70
–> items in a measurement instrument are internally consistent
The validity of measured variables (such as the dependent variable in an experiment) can be demonstrated by:
Providing precedence (has this measure been used before?)
Using sound logic (why does this measure capture the variable?)
The validity of manipulated variables can be demonstrated by
Providing precedence (has this manipulation been used before?)
Using sound logic (why does this manipulation capture the variable?)
Manipulation checks
The validity of manipulated variables can be demonstrated by
Providing precedence (has this manipulation been used before?)
Using sound logic (why does this manipulation capture the variable?)
Manipulation checks
Manipulation check
Is a test used to determine the effectiveness of a manipulation in an experimental design. It is used to ensure that the participants understood the manipulation the way the researcher intended
internal validity refers to
The ability to draw valid conclusions about the causal effects of the independent variables on the dependent variable. Internal validity can be threatened in a variety of ways in experiments
Experimenter bias
Experimenter bias occurs when the experimenter (intentionally or unintentionally) affects data, participants or results in an experiment because he is unable to remain objective. Most modern experiments are designed in a way to reduce the possibility of bias-distored results. In general, biases can be kept to a minimum if experimenters are properly trained and clear rules and procedures are put in place for the experiment
Steps can be taken to reduce the likelihood of its occurrence such as
Experimentor bias
Conducting blind studies and minimizing exposure to experiments
In a blind study:
All the information which may influence the outcome of the experiment is withheld from the experimenters and the participants (e.g. when participants are unaware of the hypothesis, they will not be able to influence the outcome of the experiment)
The less exposure respondents have to experimeters
The less likely it is that they will pick up any cues that would impact their answers. One of the common ways to minimize the interaction between participants and experimenters is to prerecord the instructions
History effect:
Events/ factors outside the experiment have an impact on the DV during the experiment
Maturation effect:
Biological/phychological changes over time
Testing effect:
Prior testing affects the DV
Threats to internal validity:
History effect
Maturation effect
Testing effect
Instrumentation effect
Selection bias effect
Mortality effect
Instrumentation effect
The observed effect is due to a change in measurement
selection bias effect
Incorrect selection of respondents (experimental and/or control grouup)
Mortality effect:
Drop out of respondents during experiment
Increasing internal validity: controlling for extraneous variables
Randomization
Design control
Statistical control
Randomization:
Random allocation of participants to different conditions (selection bias, but also instrumentation, history, ortality)
Design control:
Control group
Include group that does not receive the treatment (history and maturation, but also instrumentation, and statistical regression.)
Design control:
Extra groups:
Groups without pre-test but with an experimental manipulation to exclude the effects of pre testing
Statistical control
Measure extraneous variables and include these in the statistical analysis (covariance analytics)
Can findings from the lab experiment be generalized directly to real jobs?
No the types of participants that participate in the lab experiment may be (and typically are) very different from the population of interest. For example, university students might be assigned an artificial task in a lab experiment to study the effect of receiving bonuses on their work performance
To conduct a clean confound-free manipulation (high internal validity) researchars may have to
Conduct their study in an artificial environment, like a university laboratory, which is not representative of the real world
Internal validity of lab experiments is typically
High
External validity of lab experiments is typically
Low
Random sampling
Is the random selection of subjects from a population. It, therefore, pertains to the studys external validity
Random assignemnt
Is about randomly assigning each subject in the sample to the experimental conditions. It therefore pertains to the studys internal validity
