Experimental vs Observational Research

Question 1

What is the problem of causal inference?

Answer 1

The problem of causal inference explains how we have to infer something we do not know (causal effects) from something we do know (data)

Question 2

What are the three requirements to establish causality?

Answer 2

• An association between X and Y
• Rule out confounding variables to explain this association…
• Rule out reverse causality… whereby it turns out rather than X causing Y, Y actually causes X

Question 3

What is a spurious association?

Answer 3

Whereby a correlation is mistaken for an association

Question 4

Explain the research design of a randomised experiment

Answer 4

A randomised experiment involves a researcher intervening in the research process by controlling and randomly assigning values of independent variables to participants. Randomised experiments provide far better proof of causal relationships than natural/field ones do.

Question 5

How do randomised experiments best provide evidence of causal relationships compared to natural/field experiments?

Answer 5

Random assignment of variables in these experiments means that treatment and control groups can be compared - meaning known confounders can be isolated and ruled out. It also rules out reverse causality, as treatments are applied preceding the outcome.

Question 6

What is internal validity? What is external validity?

Answer 6

Internal validity: the degree to which we can be confident that a study identifies the causal effect of the independent on the dependent variable
External validity: the degree to which findings can be generalised to other contexts

Question 7

Why can experiments often have a weakness in terms of external validity despite being internally valid? (Ecological validity, population validity and reactivity)

Answer 7

Ecological validity: Behaviour observed in artificial experimental settings may not generalise to the real world
Population validity: Experiments often involve unrepresentative subject pools (e.g., UG students) and it can therefore be questionable whether experimental findings generalise from the study sample to the population of interest
Reactivity: People may change their behaviour when they know they are being observed

Question 8

What is a laboratory experiment? What concerns are there over these?

Answer 8

Laboratory experiments are those where the researcher has high level of control over what subjects are exposed to. It allows manipulation of broad set of variables and complex measurements.
Concerns include validity of population samples (particularly student ones), reactivity and ecological validity.

Question 9

What are field experiments? What concerns are there often over these?

Answer 9

Field experiments are experiments carried out in natural environments. They have higher ecological validity (as they assess real world behaviour) and have no issues of reactivity. Concerns arise over how researchers have less control over treatment application, making it trickier to isolate variables and relationships.

Question 10

What are survey experiments?

Answer 10

Survey experiments are those conducted in the context of a survey, either online, on phone or in person. Researchers have reduced control of the environment of the experiment, but the diverse populations that can be surveyed means it has higher population validity.

Question 11

What issues are there with conducting experiments?

Answer 11

Experiments can struggle to adequately investigate phenomena. There are also ethical issues around participant involvement and the potential for deception. Experiments may have issues with external validity.

Question 12

What are observational research designs? Why are they used

Answer 12

Observational research designs are those in which the researcher does not have control over the values of the independent variable. The researcher does not intervene.
They are used to look at questions regarding distributions, to look at meanings of concepts and, crucially, also to look at EXPLANATION.

Question 13

What issues can there be with observational research designs?

Answer 13

These designs where researchers do not intervene struggle to establish causality, especially as confounders cannot be fully ruled out.

Question 14

What are natural research designs? How can they often be treated ‘as if’ random?

Answer 14

Natural research designs involve the assignment of independent variables through naturally occurring events, those that do not involve researcher involvement. Thus they occur ‘as if’ random.

Question 15

What advantages are there of natural research designs?

Answer 15

They hold high internal validity AND external validity

Question 16

Explain cross-sectional studies. What scale of unit do they look at? Give examples of these

Answer 16

Cross-sectional studies examine a cross-section of social reality, focusing on variation between individual units, such as citizens, countries, etc. Typical example is an election study. Or a snapshot of democracy levels across the world at a given point in time. An explanatory study would leverage cross-sectional variation in the independent variable between units.

Question 17

What are time series studies? How do they differ from cross-sectional studies? Give an example.

Answer 17

Time-series studies examine evolutions of a single unit over time. A typical example would be studies looking at trends in economic performance in a single country.

Question 18

What are repeated cross-section studies? How do these differ from time-series studies? Give an example.

Answer 18

Repeated cross-sections are cross-sectional studies which are repeated at different points in time. For example, election studies are repeated after every election. Note that most likely the election study will involve different people; however, at a higher level of aggregation (the country as a whole), the study is repeated.

Question 19

What are time-series cross-sectional studies? Do they use the same units over time?

Answer 19

TSCS studies are basically like panels, but when it is countries or regions within a country that are observed multiple points over time, some people use the term “time-series cross-sectional

Question 20

What are panel studies? Do they use the same units over time?

Answer 20

Panels are very similar, but the same people are observed over time. For example, there are surveys which interview the same people multiple times.

Question 21

What is informed consent in research? How can this be achieved?

Answer 21

Informed consent is the situation whereby participants know they are involved in a study. They must be informed of how they were selected, what procedures will occur and the study’s purpose.

Question 22

What two groups are part of experimental designs? What purposes are there of each group?

Answer 22

Experimental designs are made up of an experimental group that is exposed to an intervention, and a control group that is not. The idea is that the control group will either confirm or deny the presence of a causal relationship, based on whether it experiences a change or not when it is not exposed to a variable.

Question 23

What is reactivity?

Answer 23

Reactivity is where participants alter behaviour due to being aware that they are part of a study.