Session 5: Measurement and Large-N

Question 1

Define “Measurement” in the context of Research Design.

Answer 1

Measurement is the evaluation of cases with respect to variables. It includes:

Measurement in the true sense: This refers to the assigning of numbers to the variables, often seen in quantitative research.
Classing: This refers to the assigning of categories to the variables, often seen in qualitative research.
These concepts are relevant to both qualitative and quantitative research, regardless of the scale (1- , small-, large-n). Note that the terms used to describe these actions may vary.

Question 2

Which of the following could be a case in social scientific research? A) a candidate in the Dutch election B) the Dutch selection C) The Netherland D) A voter in the Dutch election E) The Dutch electoral system

Answer 2

Any could be a case

Question 3

Which of the following is not a variable? 1) Government capacity 2) Number of new COVID-19 cases per day 3) Presence/absence of foot-ball related violence 4) The VVD 5) Attitudes towards group work in education 6) They are all variables

Answer 3

4) The VVD. A political party is not a variable. The political party someone voted for could be variable (it can vary and be measurable/classifiable)

Question 4

What are the criteria that must be met for something to be considered a variable in social scientific research?

Answer 4

• Be capable of assuming two or more values. In other words, it needs to be able to vary.
• Be measurable or classifiable. It must be possible to assign values (numbers or categories) to the variable.
• Be relevant and meaningful to the research question at hand. It should have the potential to influence or be influenced by other variables in the study.

Question 5

You are asked to indicate how often you complete the readings before each RD session by selecting from ‘always’ ‘usually’ ‘sometimes’ and ‘never’ what is this variable’s level one measurement? 1) binary 2) nominal/categorical 3) ordinal 4) interval 5) ratio

Answer 5

Ordinal

Question 6

What are the four levels of measurement in research and what characterizes each one?

Answer 6

1. Nominal Level: Categories or labels are used that do not have any order or priority.
2. Ordinal Level: Categories can be ordered or ranked but the intervals between them are not equal.
3. Interval Level: Categories can be ordered and the intervals between them are equal, but there’s no absolute zero point.
4. Ratio Level: Categories can be ordered, intervals between them are equal, and there’s an absolute zero point.

Trick to Remember: Use the acronym “NOIR” (French for ‘black’):

• N for Nominal
• O for Ordinal
• I for Interval
• R for Ratio

These levels help determine the appropriate statistical analysis to be used and aid in the interpretation of the results.

Question 7

What are the four levels of measurement in research and what characterizes each one?

Answer 7

1. Nominal: “Names” - Categories or labels are used that do not have any order or priority
2. Ordinal: “Ordered” - Categories can be ordered or ranked but the intervals between them are not equal
3. Interval: “Intervals Even” - Categories can be ordered and the intervals between them are equal, but there’s no absolute zero point
4. Ratio: “Zero True” - Categories can be ordered, intervals between them are equal, and there’s an absolute zero point

Mnemonic: “New Opera In Zurich”. NOIZ

Question 8

Define “Level of Observation” and “Level of Analysis” in research.

Answer 8

Level of Observation: The scale at which the data is collected or observed. For example, in a study of mental health in high schools, the individual students would be the level of observation if you are surveying each student individually.

Level of Analysis: The scale at which the data is analyzed or interpreted. Using the same example, if you are looking at the overall mental health trend of the entire school, then the school is your level of analysis. = things we compare in the analysis

Question 9

Define a “Case” in the context of research and state its requirements.

Answer 9

A case is a spatially and temporally bounded object, phenomenon, or event in the world. What counts as a case can vary depending on the project. Case of a country/historical process resulted in development in XXX/

Why do people vote the way they do? select individuals as case or perhaps countries, depends on the research question

For something to be considered a case, it must be:

Bounded: It needs to have clear, defined limits.
Homogenous: It should consist of similar elements or characteristics.
Note that casing is not always done well due to challenges in drawing clear and unambiguous boundaries.

Question 10

Define “Case”, “Sample”, and “Population” in research, and explain their relationship.

Answer 10

A “Case” is a case because it’s an instance of something, belonging to a “Population”. A Population is the entire set of cases that could potentially be studied.

A “Sample” is a subset of the population. However, it may only be representative of the sampling frame from which it’s drawn, not necessarily the entire population.

The process of selecting cases from the population is called “Sampling” in large-n research, or “Case selection” in 1-, small-n research.

Example: If you’re studying the reading habits of high school students in California, a “Case” could be a single high school student. The “Population” would be all high school students in California. If you survey 1000 students from various schools across the state, those 1000 students are your “Sample”.

Question 11

Define “Variable” in the context of research and state its characteristics.

Answer 11

A “Variable” in research is an operationalized dimension of a concept and an attribute of a case. Each case in a study is assigned a specific value for a variable.

Importantly, as the name implies, variables must vary. They need to have the ability to change or take on different values among different cases in the study.

For example, in a study looking at the effect of study hours on exam performance among students, “study hours” and “exam performance” would be variables. They can take on different values for different students (cases), hence they vary.

Question 12

Reliability in Research Design

Answer 12

Refers to the consistency of a measure. A test is considered reliable if we get the same result consistently. For example, if a person were to take the same personality test several times and get the same results each time, that test would be considered reliable.

Question 13

Precision in Research Design

Answer 13

Precision refers to the closeness of two or more measurements to each other. It indicates the exactness of a measurement, reflecting the level of detail. However, false precision can occur when results are reported to be more precise than the data or method allows.

Question 14

Relative Absence of Measurement Error

Answer 14

A measurement with a relative absence of error indicates a high level of accuracy. This means the measurements closely align with the true value. The more accurate a measurement is, the less measurement error it has.

Question 15

Random Measurement Error

Answer 15

Random error, as the name suggests, is random in nature and very difficult to predict. It varies in an unpredictable way, and is present in all measurements. It influences measurements inconsistently and is caused by factors that are beyond the control of the researchers.

Question 16

Systematic Measurement Error

Answer 16

Systematic error, or bias, consistently affects measurements in the same direction. This type of error is introduced by a problem that persists throughout an entire experiment. The error can be identified and minimized through careful experiment design and execution.

Question 17

Front of the flashcard:

1. Define a “true experiment” in the context of social science research.
2. Explain why a true experiment is so useful for facilitating causal inferences.
3. Discuss the reasons why not all research is based on experiments.

Back of the flashcard:

Answer 17

1. A true experiment is a type of research design in which the researcher has control over and manipulates the independent variable(s), randomly assigns participants to conditions or orders of conditions, and measures the effect that the manipulation has on a dependent variable(s).
2. True experiments are beneficial for facilitating causal inferences because they allow researchers to isolate variables, control for confounding factors, and observe whether changes in the independent variable(s) directly cause changes in the dependent variable(s). This control and manipulation provide a strong basis for inferring a cause-effect relationship.
3. Not all research is based on experiments due to various reasons. These include ethical and practical constraints, the nature of the research question, the need for exploratory or descriptive research, and the need to study phenomena in a natural, real-world context (observational studies, qualitative research, etc.). Furthermore, some research questions may be better addressed using correlational designs, surveys, or other non-experimental methods.

Question 18

Front of the flashcard:

1. Describe what is meant by “Large-N Designs” in social science research.
2. Explain when and why Large-N Designs might be used.
3. Discuss the data characteristics in Large-N Designs.
4. Describe the key question to consider when using Large-N Designs.

Back of the flashcard:

1. “Large-N Designs” refer to observational research designs where a large number of observations (N) are available. Here, ‘N’ stands for the number of cases or units in a study.
2. Large-N Designs are often used when random assignment and experimental manipulation are unavailable or not feasible. In these scenarios, researchers can rely on observing relationships among variables as they naturally occur “in the real world.”
3. In Large-N Designs, the data typically involves quantitative measurements on many cases, but relatively few aspects (variables) of each case are examined. This allows for a broad, albeit shallow, analysis across a large sample.
4. The key question when using Large-N Designs is: “Do the patterns of association between variables in our observations fit theory?” This essentially means examining if the observed relationships or patterns align with theoretical expectations or predictions.

Question 18

1. Describe what is meant by “Large-N Designs” in social science research.
2. Explain when and why Large-N Designs might be used.
3. Discuss the data characteristics in Large-N Designs.
4. Describe the key question to consider when using Large-N Designs.

Answer 18

1. “Large-N Designs” refer to observational research designs where a large number of observations (N) are available. Here, ‘N’ stands for the number of cases or units in a study.
2. Large-N Designs are often used when random assignment and experimental manipulation are unavailable or not feasible. In these scenarios, researchers can rely on observing relationships among variables as they naturally occur “in the real world.”
3. In Large-N Designs, the data typically involves quantitative measurements on many cases, but relatively few aspects (variables) of each case are examined. This allows for a broad, albeit shallow, analysis across a large sample.
4. The key question when using Large-N Designs is: “Do the patterns of association between variables in our observations fit theory?” This essentially means examining if the observed relationships or patterns align with theoretical expectations or predictions.

Question 19

1. Discuss the advantages of Large-N Designs compared to smaller-N designs.
2. Explain the significance of having many observations in Large-N Designs.
3. Describe the caution that should be taken when using Large-N Designs.

Answer 19

1. Compared to smaller-N designs, Large-N designs have the advantage of being able to identify and estimate weak and heterogeneous relationships. They can detect subtle effects or variations that may be undetectable in smaller samples due to lack of statistical power.
2. Having many observations in Large-N Designs helps to harness the power of large numbers to detect a systematic “signal” from the “noisy” data the world provides. In other words, with a larger sample size, it’s easier to distinguish between patterns that occur due to chance (noise) and those that indicate a meaningful effect or relationship (signal).
3. Caution must be exercised when using Large-N Designs as statistical analysis alone is insufficient for causal inference. Despite the ability of Large-N Designs to detect patterns, they do not automatically provide evidence of cause and effect. Any observed associations need to be interpreted within an appropriate research design, considering potential confounding variables and the context of the study.

Question 20

Explain why association is not the same as causation.

Answer 20

Association is not the same as causation because the mere co-occurrence or correlation of two variables does not mean that one variable causes the other to occur. While causation implies a cause-effect relationship, association merely reflects a relationship where the variables change together.

Question 21

Discuss the problem that arises when attempting to infer causation from observed associations.

Answer 21

The problem that arises when attempting to infer causation from observed associations is that the same association could arise due to several different causal mechanisms, or even by chance. The challenge is to accurately identify which (if any) of these mechanisms is at work.

Question 22

List and explain the five different reasons why an association between two variables might occur.

Answer 22

An association between two variables can arise for five very different reasons:

(1) Chance: The observed association might be a statistical fluke, due to random variation in the data.
(2) X causes Y: This is a direct causal relationship from variable X to variable Y.
(3) Y causes X: This is a direct causal relationship from variable Y to variable X, the opposite direction of the previous case.
(4) Z (a confounder) causes X and Y: Here, a third variable Z is causing both X and Y, creating an apparent association between X and Y.
(5) We condition on a shared effect of X and Y: This refers to a situation where X and Y are independent, but appear associated because we are considering them in a subgroup defined by a shared effect. This is known as a “conditioning on a collider” or a “selection bias.”

Question 23

Explain what is meant by a “heterogeneous causal effect.”

Answer 23

A “heterogeneous causal effect” refers to a situation where the effect of a cause varies across different subpopulations or under different conditions. This means that the causal relationship observed in one context or group may not apply universally to all contexts or groups.

Question 24

Discuss the pitfalls when a subpopulation is observed in which there is no causal effect.

Answer 24

One pitfall in causal inference occurs when a subpopulation is observed in which there is no causal effect. If this non-representative group is used to infer causal relationships for a larger population, it may lead to incorrect conclusions because the observed association (or lack thereof) in this group may not be reflective of the true causal relationship in the broader population.

Question 25

Explain how a confounder can conceal the association between two variables.

Answer 25

A confounder can conceal the association between two variables. A confounder is a variable that is associated with both the independent and dependent variables and can introduce bias if not properly controlled for. This bias can lead to spurious associations or mask true associations, leading to incorrect causal inferences.

Question 26

Give an example of how a confounder can influence the relationship between two variables.

Answer 26

An example of this is if we’re trying to study the relationship between motivation and study success. However, both motivation and study success could be influenced by intelligence - a potential confounder. If more intelligent individuals are less motivated but still achieve study success due to their intelligence, it could potentially conceal or distort the positive association between motivation and study success.

Question 27

Discuss the role of statistics in Large-N designs.

Answer 27

In Large-N designs, statistics play a crucial role in analyzing and interpreting data. When used appropriately, they provide a rigorous method for drawing inferences from the data.

Question 28

Explain the benefits of Large-N designs when estimating probabilities and effect sizes.

Answer 28

Large-N designs are beneficial for estimating probabilities and effect sizes due to their larger sample size. The large number of observations allows for more robust probability estimates for an effect resulting from chance alone. Additionally, they provide more precise estimation of effect sizes, even in the presence of “noise” or random variations in the data.

Question 29

Discuss the role of research design in Large-N designs.

Answer 29

The research design is vital in Large-N studies. While the large sample size provides statistical advantages, an appropriate research design is necessary to correctly interpret the associations observed.

Question 30

Describe how an appropriate research design can address issues such as reversed causality, confounders, and collider bias in Large-N designs.

Answer 30

An appropriate research design in a Large-N study can help rule out alternative explanations for observed associations, such as reversed causality (where Y causes X rather than X causing Y), confounders (variables that are associated with both the independent and dependent variables and could bias the observed relationship), and collider bias (bias introduced by conditioning on a variable that is an effect of both the independent and dependent variables). By carefully designing the study and considering these potential pitfalls, researchers can make more accurate causal inferences from observed associations.

Question 31

Explain the conditions under which observations in Large-N designs support causal inference.

Answer 31

Observations in Large-N designs can support causal inference if certain conditions are met. These typically involve having a representative sample, correctly identifying and controlling for confounders, and ensuring that the observed associations are unlikely to be due to chance alone.

Question 32

Discuss the role of assumptions in Large-N designs.

Answer 32

However, not all conditions can be tested empirically. In many cases, researchers must assume that these conditions are met. These assumptions might pertain to the independence and identical distribution of observations, absence of unmeasured confounders, and the linearity and additivity of relationships, among others.