Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Policy Analysis > Policy Analysis Exam > Flashcards

Policy Analysis Exam Flashcards

1
Q

What do the following acronyms stand for: T, C, RCT, PE, PIE, and SRS?

A

T = treatment or treatment group
C = control group
RCT = randomised controlled trials
PE = policy evaluation
PIE = policy impact evaluation
SRS = simple random sampling

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What are the two different categories of evaluation?

A
  1. Operational evaluations: examines how effectively programs were implemented and whether there are gaps between planned and realized outcomes.
  2. Impact evaluations: studies whether the changes in well-being are due to the program intervention and not to other factors.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Define Policy Impact Evaluation (PIE).

A
  • An assessment of the causal effect of a specific intervention on some measurable outcomes in a target population.
  • This causal effect is called treatment / intervention impact.
  • This also includes assessing if a policy reaches the goals for which it was implemented
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What questions should we ask when determining the difference between intervention design and evaluation design?

A
  • Who is more in need?
  • Who will be more reactive to the intervention?
  • Who is less at risk of adverse outcomes?
  • There are also political constraints.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is the main tradeoff in PIE?

A

Effectiveness and generalisability.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the issue of heterogeneity?

A
  • That the effects of a program and inputs affecting outcomes may vary over its expected lifetime.
  • Thus, monitoring long-term as well as short-term outcomes may be of interest to policymakers.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What are the four criteria for effective intervention designs?

A
  1. Efficacy: establish a detailed, plausible chain of causal mechanisms.
  2. Compliance: promote real participation to T.
  3. Cost-effectiveness.
  4. Standardisation: a reasonable degree of uniformity.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are some examples of social issues?

A
  • Community support programs improve the health outcomes of babies.
  • Reducing the size of classrooms in primary schools benefits disadvantaged students.
  • Urban desegregation improves the occupational opportunities of ethnic minorities.
  • Smoking marijuana enhances the risk of cancer.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What is causality?

A

If x (cause, treatment, intervention) is changed, there will be a change in y (effect, impact, outcome).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What are four reasons for which PIE are important?

A
  • Informs evidence of outcomes.
  • Cost-effectiveness.
  • Accountability.
  • Job opportunities.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What does ex-ante PIE predict?

A

The policies impacts and its viability using data before the program intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What does ex-post PIE predict?

A

It predicts outcomes after programs have been implemented.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What are reflexive comparisons?

A

A type of ex-post evaluation; they examine program impacts through the difference in participant outcomes before and after program implementation (or across participants and nonparticipants).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q
A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What is the main challenge across different types of policy evaluation?

A

To find a good counterfactual—namely, the situation a participating subject would have experienced had he or she not been exposed to the program.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

When were policy evaluations invented, and who uses them?

A

1980s. Policymakers, practitioners, private foundations, NGOs, associations, beneficiaries, proponents and skeptics. Mostly used in Anglo-Saxon countries.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

What is the minimum amount of time to complete a policy evaluation?

A

Six months

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

What is the first step of every evaluation?

A

To determine who are the key stakeholders, which may include individuals who have:
- Access to the field,
- Data, context and information,
- Funding and support,
- Interferences.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Why must the policy goal be specified clearly?

A
  • For it to be measurable.
  • The policymaker will state the outcome in a very generic and vague way – to the researcher that clearly identifies how to measure the efficacy of the policy.
  • It is key, for a researcher, to establish the specific outcomes so that the policymakers (who come up with the idea) cannot argue against the results once released.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Are ex-ante or ex-post evaluations preferable?

A

Ex-ante (may be referred to as simulations)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

What is input?

A

Set of resources required for a policy, including economic resources (data), human resources (IT employees to create an app).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What is output?

A

The set of products/results/deliverables of the policy that comes from the actions of the policy. For example, how many people have downloaded the app?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

What are outcomes?

A

Outcomes refers to the causal impact and the net effect of the policy – how it changed the course of events of individuals and groups.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

What is the main difference between output and outcomes?

A

The causal effect and the fact that outputs focus on the results produced during the program, and not after.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

What is the main tension between policymakers and researchers?

A

Time. There is a tension between policymakers that want to know the results as soon as possible, and the fact that researchers are interested in long-term outcomes and not short-term outcomes.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

When the EU assesses policies, what do they evaluate?

A
  1. Budget
  2. Legal
  3. Deliverables
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

What is the mixed methods approach?

A

It mixes quantitative and qualitative methods such as PIE and the process assessment.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

What is the theory of change?

A

An explanaton of the process of change by outlining causal linkages in an initiative. It is best to start by the long-term outcomes and work backwards to treatment implementation, identifying hurdles.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
29
Q

Define nudging and provide an example.

A

An attempt at influencing people’s behaviour in a predictable way by using their biases and habits.

For example, sending people a reminder to book a doctor’s appointment, or placing healthy foods at eye level at supermarkets.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
30
Q

What is Kahneman’s ‘Dual Process Theory’?

A

At times, humans act irrationally and not in their interests because they have two systems of thought:
- System 1 (intuition): fast, unconscious, automatic, everyday decisions, error prone.
- System 2 (reasoning): slow, conscious, effortful, complex decisions, reliable.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
31
Q

What does the EAST Framework intend to do?

A

To encourage behaviour (nudges) to make it easy, attractive, social and timely.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
32
Q

What are the components of the EAST Framework and their biases?

A
  • Easy: reducing effort. Biases: endowment effect, status quo effect, cognitive overload.
  • Attractive: presenting benefits.Biases: availability bias, anchoring effect, loss aversion, optimism bias, scarcity bias.
  • Social: harnessing social pressure. Biases: confirmation bias, harding, commitment bias, authority bias.
  • Timely: prompting when they are most receptive. Biases: present bias, hyperbolic discounting, duration neglect, hot/cold states.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
33
Q

What is the endowment effect?

A

How individuals place more value on items they own, than the same items they don’t own.

Example: shares inherited from deceased relative.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
34
Q

What is the status quo effect?

A

How individuals prefer to maintain their current situation, and oppose actions that may change it.

Example: gender norms.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
35
Q

What is cognitive overload?

A

How individuals are not capable of processing more than a certain amount of information at a time.

Example: overloaded nurse.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
36
Q

What is the availability bias?

A

How individuals rely on information that comes readily to mind.

Example: plane crashes in contrast with car crashes.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
37
Q

What is the anchoring effect?

A

How individuals tend to rely and base their decisions on the first piece of information offered.

Example: products at discount.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
38
Q

What is loss aversion?

A

How individuals experience losses more intensely than equivalent gains.

Example: loss or gain of $100.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
39
Q

What is optimism bias?

A

How individuals overestimate the likelihood of positive events and underestimate the likelihood of negative events.

Example: refusal to have job security fund.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
40
Q

What is scarcity bias?

A

How the more difficult it is to acquire an item, the more value is placed upon it.

For example: the sale of luxury goods.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
41
Q

What is confirmation bias?

A

How individuals prefer information that confirms their current beliefs.

Example: certain news outlets.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
42
Q

What is herd mentality?

A

How individuals can be influenced by the majority.

Example: panic reactions in crowds.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
43
Q

What is commitment bias?

A

How individuals tend to stay committed to their past behaviours, particularly those exhibited in public.

Example: public commitment to exercise.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
44
Q

What is authority bias?

A

How individuals believe opinions of authority figures over others.

Example: doctors and dentists on toothpaste advertisements.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
45
Q

What is present bias?

A

How individuals give stronger weight to payoffs that are closer to present time.

Example: preferring $10 today, to $20 tomorrow.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
46
Q

What is hyperbolic discounting?

A

How individuals value immediate though smaller rewards than long-term, larger rewards.

Example: free shipping with bundle purchases.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
47
Q

What is duration neglect?

A

How individuals’ judgement of the unpleasantness of painful experiences depends very little on the duration of them.

Example: mothers giving birth.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
48
Q

What is the hot/cold empathy gap?

A

How individuals underestimate the influences of visceral drives on their own attitudes, preferences and behaviours, such as hunger.

Example: (cold) physicians medicating (hot) patients.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
49
Q

How can a researcher design an effective treatment group ex-ante?

A

While difficult:
* Use empirical evidence of previous PIE and PE,
* Have deep qualitative knowledge of the context,
* Conduct pilot studies, and
* Use implementation analysis.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
50
Q

List the four types of correlational approaches.

A
  • Ecological and comparative analyses
  • Time comparisons: pre-post
  • Matched-case control studies
  • Difference-in-differences
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
51
Q

List the three types of causal approaches (PIE).

A
  • RCT
  • Regression discontinuity
  • Instrumental variables
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
52
Q

What are the steps required to determine the outcomes? (provided visual)

A
  1. Policy issue, policy goal, outcomes
  2. Or - input, intervention, output, outcomes
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
53
Q

What are the steps/types of implementation in policy evaluation?

A
  • Budget analysis
  • Legal audits
  • Reception of the beneficiaries
  • Feedbacks from implementers
  • Analysis of the deliverables
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
54
Q

Define treatment impact, or intervention effect?

A

Changes in outcomes that are attributable to the treatment.

The difference between the outcome some time after the intervention has been implemented, and the outcome had the intervention not been implemented.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
55
Q

Define counterfactual.

A

The state of the world that participants would have experienced in the absence of the intervention.

That is, the outcome had the intervention not been implemented.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
56
Q

Define average treatment (intervention) effect.

A

A measure used to compare treatments in randomised experiments, evaluation of policy interventions, and medical trials.

It measures the difference in mean outcomes between units assigned to the treatment and units assigned to the control.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
57
Q

What is the fundamental problem of causal inference?

A

Intervention and the counterfactual cannot be observed simultaneously.

Individual-level causal inference is impossible.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
58
Q

How is a counterfactual constructed?

A

By making a control group, which involves selecting a group of individuals who did not participate in the intervention.

Goal: to reliably attribute any difference in outcome between the treated and control group after the treatment. RCTs are good for this.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
59
Q

How do correlational approaches work?

A

Correlational approaches to PIE use comparisons over space/time and related variations of policies and outcomes to approximate the counterfactual in the absence of random assignment.

The issue is it has a large number of confounding factors, and the comparison between the groups are not really comparable.

Thus, it is difficult to isolate the specific causal impact of the intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
60
Q

What are pre-post correlational approaches?

A

An assessment conducted at the start and at the end of a term, in order to observe changes caused by an intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
61
Q

Define the diff-in-diff approach.

A

Differences-in-differences: calculate the effect of a treatment (independent variable) on an outcome (dependent variable) by comparing the average change over time in the outcome variable for the treatment group to the average change over time for the control group.

For example, testing minimum wage in Paris, but not in Marseille, and observing the differences between outcomes.

It tries to mitigate the effects of extraneous factors and selection bias. However, it is still subject to biases such as mean regression, reverse causality, and omitted variable bias.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
62
Q

What are the levels of diff-in-diff approaches?

A
  • Post-intervention assessment,
  • Pre & post intervention assessment,
  • DFD for time trend,
  • DFD with follow-up and fade out or maturation,
  • DFD with follow-up and concomitant factors,
  • Matched-case control study with DFD and follow-up,
  • Matched-case control study.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
63
Q

How does the ecological / comparative approach work?

A

It considers the context of public administration, many different disciplines, and takes a comparative approach across regions and governments.

For example, when studying the impact of urban segregation on employment outcomes of minority groups, we must consider ethnicity, SES, cultural attitudes, sector composition, industrialisation, globalization, GDP per capita, current policies (employment, educational, welfare).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
64
Q

What are the advantages and disadvantages of the ecological / comparative approach?

A

Advantages: holistic, context, policy

Disadvantages: complex, subjectivity, generalisability, bad data

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
65
Q

When are causal inferences impossible?

A

When there is only distributional information about the data and no information about how the data was generated.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
66
Q

What is the purpose of PIE?

A

Learn lessons and have accountability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
67
Q

Define independent and dependent variables - and how they interact?

A

Independent variable are manipulated to affect the dependent variable.

Independent variable: the cause of a change.

Dependent variable: the observation of or the change itself.

Example of variables: gender, nationality, age, income, having a Velib pass, receiving the culture pass.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
68
Q

Define frequency distribution.

A

A representation that displays the number of observations within a given interval; it shows the frequency of each variable.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
69
Q

Define absolute frequency.

A

A count of the number of times something has occured.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
70
Q

Define relative frequency.

A

The number of times something has occurred divided by its frequency.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
71
Q

Define cumulative frequency.

A

The sum of relative frequencies.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
72
Q

Define bivariate analysis.

A

A statistical method examining how two different things are related.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
73
Q

What is the Pearson correlation coefficient?

A

A number between 1 and -1 that indicates the strength and direction of a relationship between two variables.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
74
Q

What is a confounding variable?

A

A third variable that influences both the independent and dependent variable, causing a spurious association.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
75
Q

What are a spurious relationships?

A

False statistical relationships.

76
Q

What is selection bias?

A

How participants in a study differ systematically from the population of interest, leading to a systematic error in association or outcome.

Example: health studies that recruit participants directly from clinics miss all the cases who don’t attend those clinics or seek care during the study.

77
Q

What are matched-case control studies?

A

In individually matched studies, the population of interest is identified and cases are randomly sampled or selected based on particular inclusion criteria.

78
Q

What are the advantages and disadvantages of matched-case control studies?

A

Advantage: it removes the original confounding effect by matching factors, such as age or sex.

Disadvantages: if the matching factor is associated with the exposure of interest, it introduces a new bias because the matching factor itself is confounded.

Another is overmatching, which happens when cases and controls are matched on variables that have a high relevance to the exposure. For example, in a study of smoking and lung cancer, matching on the variable “carrying a cigarette lighter.”

79
Q

In RCTs, what is the relationship between the two groups?

A

In RCTs, the two groups are perfectly equivalent because they are drawn from the same population of people who accepted to participate in the study.

80
Q

Do RCTs have external validity?

A

No, they cannot be generalised to individuals who refused to participate. However, they do have internal validity.

81
Q

Define randomisation.

A

If two participants are randomly extracted from the same population, on average they will display the same values on all parameters, or be equivalent.

82
Q

RCTs = (3 factors)

A

RCTs = randomisation + manipulation + control

83
Q

How is randomisation sold?

A

By:
* Showing the delayed/compensatory treatment;
* Explaining why comparability is important;
* Explaining that there are not enough resources to treat everyone;
* Using ‘action research’ instead of experiments.

84
Q

Always use random assignment if it is ethical. If it is not…?

A
  • It can be a temporary exclusion,
  • It can be compensated for,
  • Consider everyone has the same chance of getting it,
  • Consider it benefits greater society.
85
Q

With policy evaluations, are there any other ethical duties to consider?

A

There is an ethical duty to consider whether a policy evaluation will have negative outcomes, which is why there is informed and revocable consent, and an institutional review board.

86
Q

What are the five types of randomisation structure?

A
  1. Nested structure
  2. Individual-level randomisation
  3. Individual-level randomisation with blocking: divide individuals into blocks and randomise within them
    - Pros: higher statistical power
    - Cons: higher risk of contamination than non-blocked designs
  4. Cluster randomisation (can be between schools or classes): assigning groups to randomisation instead of individuals
    - Pros: more practical and cost-effective, less risk of contamination
    - Cons: less statistical power, higher N and data collection costs
  5. Cluster randomisation with blocking
87
Q

Define factorial design.

A

An experiment that has multiple factors or independent variables.

88
Q

Define placebos.

A

A treatment specifically designed to have no effect.

89
Q

What methods can a researcher use to analyse experimental data?

A
  • Ex-ante equivalence analysis
  • Simple comparison of group means or relative frequencies of the outcome variables
  • More advanced techniques for treatment compliance
90
Q

What is the best possible design to assess impacts ex-ante?

A

PIE via RCTs

91
Q

To assess the total quality of a policy evaluation, what can a researcher measure?

A

Internal and external validity – which must be valid causal inferences, generalisable to the relevant population that can benefit from the policy.

92
Q

What are quasi-experimental methods of PIE?

A

It is the same as a true experiment, except it does not rely on random sampling and requires that subjects be assigned to groups based on non-random criteria.

93
Q

Define Randomised Control Trials (RCTs)?

A

A trial in which subjects are randomly assigned to one of two groups – the experimental group receiving the intervention that is being tested, and the control group receiving an alternative treatment.

Subjects must be willing to participate in the intervention, and then halved into these groups.

94
Q

What is internal validity?

A

The degree of confidence that the causal relationship is not influenced by other factors and variables.

95
Q

When are quasi-experimental designs useful?

A
  • When experiments are unethical or logistically difficult
  • When randomisation is not accepted
  • When the policy has been implemented
96
Q

Who assigns treatment and control groups in quasi-experimental designs?

A

A third party

97
Q

What are the four types of quasi-experimental designs?

A
  • Regression discontinuity designs (RDD)
  • Instrumental variables (IV)
  • Difference-in-differences (DiD)
  • Matching methods / propensity scores
98
Q

What are some examples of regression discontinuity designs (RDDs)?

A
  • Effect of training program for youth unemployment – assignment based on age.
  • Effect of conditional cash transfers on contraceptive behaviour – assignment based on income.
  • Effect of broadening access to the healthcare system on low-income children’s health – assignment based on income.
99
Q

What is the intuition behind regression discontinuity designs (RDDs)?

A
  1. Policies often use thresholds to determine eligibility
  2. These thresholds are arbitrary: subjects located around them are almost identical
  3. With RDD, those just above and below the threshold can be compared
100
Q

What is the difference between randomised control trials (RCTs) and regression discontinuity designs (RDDs)?

A

RCT: researcher randomly assigns control and treatment group.
RDD: researcher randomly assigns based on a cut-off value.

101
Q

What are the three limitations of regression discontinuity designs (RDDs)?

A
  1. External validity
  2. Must be many observations around the cutoff point
  3. If the same cutoff is used for other programs, treatment effects are confounded
102
Q

Why are instrumental variables used?

A

They are used to provide true effects, rather than biased effects.

103
Q

What are some examples of instrumental variables in use?

A
  1. Effect of contraceptive knowledge on number of babies – Z = mass media exposure
  2. Effect of housing vouchers and educational attainment – Z = random assignment
104
Q

Explain what exclusion and relevance variables must do.

A

Exclusion: it must have no direct effect on the variable of interest.
Relevance: it must have a significant correlation with the instrumented variable.

105
Q

What are the limitations of instrumental variables?

A
  1. The estimates of treatment impacts have high uncertainty when the relationship between variables is weak. (Test: F-test)
  2. The exclusion variable is not respected and must be controlled for.
106
Q

Identify and define the three families of causal effects?

A
  1. Average treatment effect (ATE): the overall effect of the policy on the whole sample or population.
  2. Conditional treatment effect (CTE): the overall effect of the policy on a specific subgroup.
  3. Local average treatment effect (LATE): the effect of the policy on compliers in the whole sample or population.
107
Q

Why sample?

A

Saves money and time - hence improves the quality of evaluation.

108
Q

What are the five types of probability sampling?

A
  1. Simple random
  2. Systematic random
  3. Area
  4. Cluster
  5. Stratified
109
Q

What are the five types of non-probability sampling?

A
  1. Voluntary
  2. Convenience
  3. Purposive
  4. Snowball
  5. Quota
110
Q

What is the objective of sampling?

A

The objective of sampling is to approximate the characteristics that are relevant to the question about a larger population.

111
Q

Identify and define the four limitations of sampling.

A
  1. Population-specific error: when researchers do not understand who they should survey.
  2. Selection error: when respondents self-select their participation in the study (only those that are interested respond).
  3. Sample frame error: when the wrong sub-population is used to select a sample.
  4. Non-response error: when potential respondents are not successfully contacted or refuse to respond.
112
Q

Define confidence interval.

A

An indication of how precise the sample is and the degree of uncertainty of an estimate.

It is used to quantify sampling uncertainty.

113
Q

What are the steps to sampling?

A
  1. Define the target population
  2. Find a reliable list of units of this population
  3. Choose the most appropriate sampling method
  4. Determine sample size
  5. Implement sampling plan
114
Q

Define statistical significance.

A

A measure of how ‘rare’ the results are, under the assumption that the null hypothesis is true.

115
Q

Define probability sampling and provide an example.

A

Probability sampling: the randomised selection of a sample from a population.

For example, after a training course for the unemployed, a short questionnaire on customer satisfaction is left and participants may or may not fill it in.

116
Q

Define non-probability sampling and provide an example.

A

Non-probability sampling: the subjective selection of a sample from a population.

For example, interviewing every house on a given street.

117
Q

Define simple random sampling (SRS) and explain the procedure.

A

Simple random sampling: a type of probability sampling in which the researcher randomly selects a subset of participants from a population.

Procedure:
1. Assign a number to each unit in the list
2. Draw randomly N numbers
3. Select the units corresponding to these numbers

118
Q

Define systematic sampling and explain the procedure.

A

Systematic sampling: a type of probability sampling where researchers select members of the population at a regular interval; for example, alphabetical.

Procedure:
1. Assign a number to each unit in the list
2. Determine k = population size / sample size
3. Draw randomly one number, select it, then select every kth element

119
Q

Define stratified sampling and explain the procedure.

A

Stratified sampling: a type of probability sampling where researchers divide subjects into subgroups called strata based on characteristics they share; for example, race or gender.

Procedure:
1. Identify one or more relevant predictors of the outcome.
2. Create sublists of the population based on the categories of this predictor. For example, men and women.
3. For every stratum, draw a number of units corresponding to its share in the population (proportional stratification).
4. With these sublists, select the units using SRS or SS.

120
Q

Define quota sampling and explain the procedure.

A

Quota sampling: a non-probability sampling method that relies on the non-random selection of a predetermined number or proportional of units.

Procedure:
1. Identify one or more relevant predictors of the outcome.
2. Create sublists of the population based on the categories of this predictor. For example, men and women.
3. For every stratum, draw a number of units corresponding to its share in the population (proportional stratification).
4. Select the units freely.

121
Q

What are the drawbacks of quota sampling?

A
  1. It is not random.
  2. It is exposed to selective bias. For example, interviewers will interview easy interviewees more often.
122
Q

Define clustered sampling and explain the procedure.

A

Clustered sampling: a probability sampling method in which researchers divide a population into clusters and then randomly select some of these clusters as the sample; for example, districts or schools.

Procedure:
1. Divide the population into groups of homogeneous units (clusters), usually based on geographical proximity.
2. Draw a random sample of these clusters, using SRS or systematic sampling.
3. All units from the selected clusters are selected (or can be drawn randomly from within each cluster).

123
Q

How do researchers choose the best sampling method?

A
  1. List research goal
  2. Identify potential sampling methods
  3. Test them
  4. Choose the best
124
Q

Define external validity.

A

External validity: the degree of confidence that the causal relationship being tested is not influenced by other factors or variables.

125
Q

Define attrition.

A

Attrition: the loss of study units from a sample.

126
Q

Define non-response bias.

A

Non-response bias: when those unwilling or unable to take part in a research study are different from those who do.

127
Q

What does N signify?

A

The total number of individuals or observations in a sample.

128
Q

What function is chosen in Excel to sample and randomise?

A

RANDBETWEEN

129
Q

What four features must be calculated to determine sample size?

A
  1. Population size
  2. Margin of error
  3. Confidence level
  4. Expected variance
130
Q

Define standard deviation.

A

Standard deviation: a measure of how dispersed the data is in relation to the mean.

131
Q

Define effect size.

A

Effect size: how meaningful the difference between variables or the difference between groups is.

Null effect = 0
Small effect size = -0.05 to -0.15
Moderate effect size = -0.15 to 0.30
Large effect size = 0.30 and above

132
Q

Define minimum detectable effect size (MDES).

A

Minimum detectable effect size (MDES): the effect below which we cannot precisely distinguish the effect from zero, even if it exists.

133
Q

Define explained variance.

A

Explained variance: the proportion to which a design takes into account the variation of a data set.

134
Q

Define Cohen’s d (effect size).

A

Cohen’s d (effect size): measures the size difference between two groups.

135
Q

Define null result (or effect).

A

Null result (or effect): a result without the expected content.

136
Q

What is sensitivity analysis?

A

Sensitivity analysis: used to identify how much variations in the input values for a given variable impact the results for a model.

137
Q

What are the seven steps to be taken to design a study?

A
  1. Choose a randomisation type
  2. Choose the MDES
  3. Choose participants assigned to treatment
  4. Estimate explained variance
  5. If cluster randomisation, estimate RHO
  6. Conduct sensitivity analysis
  7. Integrate non-response and attrition rates
138
Q

What information should be collected to increase explained variance?

A
  1. Pre-treatment values (pretest)
  2. Individual predictors
  3. Contextual predictors
139
Q

Define pretest.

A

Pretest: when researchers test questionnaires on some members of the treatment group to evaluate the reliability and validity of them before they are distributed.

140
Q

What are three reasons why the pretest is important?

A
  1. Measures the equivalence of the treatment and control groups
  2. Analyses the dynamic effects of the treatment
  3. Increases the precision of estimates (considerably)
141
Q

Define statistical power (or sensitivity).

A

The likelihood of a significance test detecting an effect when there actually is one.

142
Q

What are the five factors that affect statistical power?

A
  1. N
  2. Number of treatment and control groups
  3. Percentage assigned to treatment group
  4. Randomisation design
  5. Sampling design
143
Q

What are the seven types of data collection methods?

A
  1. Experiment
  2. Survey
  3. Interview/focus group
  4. Observation
  5. Ethnography
  6. Archival research
  7. Secondary data collection
144
Q

Define experiment data collection.

A

To test a causal relationship.

145
Q

Define survey data collection.

A

To understand the general characteristics or opinions of a group of people.

146
Q

Define interview or focus group data collection.

A

To gain an in-depth understanding of perceptions or opinions on a topic.

147
Q

Define observation data collection.

A

To understand something in its natural setting.

148
Q

Define ethnography data collection.

A

To study the culture of a community or organisation first-hand.

149
Q

Define archival research data collection.

A

To understand current or historical events, conditions, or practices.

150
Q

Define secondary data collection.

A

To analyse data from populations that cannot be accessed first-hand.

151
Q

What is administrative data?

A

Data that organisations collect about their operations

152
Q

What is big data?

A

Large and hard-to-manage volumes of data that inundate organisations.

153
Q

What are computer-assisted personal interviews (CAPI)? List their advantages and disadvantages.

A

Data collected face-to-face with the assistance of technological devices that capture the responses.

Advantages: population coverage, reliable data, can be long and complex.
Disadvantages: expensive.

154
Q

What are computer-assisted telephone interviews (CATI)? List their advantages and disadvantages.

A

Data collected from a trained interviewer calling from a call centre.

Advantages: population coverage, reliable data, cheap.
Disadvantages: response rates, can’t be long and complex.

155
Q

What are computer-assisted web interviews (CAWI)? List their advantages and disadvantages.

A

Data collected through online surveys or interviews.

Advantages: cheap.
Disadvantages: response rates, can’t be long and complex, population coverage and unreliable data.

156
Q

What does the quality of a study rely on?

A
  1. Internal validity
  2. External validity
  3. Measurement
157
Q

Identify the three aspects of internal validity and define each.

A
  1. Initial equivalence: before the manipulation of the causal variable, participants in different conditions are the same on average on all the measured variables.
  2. Treatment contamination: when members of the control group inadvertently receive the treatment or are exposed to the intervention.
  3. Treatment replacement: when members of the treatment group are put back into the population before other members are sampled.
158
Q

What are the five aspects of external validity?

A
  1. Population list
  2. Response rate
  3. N
  4. Sampling design
  5. Randomisation design
159
Q

What are the three aspects of measurement?

A
  1. Time frame
  2. Coverage
  3. Validity of the indicators
160
Q

What are the two types of external validity?

A
  1. Population validity: how generalisable the findings are to other populations.
  2. Ecological validity: how generalisable the findings are to other situations or settings.
161
Q

What is the tradeoff between internal and external validity?

A

The more generalisable, the less extraneous variables are accounted for; and vice versa.

162
Q

What are the seven factors that determine coverage?

A
  1. Primary and secondary outcomes
  2. Primary and instrumental outcomes
  3. Positive and negative outcomes
  4. Reception of the intervention
  5. Baseline information to enhance statistical power
  6. Contextual and individual moderators
163
Q

Define heterogeneity analysis.

A

A measure of the differences between participants in a study.

164
Q

What is an indicator?

A

An observed value of a variable.

165
Q

Define reliability.

A

The extent to which results can be reproduced when the research is repeated under the same conditions.

166
Q

Define validity.

A

The extent to which results measure what they are supposed to measure.

167
Q

What are the four types of questionnaire designs? Define them.

A
  1. Multiple choice questions
  2. Open-ended questions
  3. Likert scale: a scale used to represent people’s agreement or disagreement towards a statement or question on a topic
  4. Semantic differentials: a scale used to represent a person’s attitude towards something using bipolar adjectives
168
Q

What are the two golden rules for designing questionnaires?

A
  1. Run a pretest
  2. Keep in mind that people are not always accurate. They are subject to social desirability, short memory, superficiality, and are easy to influence

For example, respondents understate alcohol and drug use, and tax evasion and racial bias; while they overstate church attendance, charitable contributions, and the likelihood they will vote in elections.

169
Q

Why does wording matter in questionnaires?

A
  1. Clarity and understanding
  2. Consistency
  3. Bias prevention
  4. Sensitivity
  5. Response options
170
Q

What is the guideline for questionnaires?

A
  1. Must be short
  2. Must be simple
  3. Not assume ‘they know’
171
Q

What is the guideline for answers?

A
  1. Exhaustive answers
  2. Mutually exclusive
  3. Include don’t know, or don’t want to answer
172
Q

What two elements should be avoided in questionnaires?

A
  1. Double negatives
  2. Jargon and unfamiliar abbreviations
173
Q

What variables are important when thinking about questions and answers?

A

Education level, age, and nationality

174
Q

How do aspects of quantitative and qualitative approaches to program evaluation differ?

A

Quantitative:
- Standardised data collection
- Random sampling
- Large samples
- High reproducibility

Qualitative:
- Flexible data collection
- Ad hoc sampling
- Small samples for in-depth analysis
- Low reproducibility

175
Q

Define reproducibility.

A

When researchers arrive at the same results using their own data and methods.

176
Q

Define replicability.

A

When separate researchers arrive at the same results using the original researchers’ material.

177
Q

What are the three functions of program evaluation?

A

Function 1: Monitoring evaluation delivery
Function 2: Understanding the limitations of RCTs
Function 3: Intervention design (legal and economic issues)

178
Q

Who should be interviewed and sampled?

A
  1. Participants: including non-compliers and leavers
  2. Secondary subjects: including policymakers, implementers, and external observers
179
Q

What are the three most common methods of data collection?

A
  1. Individual interviews
  2. Direct observation
  3. Focus groups
180
Q

How is the global quality of an evaluation study measured?

A
  1. External validity
  2. Internal validity
  3. Measurement
  4. Process evaluation
  5. Acknowledgment of limitations
181
Q

What are the four threats to external validity?

A
  1. Targeting of a highly selected population. Solution: multi-site RCTs.
  2. Sampling: non-randomised and small N. Solutions: List of the population, Sampling method, Ex ante power calculations.
  3. Enrolment: refusal to participate, attrition. Solutions: Emphasise benefits and importance of evaluation, Minimise hidden costs of participation (time), Get support from trusted stakeholders, Use delayed treatment (waiting list).
  4. Data collection: refusal to respond, missing, longitudinal attrition. Solutions: Proper method, Short and simple questionnaires, Pretest, Response incentives (money, small gifts), Interviewers’ quality and training.
182
Q

What are longitudinal studies?

A

A correlational type of research in which researchers observe and collect data on a number of variables without trying to influence those variables.

183
Q

What are the three threats to internal validity?

A
  1. Different participation, response and attrition rates. Solutions: Delayed treatment, Different incentives, Initial agreement.
  2. Spillover-effects between treatment and control group. Solutions: Delayed treatment (may cause loss of long-term outcomes), Proper randomisation design, Survey controls to combat contamination, Reduce circulation of materials and the content of the intervention.
  3. Treatment replacement. Solution: delayed treatment.
184
Q

To whom and how is data disseminated after the study has been completed?

A
  • Policymakers: report with executive summary; workshops and meetings.
  • Practitioners: websites, podcasts, social media, blogs; kit of materials with instructions.
  • Public / Participants: websites, podcasts, social media, blogs; short reports.
  • Researchers: research protocols; scientific journals, books and conferences; data repositories and meta-analyses.
185
Q

What are the three ideals of data dissemination?

A
  1. Independence
  2. Simplicity
  3. Replicability

Policy Analysis (1 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions