reading 3 - review of core ideas Flashcards

Question

critiques, responses, and trade offs: drawing together the debate doing research that is important

Answer 1

KKV: scholars should study topics that are important in the real world and in relation to a given scholarly literature + limited attention to theory critique: - provides no guidance for how to choose important topics + how to achieve major advances in our substantive and theoretical understanding of politics and society (Rogowski) - doesn't address concern that KKV methodological norms (narrowing RQ) may make it harder to do research that is important - KKV provides no heuristics for theory construction (McKeown) - KKV warns against no-variance research designs, other scholars argue that these design are valuable (for within-case analysis) Rogowski: conflict between a. methodological goals of improving descriptive and causal inference on the basis of empirical data b. objective of studying humanly important outcomes and developing theory that helps us to conceptualize and explain them statistical response: - KKV explicit in saying it does not attempt to provide guidelines for theoretical innovation (discovery is irrational/creative intuition) - KKV focus on valid scientific inference NOT generation of hypothesis -> rejects no-variance design based on weak basis for causal inference - statistical inference = guidelines that increase the probability of generating a correct inference - conflict between achieving inferential goals and carrying out theoretically productive research = dilemma for all researchers, not just KKV challenges of promoting creativity can we establish procedures that promote theoretical creativity and lead to important research? - one hand = view that we lack systematic procedures for generating novel insights into political phenomena (ideas come from e.g. imagination) - other hand = reason to believe that some research practices are more likely to produce theoretical insights than others (e.g. deductive and inductive tools for gaining new insights) - specific research activities can be useful stimuli for theoretical innovation, e.g. fieldwork, studying anomalous cases "scholars can identify research practices that contribute either to improving inference or to promoting theoretical innovation, but not necessarily to both. we may often face a trade-off" -> scholars must recognize the value of both goals

Answer 2

KKV: - scholars should maximize the validity of measurements + use reliable data-collection procedures - choose observable rather than unobservable concepts wherever possible - skepticism about use of typologies - trade-off between maximize concreteness of theories and encompassing stating theories + descriptive richness and facilitation of comparison + measurement validity and reliability/precision critique: 1. KKV only briefly mentions conceptualization and measurment (while they require extensive attention) 2. KKV advice to employ concepts readily operationalizational -> underscores challenge to define difficult concepts (civil society, democracy, nationalism) 3. typologies are important (qualitative small N research), e.g. types of political regimes 4. KKV framework neglects basic ideas treatment of measurement (measurement error etc.) (Bartels + Brady) 5. KKV almost no attention to contextual specificity of conceptualization and measurement -> problematic claims as 'increase N', is not good: can bring analysis outside of the domain where concepts are appropriate and valid statistical/psychometrics response: writing linked to the traditions of psychometrics, mathemetical measurement theory, and statistics support the critics of KKV with respect to conceptualization and measurement validity: careful decisions about this are crucial - need for close attention to concept formation, measurement validity, and the contextual specificity of measurement - psychometric tradition: measurement validity and theory are mutually dependent - KKV warnings about avoiding unobserved/unmeasurable variables seems at odds with tradition of covariance-structure model + factor analysis - statistics + psychometricians +measurement theory = validity of a given indicator must always be treated as context-specific (KKV: we focus on descriptive and causal inference, not on concept formation and theory creation)

Answer 3

KKV: advice about selection bias, framing it as central problem in causal inference - focus on investigator-induced selection bias - researchers should select cases across the entire range of the DV selection bias = when there is unrepresentative sampling of cases or when nonrandom process assigns causes to cases critique = concern with selecting extreme DV values has been oversold + qualitative researchers have distinctive tools for making valid causal inferences (even with truncated sample) - KKV overextends rules and norms of conventional quantitative research. instead selection bias should be considered in light of trade-offs with other methodological and theoretical priorities - Collier, Mahoney and Seawright: within-case analysis no selectio bias when based on causal-process observations - KKV sees no-variance research designs as case of selection bias, this is true for regression analysis, but not neccesarily for other analytical tools 4. definition of selection bias depends on defining universe of cases (you need a perfectly defined universe to know if a sample is nonrandom) statistical response: in general agrees with KKV claims selection bias in regression analysis BUT agrees with critique concerning application KKV ideas on qualitative research - no-variance designs useless in regression analysis (effect will be equal to error term -> overall estimate causal effect is zero) !no-variance design on the IV is impossible for regression analysis, no-variance on the DV is not (causal estimates go to zero due to selection bias not bc maths) - many issues of bias can't be addressed without having a clear understanding of the relevant population - qualitative judgement is required if we are to consider broader goals of research design (broader than quantifying bias): describe amount of new theoretical and substantive knowledge the design will produce !critical note = investigator-induced bias is practice relatively uncommon -> concerns about selection bias due to truncation overstated +distracted attention from other forms of bias (e.g.: - self-selecting of individuals into the categories of included variables (observational studies) = democracies more likely than authoritarian regimes to break down in the face of poor eco performance -> some countries will be selected into regime type (IV) due to their scores on the DV (eco performance) - nonrandom sampling process: survey research, same type of people don't respond

Answer 4

KVV: exclusively probabilistic model of causation (rather than deterministic) - deterministic causation = models in which error variance is specified to be zero (no random component) - qualitative research: deterministic causation = models of necessary and/or sufficient causation critique: - KKV fails to recognize importance in qualitative research of hypotheses about deterministic causation + need to dev. tools that test such hypotheses - KKV rec. to seek variance on DV and IV -> may impede efforts to test deterministic causal models statistical response: some response for KKV, some support for the critique - probabilistic tests of deterministic causes: statistics supports critics: if deterministic cause is present and only a probabilistic model is tested there may be invalid inferences (will infer that there is some probability in the absence of a necessary cause) -> needs own procedure - necessary and/or sufficient causes and selection bias: testing deterministic causation often with no-variance research design -> KKV says bias not true: variance of the error term is zero -> no truncation (= when selection -> correlation between error term in the causal model and the IV -> overrepresenting atypical cases) overrepresentation atypical cases in deterministic models irrelevant bc models require even atypical cases to follow the overall pattern - identifying the most efficient test (are no-variance designs the most productive to assess deterministic causation?) -> KKV advice to seek variance on the IV and DV correct, but for diff reasons: all-cases design sometimes better (when small nr of cases), shows that no cases lack the IV and have the DV *cases with cause but not outcome are not relevant to falsify the hypothesis

Answer 5

deterministic causes increasingly seen as substantively important in the social sciences - deterministic causation = models in which error variance is specified to be zero (no random component) - qualitative research: deterministic causation = models of necessary and/or sufficient causation basis = if a single case deviates from a hypothesized causal pattern -> serious doubt on the hypothesis => single variable on its own has a distinctive causal impact - IV presence inevitably leads to an outcome if it is a sufficient cause - IV absence definitively prevents an outcome if it is a necessary cause

Answer 6

= useful when small nr of cases -> can draw on large pool where the outcome did not occur (any of the cases may not have the IV, but have the DV, but they did not) most appropriate tests for deterministic hypotheses = cases with the outcome OR cases that don't have the cause - cases with cause (IV) but not the outcome (DV) are irrelevant !Seawright: observational studies -> IV and DV not assigned by the researcher -> should not treat them as fixed -> should use all cases additional advantages all-cases design: - evidence against deterministic causation -> data can be used to test strength of probabilistic association - productive when: necessary or sufficient cause turns out to be both necessary and sufficient - productive when: what was hypothesized to be necessary proves to be sufficient or vice versa

Answer 7

trade-offs -> conflicts among goals of researchers - overarching goal = seek valid descriptive and causal inferences about important phenomena in the political and social world + refine theory e.g. KKV - intermediate goals = more specific research objectives (precise comm concepts, generalizability, parsimony) studies with divergent intermediate goals can make complementary contributions to overarching goals trade-offs bc tools employed in pursuing the goals - tool = specific research practice/procedure to achieve intermediate goals and through them overarching goals methodology = developing tools with reasoning about how particular tools succeed/fail in achieving research goals Rogowski = goals and tools involve trade-offs e.g. goals: more general theories often less accurate and parsimonious e.g. tools: no-variance may be biased, but may be good hwn basic descriptive info is lacking we do have standards: we need to be explicit about goals + strengths and weaknesses of alternative means for pursuing the goals

Answer 8

essence of research design = making choices among potentially incompatible goals trade-offs between: *KKV mentions 5 trade-offs, chapter doesn't mention them - precision and generality offered by quantitative tools + reliance on often untested assumptions required by these tools - between avoiding bias by including all relevant IV and seeking to maintain inferential leverage by limiting the nr of IVs - between representativeness and interpretabiity of quantitative tests associated with random sampling vs close focuso n theoretically relevent comparisons KKV most important = increasing nr of observations and other significant goals - increase N -> strengthens falsifiability + enhances explanatory leverage + addresses interdeminacy and multicollinearity - disadvantages = analysis drawn out of its appropriate domain + lac of measurement validity (is context specific) + diff to get knowledge of the context + adding cases can undermine independence of observations (temporal/spatial subunits -> can add observations that are not independent) trade-offs -> not one set of methodological guidelines can ensure that researchers will do good work

Answer 9

trade-offs -> methodological issues more complex than they appear in KKV (esp. increasing N) decisions about concepts, typologies, measurement relations and domains of measurement validity are largely neglected by KKV