02 Research Design - Issues and Challenges Flashcards
What do the authors Gschwend, Thomas and Frank Schimmelfennig mean by “research design”?
A research design is a plan that specifies how you plan to carry out your research projectand, particularly, how you expect to use your evidence to answer your research question.What is a relevant research problem? How can I improve concepts and measurements in myresearch? Which and how many variables and cases should I select? How can I evaluate rival explanations and which theoretical conclusions can I draw from my research? Whichevidence would lead me to reject and reformulate my initial theory? These are centralquestions political science students inevitably face when they embark on their own researchprojects in a Master’s or a PhD program.
Which are the five shared beliefs that provide the common ground for research?
- The methodological pluralism in our discipline is a strength rather than a weakness.
- The basic problems of research design are the same for qualitative and quantitativepolitical science research.
- The methodological debate in the discipline often remains at an abstract level and doesnot give sufficient practical guidance for dealing with basic research design problems.
- The distinction between qualitative and quantitative research is often inadequate. Somesolutions to research design problems are common to both types of research; others crosscutthe traditional qualitative-quantitative divide.
- At any rate, finding solutions to research design problems involves substantial trade-offsalong the way. Each solution has its strengths and weaknesses.
When you define your research problem, it is important to decide and justify what tostudy. What do the authors mean with “theoretical and scientific relevance” and “socialrelevance”?
When deciding on what to research, the researcher must consider both theoretical/scientific relevance and social relevance. Scientific relevance involves advancing the collective dialogue between theory and data, identifying problems in the discipline, and improving theory and data. Social relevance involves addressing social problems, improving understanding of the problem, and potentially offering solutions. Researchers must justify which problems they choose to address as there are many problems and puzzles to choose from.
Regarding the specification of concepts, the authors state that “It must be clear whatwe mean by a specific concept, that is what its defining attributes are, how attributes andconcepts relate to each other, and which empirical phenomena they include and exclude.” What is meant by this?
Specifying concepts: Whether we formulate and test theories or describe and explainobservations, we inevitably use concepts such as ‘democracy’, ‘party’, ‘conflict’, and ‘peace’.In order to make research relevant, these concepts need to be theoretically and/or sociallyimportant. But they also need to be (properly) specified. It must be clear what we mean by aspecific concept, that is what its defining attributes are, how attributes and concepts relateto each other, and which empirical phenomena they include and exclude. What attributesdefine a ‘democracy’? Does ‘peace’ exclude ‘conflict’? How do ‘parties’ differ from otherorganizations? Clear and unambiguous concepts are not only required for formulatingtestable theories in the first place. When engaging in a theoretical controversy, the researcher needs to examine the concepts of the competing theories – especially when thetheories use the same terms. Starting from the data, descriptive inference requires no lesscareful concept specification – for instance, if you make statements like ‘The majority ofstates are democracies’ or ‘The occurrence of war is decreasing’.
In “specifying theory”, the authors argue that researchers need to specify therelationship between concepts. What different types of relationships do the authors list?
Theories in social science aim to specify cause-effect relationships between concepts, including the order, form, and direction of the relationship. They also need to specify how various causes are related and whether they are necessary and/or sufficient conditions of the outcome. Theories should explain the causal mechanisms that link cause and effect and specify microfoundations that show how social structures and environments translate into individual desires and beliefs, which in turn produce actions and social outcomes. The more a theory is specified, the better it can explain observations and be tested.
How do we measure concepts?
By specifying concepts and theory, we arrive at testable theoreticalpropositions. In order to conduct the empirical test, however, the concepts need to beoperationalized and measured. Obviously, democracy – even if clearly specified as a concept– cannot be observed directly. This is often also true for the defining attributes. Alvarez andcolleagues (1996), for instance, define democracy as a political regime in which offices arefilled by contested elections. They then go on to provide ‘operational rules’, which specifythe offices that need to be included (the chief executive and the legislature) and indicatorsof ‘contestation’ (above all that there has to be more than one party).
How are cases selected? On what basis?
Problems of case selection and selection bias are core issues in bothquantitative and qualitative methods textbooks. To be precise, we need to distinguishbetween units of analysis, cases, and observations. The unit of analysis is the abstract entitythat we study (e.g., states, institutions, decisions) which is often given by the theory. ‘Case’refers to the specific units of analysis that we choose to analyze. If the unit is ‘state’, thiscould be a single-case study of Sweden or a comparative case study of Sweden and Norway.
What is meant by “controlling for alternative explanations”?
In the dialogue between theory and data, we use theory to test it based on selected cases and measurements or to explain a set of observations or a specific outcome. However, even if we find a strong relationship between the theorized causes and observed effects, we need to ensure that this relationship is not spurious and that other causal factors would not explain the observations just as well or better. Therefore, we must address and control for alternative causal factors and explanations in our research. For example, the democratic peace may be attributed to the hegemony of liberal great powers or high economic interdependence between democratic countries.
How are theoretical conclusions drawn?
Let us assume we have successfully tested a well-specifiedtheory with valid and reliable measurements on an unbiased selection of cases and that wehave been able to reject alternative explanations. In this case, the theory is corroboratedand does not need to be revised or rejected. Often, however, we will encounter anomaliessuch as deviant cases or statistically insignificant relationships.
On page 8, the authors Munck, Gerardo L. and Jay Verkuilen write: “because conceptualization is both intimately linked withtheory and an open, evolving activity that is ultimately assessed in terms of thefruitfulness of the theories it helps to formulate (Kaplan, 1964, pp. 51-53, 71-78), “there isno point in arguing about what a ‘correct’ definition is” (Guttman, 1994, p. 12; see also p.295).” What do the authors mean by this?
To construct a dataset, attributes that are part of the concepts under consideration need to be identified. However, there is no imposition on what attributes should be considered. Conceptualization is linked to theory and an open, evolving activity that is assessed in terms of the theories it helps to formulate. Therefore, there is no correct definition, and scholars should avoid including too much or too little in a definition relative to their theoretical goals.
Concept specification: Avoid maximalist definitions (the inclusion of theoretically irrelevantattributes) or minimalist definitions (the exclusion of theoretically relevant attributes)Conceptual logic: Isolate the “leaves” of the concept tree and avoid the problems ofredundancy and conflation
What problems do the authors see with maximalist definitions?
Maximalist definitions, which include too many attributes, have two potential drawbacks. First, it may decrease the usefulness of the concept by making it impossible to find empirical referents, such as including social justice as an attribute of democracy. Second, even if empirical instances can be found, maximalist definitions tend to be overburdened and of little analytical use, such as linking market-based economic systems to democracy, which forecloses analysis of interesting issues that cannot be resolved by a definitional fiat.
What problems do the authors see with minimalist definitions?
To avoid the problem of maximalist definitions, analysts often use minimalist definitions that make it easy to find instances of a concept and study numerous empirical questions. However, if a concept is too minimalist, all cases automatically become instances, and researchers must add attributes to give it more content to address theoretical concerns and discriminate among cases. Therefore, analysts must also be aware of the problem of minimalist definitions, the omission of a relevant attribute in the definition of a concept.
On page 12, the authors Munck, Gerardo L. and Jay Verkuilen write “First, the specification of a concept’s meaning frequently
entails the identification of attributes that vary in terms of their level of abstractness.” What
do they mean by this? (See also Figure 1)
To fully understand a concept, analysts need to identify its attributes and organize them vertically by level of abstraction. This task is often overlooked but affects data generation and subsequent challenges of measurement and aggregation. Attributes vary in their level of abstractness, and organizing them in this way helps analysts tackle the challenge of measurement by isolating the most concrete attributes, which serve as the starting point for measurement. This organization is achieved by distinguishing attributes according to their levels of abstraction and labeling them as attributes, components, sub-components, etc.
What do the authors Munck, Gerardo L. and Jay Verkuilen mean by “aggregation”, “conflation” and “redundancy”?
aggregation refers to the process of combining disaggregated data or scores assigned to each leaf of a conceptual tree. Redundancy is a logical problem that occurs when attributes at the same level of abstraction tap into overlapping or redundant aspects of the attribute at the immediately superior level of abstraction. Conflation is another logical problem that arises when attributes are conjoined with attributes that are manifestations of a different overarching attribute, resulting in the mixing of distinct or vaguely related aspects of the concept.
How does BenNun Bloom specify concepts? (State-Level Restriction of ReligiousFreedom and Women’s Rights:A Global Analysis)
Veryimportant, but also more important in some cases than in others(e.g., in political theory, clear specifications are much more important thanin quantitatively oriented cross country comparative research)
Much easier, if you already know the literature in the field (this also meansthat you know about different definitions and conceptualizations, andprobably also about their strenghts and weaknesses)
Important to be clear: If you are not able to specify the concepts you use, itusually means that you are not thinking clearly about your topic.
However, common problems are to be vague about concepts, and write too much about definitions. It is also important to think about the theory beforehand, and make clear use of existing literature by showcasing it. Mention how you expect your theory, variables etc to lead to your research answer and how you you think about your solution in comparison to previous findings.
