Week 3 : Conceptualization, Generalization, and Validity

Question 1

Process of conceptualization & operationalization

Answer 1

• abstract concept in hypothesis –>
• (conceptualization) concrete definition of concept –>
• (operationalization) measure of defined concept

Question 2

Definition of conceptualization

Answer 2

the process of precisely defining ideas and turning them into variables

Question 3

Definition of Operationalization…

Answer 3

The process of linking the conceptualized variables to a set of procedures for measuring them

Question 4

Example conceptualizing poverty

Answer 4

• defining the concept ‘poverty;
• unit of analysis = person, household, country, etc.
• ‘have less’, ‘have not’, ‘minimum’, ‘basic needs’

Question 5

Example operationalizing ‘poverty’

Answer 5

• measuring the concept ‘poverty’
• absolute standards (specific quantitative assessment) = whether the person has food, clothing, shelter, etc.
• relative standard = whether the person falls below the average income level (many other options)

Question 6

Concepts

Answer 6

• highly abstact ideas that summarize social phenomena & are linked together within hypotheses
• rigorous empirical social science wants to make these concepts into something more concrete (this starts with translating concepts into variables

Question 7

Variables…

Answer 7

• representations that capture the presence or absence of a concept or the level of a concept
• E.g. ‘are you in love? Yes or no’ and ‘how much would you say you love your partner?’
• To study love, researchers have to change the concept of ‘love’ to something concrete that someone can have more or less of than someone else
• in this way, variables break down concepts into data points that can be compared

Question 8

Units of analysis

Answer 8

• the unit of analysis is the level of social life that we want to generalize
• social artifacts are a unit of analysis that encompasses aspects of social life that can be counted
• a single concept can be studied with different units of analysis

Question 9

Dimensions…

Answer 9

• general concepts usually encompass multiple dimensions
• components that represent different manifestations, angles or units of the concepts
• dimensions are about conceptualization

Question 10

4 main types of variables

1) Nominal variable

Answer 10

• categorial variable (finite set of values)
• categorizes states/statuses that are parallel and cannot be ranked or ordered
• e.g. race, hair colour, school sector

Question 11

4 main types of variables

2) Ordinal variable

Answer 11

• categorial
• have categories that can be organized in some way
• categories can be ranked from low-high, but the difference/distance between ranks cannot be known
• e.g. school quality

Question 12

4 main types of variables

3) Interval variable

Answer 12

• continuous
• have a continuum of values with meaningful distances (or intervals) between them, but no true zero
• categories can be compared explicitly instead of just saying one is better than the other, but we cannot think of them in terms of proportions (cuz no true 0)
• E.g. temperature, time of day, SAT score

Question 13

4 main types of variables

4) ratio variables

Answer 13

• continuous
• interval variables that do have a true 0
• so they can be compared & contrasted with most detail
• differences can be expressed in terms of the distance between values but also in proportion or percentage of another
• e.g. school size, body weight

Question 14

Composite variables

Answer 14

• one that averages a set of variables to measure one conept
• internal reliability… the degree to which the various items in a composite variable lead to a consistent response (use Chronbach’s alpha)

Question 15

Indicators

Answer 15

• the last step
• indicators are the values assigned to a variable
• they are part of the plan that researchers develop to sort the variable into various categories
• Identifying an indicator is a conceptual procedure that provides the basis for measurement
• Indicator : a household will be classified as ‘poor’ if they have the value of ___ in the variable___

Question 16

Operationalization involves 2 steps…

Answer 16

1. converting a conceptual definitions into an operational definitionn that sets the parameters for measurement
2. using the operational definition to collect data

Question 17

Operationalization in quantiative vs qualitative…

Answer 17

• quantitative = operationalization is an end result of the conceptualization process
• qualitative = usually starts with a more open conceptualization process & observation refines the process to lead to a conceptual definition

Question 18

Reductionism…

Answer 18

• occurs when conclusions are made about the macro level unit on the basis of analyses of micro-level data
• reduces something complex into too-simple terms or simplifies societal or group phenomena into individualist explanations

Question 19

4 basic forms of measurement

1 - Reports

Answer 19

• direct feedback, written or verbal, from people
• 1) open ended questions
• 2) closed ended questions
• 3) response categories (must be exhaustive & categories must be mutually exclusive)

Question 20

4 basic forms of measurement

2 - Observation

Answer 20

• Observation is the process of seeing, recording & assessing social phenomena
• commonly used in qualitative studies (ethnography)
• generally focuses on… degree & amount/frequency

Question 21

4 basic forms of measurement

3 - artifact counts/assessments

Answer 21

• are tied to content analysis and similar materials-based methodologies
• involves the cataloging of social artifacts & objects qualitatively or quantitatively

Question 22

4 basic forms of measurement

4 - Manipulation

Answer 22

• usually paired w other forms of measurement during experiments
• experimenters need to ‘do something’ to subjects & observe results
• control group & experimental group

Question 23

Cross-sectional study designs

Answer 23

• data are collected at only one point from different groups of ppl

Question 24

Longitudinal study designs

Answer 24

• data are collected at multiple points in time
• repeated cross-sectional study design = data are collected at multiple points but from diff peple ay each time (new sample at each stage)
• panel design = data are collected multiple times from the same ppl, the same sample at each stage (attrition problems)

Question 25

Assessing measurements

Reliability

Answer 25

• refers to dependability, consistency and predictability
• how consistently the same operation yields the same results
• not about being correct, just being stable & predictable

Question 26

Assessing measurements

Validity

Answer 26

• Refers to the degree to which a measure truly and accurately measures the defined concepts
• This is highest when conceptualization is strongly linked to operationalization
• a highly valid measure/method gets to the heart of the concept being mesured
• validity is about being correct, not stability
• much harder to assess than reliability

Question 27

Qualitative research

credibility & dependability

Answer 27

• credibility = Do the respondents agree with how the researcher is presenting and interpreting their words?
• dependability = whether the findings are consistent & whether the same findings would be repeated given the same circumstances/context

Question 28

Reliability

Chronbach’s alpha

Answer 28

• straightforward calculation that measures a specific kind of reliability for a composite variable
• composite variable = a variable that averages a set of items (typically from a survey) to measure some specific concept
• internal reliability = concents the degree to which the various items lead to a consistent response & therefore is the same concept
• higher the alpha the more internally reliable the composite is - meaning ppl respond to the various items in a consistent way (0-1)

Question 29

Reliability

Intercoder reliability

Answer 29

• when multiple observers or coders are used in data collection, it’s important to report meaures of intercoder reliability
• reveals how much difference coders/observers agree with one another when looking at the same data
• Cohen’s kappa = statistic for assessing intercoder reliability that guages level of agreement between 2 raters in categorizing the same set of cases and is measured on a 0-1 scale

Question 30

Reliability in design

Answer 30

• the most effective method for increasing the reliability of a research method is to do a very careful job of conceptualization
• Conceptualization that is both thorough and sound will generally point to the most reliable operationalization

Question 31

Reliability

Precision

Answer 31

• precision is a key elemenet of meaurement that supports reliability
• the more detailed & precise the measures are, the more reliable they tend to be
• tip is to use multiple indicators to measure any one construct

Question 32

Reliability

Robustness

Answer 32

• how well the operational protocol is working
• aims to guage the robustness of some operational approach in the early stages of a study so that the researcher can asses any reliability problems before the study is fully underway
• split-half method, test-retest method, pilot testing

Question 33

Reliability - robustness

1 - split-half method

Answer 33

• appropriate when dealing with a measure consisting of multiple items
• researcher randomly splits the set of items for a measure into 2 sets to create two seperate measures instead of one
• these two measures are then tested in a sample of individuals
• comparing how the same ppl respond to the two measures gives some idea of how reliable the overall measure would be

Question 34

Reliability - robustness

2 - test-retest method

Answer 34

• the same measure is administered to a sample and then readministered ot the same sample later
• if the scores are similar it is reliable
• issue… other factors could change responses that have nothing to do with reliability

Question 35

Reliability - robustness

3 - pilot testing

Answer 35

• involves administering some version of a measurement protocal to a small preliminary sample of subjects as a means of assessing how well the measure works & whether any problems need fixed
• after the polit test, researchers make necessary corrections before officially administering the measure to a larger sample

Question 36

Validity

Internal validity of a study

Answer 36

• refers to the degree to which a study establishes a causal effect of the independent variable on the dependent variable
• experimental designs are the most internallt valid cuz they can establish causality

Question 37

Validity - internal validity of a measure

1 - face validity

Answer 37

• the simplest dimension of internal validity
• whether something looks valid on the face of it
• the 1st and most shallow assessment of validity that a reasearcher could make

Question 38

Validity - internal validity of a measure

2 - criterion-related validity

Answer 38

• assesses the validity of a measure according to how closely it is associated with some other factor
• we gain confidence in the validity of a measure when it’s related to things which it is supposed to and unrelated to things which it should be unrelated
• concurent validity = will correlate strongly with some pre-existing measure of the construct that has already been deeemed valid
• predictive validity = it will correlate strongly with a measure that it should predict
• why not just use the best pre-existing meaure? feasibility

Question 39

Validity - internal validity of a measure

3 - content validity

Answer 39

• all about coverage
• how well a measure is encompassing the many different meanings of a single concept & all possible answers

Question 40

Validity - internal validity of a measure

4 - construct validity

Answer 40

• refers to how well multiple indicators are connected to some underlying factor
• important cuz some phenomena are not directly observable and can only be identified by catalouging their observable symptoms
• convergent validity = a dimension of validity guaging whether concepts that should be associated with each other are
• discriminant validity = a dimension of validity guaging whether concepts that should not be associated with each other are not

Question 41

construct validity example

Answer 41

Example: Happiness
A. In the past week, how often have you felt at peace?
B. In the past week, how often have you felt angry?
C. In the past week, how often do you take public transport?

Question 42

Factors threatening internal validity…

Answer 42

• Participant attention
• Experimentor effects
• Selection bias (control & treatment groups differ)

Question 43

External validity…

Answer 43

• the degree to which the results of a study can generalize beyond the study
• involves 2 basic questions
• 1) how representative is the group being studied? (E.g. if only men are included on a study about parenting that doesn’t really make any sense)
• 2) how ‘real’ is the study?