Poli399 Quiz 2 Flashcards
Study
What is validity?
Validity – are we measuring what we are measuring, measurement validity
What is reliability
Reliability – Does our measurement process assign values consistently
What is a measurement error?
Measurement error – difference in the values assigned to observations, attributable to flaw in the measurement
What are systematic errors?
Systematic errors – occurs when our indicator is picking up some other property it is supposed to measure. This type of error systematically biases our results
Systematic –> Validity based
What are random errors?
Random errors – chance fluctuations in the measurement results that do not reflect true differences in the property being measured.
Random error –> Reliability and validity
What is content validity?
Content validity is the measure of how relevant or a measurement is actually what the information actually is.
What is face validity?
The degree to which a procedure, a test or assessment, appears effective in terms of its stated aims.
What is sampling validity?
The degree to which the measurement represents the full range of meaning to its target. And if this measure is complete.
What are some potential problems with face validity?
Relies on subjective judgement.
No replicable rules for evaluating the measure.
What are some potential Problems of sampling validity.
Relies on subjective judgement.
No replicable rules for evaluating the measure.
Difficult to specify the universe of content of abstract concepts.
Harder to represent that content completely.
What is criterion validity?
Is an indicator is valid is there is an empirical correspondence between the results obtained using the indicator and the results obtained using another indicator of the same concept that is already known to be valid.
How does the convergent discriminant approach work?
Requires indicator of at least two different concepts, each measured using at least two different methods.
(Criterion related validity) This form of validation(convergent discriminant approach) raises three questions.
- Why not use the criterion instead?
- How do we know the criterion is valid?
- What if we lack a valid criterion?
I don’t know…
What is construct validity?
And what is this known as?
The degree to which a test measures what it claims, to be measuring.
Does our indicator produce relationships with indicators of other concepts that our theoretical understanding of the target property would lead us to predict?
Construct validity has an acronym for its approach. What is the acronym?
AHEM.
Assume Hypothesis,Evaluate the Measure
What are some potential problems of Construct validity?
Our indicator is not valid.
The theoretical framework is flawed.
The indicators of other concepts were not valid.
What is the solution to the problems of construct validity’s problems?
Conduct Multiple tests.
What are the four ways to assess reliability?
The Test retest method.
Alternative forms/ Split half method.
Coefficient Alpha
Sub sample Method.
What is test retest method?
What are the advantages and disadvantages of the test retest method.
You literally do multiple sets of the method to see if you get the same findings
Best when, your data does not react.
Bad when, feasibility, reactivity, real change in the cases.
What is Alternative forms method, and what are its advantages and disadvantages?
Two forms on the same data.
Best because, no reactivity, no time elapse, no confounding effect from possible changes in the cases
Worst cases, difficulty of ensuring that the two forms
What is the Split half method and what are its advantages and disadvantages?
Reliability is assessed by random dividing the items in half and comparing the results.
Advantages, avoids the problem of having to come up with two parallel forms
Disadvantages
Difficulty coming up with sufficient items
Are two halves really are equilivent
Different splits may lead to different assessments
What is the internal consistency method (Alpha coefficient) and what are its advantages and disadvantages?
The alpha coefficient is based on the average correlation for every possible combination of items into two half tests. Items that produce low correlations are deleted
Advantages
No reactivity
No time elapse
No confounding effect from possible changes in the cases
Feasibility
What is the subsample method and what are its advantages and disadvantages?
Divide the sample randomly into several subsamples. the same items are administered to each subsample and reliability is assessed by the similarity of responses across the subsamples.
No reactivity No time elapse No need to come up with twice as many items as needed Feasibility a large sample size is required
Why is research design so important?
Purpose – to impose controlled restrictions on our observations of the empirical world
Allows the researcher to draw causal inferences with confidence
Defines the domain of generalizability of those inferences
What is the nature of causal inferences?
We can never be certain that one variable “causes” another.
How to we Demonstrate co-variation
Show that the IV and DV vary together in a patterned, consistent way
How do we eliminate sources of spuriousness
Rule out the possibility that the IV and DV only co-vary because they share common cause
How do we establish time order?
how that a change in the IV preceded a change in the DV
How can we increase confidence in our causal inferences?
Demonstrate co-variation, eliminate sources of spuriousness, and establish time order.
What does the classical experimental design consist of?
An experimental group and a control group.
What is the difference between the control group and the experimental group?
The two groups are equivalent in every respect except that the experimental group is exposed to the IV and the control group is not .
The classic experimental design has 3 essential components that enable us to meet the 3 requirements for demonstrating causality.
What are the components?
Comparison –> Covariation
Manipulation –> Time order
Control –> Non spuriousness
What is Internal validity
A research design has internal validity when it enables us to infer with reasonable confidence that the IV does indeed have a causal influence on the DV.
What are Extrinsic & intrinsic threats to internal validity
Extrinsic threat to internal validity typically arises from the way we select our cases. Refers to selection biases that cause the experimental group and the control group to differ before the experimental group is exposed to the IV.
How do we ensure the groups are equivalent? (Extrinsic & intrinsic threats to internal validity)
Precision matching
Frequency distribution matching
Randomization
What are some Intrinsic threats to internal validity?
Changes in the cases being studies
Flaws in the measurement
Reactive effects of being observed
What are 6 intrinsic threats?
History
Events may occur while the study is underway which affect values on the DV quite independently of exposure to the IV
Maturation
The physiological and or processes may affect values on the DV quite independent of exposure to the IV
Morality
Selective dropping out may cause the experimental group & control group to differ on the post test
Instrumentation
Measuring instruments may perform inconsistency
Regression effect
Atypical pre test scores will appear more typical when they are post tested apart from exposure to the iV
Reactivity
The very fact of being pretested may cause peoples values change quite apart from exposure to the IV
How do we counter the 6 intrinsic threats?
History
Both groups are exposed to the same events
Maturation
Both groups undergo the same maturational processes
Morality
Selective dropping out will affect both groups equally
Instrumentation
Both groups will be equally affect by random errors in measurement
Regression effect
Both groups will be equally susceptible
Reactivity
If the pre test does affect values on the post test this will be true of both groups
What are threats to external validity?
Unrepresentative cases
Too specific to our study alone
Where we use volunteers, they are willing, which skews answers
Student samples are concerning cause they are a type of people, there is a power dynamic.
Try to be representative sample of a type a people
Artificiality of the research setting
We want to make environments that make people feel things
The more contrived and confusing it is, the more effective
Reactivity
People might think one thing, so they hide their true opinion and present themselves as more favorable.
They differ from real world behavior
What is a Quasi-Experimental design?
Attempt to use the logic of the experimental design where the researcher cannot randomly assign observations
Comparison and control are achieved statistically
Ex-post facto experiment
Attempts to approximate the post-test only control group design by using multivariate statistical methods.
What are control variables?
Testing a hypothesis showing that the IV and the DV vary together in a consistent patterned way.
Not enough to demonstrate an empirical association between the IV and the DV
Must go on to look at other variables that might alter or eliminate the observed relationship
Control variables are variables whose effects are “held constant” while we examine the relationship between the IV and the DV.
What are sources of spuriousness/Relationships?
A source of spuriousness variable is a variable that causes both the IV and the DV. Remove the common cause and the observed relationship between the IV and the DV will weaken or disappear.
What are intervening variables?
Mediate the relationship between the IV and the DV
Provide an explanation of why the IV and the DV correspond to the presumed causal mechanism
Ask your self why you think the IV would have a causal impact on the DV
What are Conditional Variables?
Conditional variables are varibales that literally condition the relationship between the IV and the DV by affecting
The strength of the relationship between the IV and the DV
Or the form of the relationship between the IV and the DV
To identify plausible conditional variables ask yourself whether there are some sorts of people for whom the IV will have predicted effect on the DV, in example they will have a particular value on the DV regardless of their value on the IV
What are three types of variables that condition relationships?
Variables that specify the relationship in terms of interest knowledge or concern (sailence).
Variables that specify the relationship in terms of place or tome (place).
Variables that specify the relationship in terms of social background characteristics (social background).
What is a research problem?
are always questions that display how one concept is related to another concept.
-the goal of a research problem is to maximize generalizability.
Why is generality important?
- the scientific method has generality as one of the goals.
- the research that one engages in has implications for the sample and the relationship being studied in that specific instance.
- the reason that people care about a research problem is due to its implications such as policy.
What is the research process?
find something to explain formulate research problem develop hypothesis (operationalize) identify plausible sources of spuriousness, intervening and conditional variables choose indicators collect and analyze data
What are Descriptive Statistics?
Used to describe characteristics of a population or a sample.
What are Inferential statistics?
Used to generalize from a sample to the population from which the sample was drawn they involve using a sample to make inferences about the population.
What is the difference between uni-variate, bi variate and multi variate statistics?
Uni-variate made to describe or make inferences on one variable.
Bi-variate use to describe or make inferences on the relationship between 2 variables
Multi-variate describes or makes inferences on the relationship between multiple variables.
What is a frequency distribution?
A list of the number of observations in each category of the variable. It displays the frequency with which each possible value occurs.
Called absolute frequencies or raw frequencies.
What is central tendency?
Central tendency indicates the most typical value the one value that best represents the entire distribution.
What is dispersion?
A measure of how typical a value is.
What is the preferred way to measure central tendency at the interval/ ratio level?
The mean is the preferred measure of central tendency because it takes into account the distance (or intervals) between cases.
If the central tendency at the interval/ ratio level has few cases of extreme values, what other method should we use to measure instead?
The median should be used instead.
What level of measurement is a cross tabulation measure at?
The nominal level.
What is the most common error in constructing a cross tabulation?
Is putting percentage wrong way.
What is the difference between a type 1 error and a type 2 error?
Type 1 Inferring that there is a relationship when none actually exists
Type 2 Inferring that there is no relationship when there really is a relationship.
Type 1 Error is always viewed as much more serious than a type 2 error.
What is a chi square test?
Gives the likelihood of each possible degree of relationship occurring in a sample if there were no relationship in the population from which the sample was drawn.
What is the logic of the chi square test?
Set up a null hypothesis
Calculate the expected frequencies
Compare expected cell frequencies with observed cell frequencies
Make a partial adjustment for sample size
Calculate the degrees of freedom
Consult the theoretical chi square distribution.
