Exam 2 Flashcards
sampling error
measurement =/ reality ex: variable (scale) – 1,3,4,5 i. Statistic (sample) – average = 3.75 ii. Parameter (population) – average = 3.72
i. Thus as sample size increases…
1. Sampling error decreases
2. Also, statistical power increases
ii. You need to get the sampling error down to a reasonable number
Perfect Probability Sample
a. The bar charts are exactly the same sample vs. population
Less-than-perfect probability
a. The bar charts differ sample vs. population
Normal curve
(normal distribution/bell curve) U shape symbol = mean/average
Nonprobability Sampling
i. Available subjects (convenience, generalization)
ii. Purposive sampling (judgment of usefulness and representativeness)
iii. Snowball sampling (person to person to…)
1. Sample is getting bigger as it passes from person to person and as we progress
iv. Quota sampling (pre-specified characteristics to match population)
1. Pick a specific number of samples to match “quota”
Types of sampling designs
a. Simple Random Sampling (random numbers 1-100)
b. Systemic sampling (kth numbers 1-100)
c. Stratified sampling (random selection from homogeneous strata)
d. Cluster sampling (random sample of clusters and then random selection from clusters)
Thomas E. Dewey
a. Born in Owosso, Michigan
b. University of Michigan and the Columbia university law school
c. Federal prosecutor and then governor of New York
d. Republican candidate in the 1948 presidential election
e. Gallup polls
i. Polling ended in early October
ii. Undecided voters went Truman
iii. Reliance on quota sampling (following WWII: population movement to cities
Probability Sampling
a. Population (aggregate of elements)
b. Sample
c. Sampling bias (sample =/= population)
i. Ex: 2 men having to represent 50 men.
d. Random selection (equal chance of selection)
e. Representativeness
f. Sampling error (degree of error in sample design)
g. Statistic (descriptive summary of a variable)
h. Sampling frame (a list of units that compose a population from which a sample is selected)
i. Weighting
Confidence level
– the estimated probability that a population parameter lies within a given confidence interval
i. Tells us how sure we can be
ii. Normally 95% confidence interval
Hawthorne Effect
a. People approve upon something simply as a result of their being studied.
b. Hawthorne Works commissioned an evaluation of how its workers would respond to having lower-to-higher levels of lights
c. Worker productivity improved during the study, but slumped afterward. (People didn’t want to lose their jobs when they were being watched)
List of appropriate topics for Survey Research
a. Units of analysis = respondents
b. Large samples, original data, measuring attitudes and orientations
c. Questionnaires:
i. Open-ended (broad; Asking without getting certain responses – “What did you do last night)
ii. Closed-ended (yes or no; choose from a list of answers – “What’s your age?)
d. Make items clear/relevant.
e. Avoid double-barreled questions (and/or)
i. Ex: “do you like Texas A&M or liver? Yes or no?
ii. Instead of asking two different things at the same time, ask two questions
f. Avoid negative items (NOT)
g. Avoid biased items and terms (welfare)
h. Avoid leading questions (like most Americans, don’t you think…)
Filter questions on survey questionnaires
i. “Have you ever smoked a cigarette in your life?”
1. The people who haven’t don’t answer anymore questions about it.
2. People who have get asked, “Have you smoked it in the last year? Have you smoked in the last month? How many do you consume each day?
3. This saves time and energy for those taking the survey…
Contingency Questions in Survey questionnaires
i. Questions that you answer in accordance with the first answer.
1. “Do you go to Texas A&M?” – “yes”
a. “Where do you live: on-campus or off-campus?”
Self-administered survey
- Open-ended vs. close-ended questions
- Mutually exclusive and exhaustive response choices
- Don’t know/no response/no opinion/undecided
- Primary vs. secondary
- Response rate: Ideal = higher than 70%
Primary vs. secondary data
a. Primary: the researcher has selected it themselves
b. Secondary: readily available and researched by another researcher
i. Sometimes it doesn’t have the proper question we want to use.
response rate: Mail distribution and return
why do people not return questionnaires?
- People perceive that they don’t have enough time
- We are constantly being sent surveys and we get overwhelmed, and therefore don’t respond.
Telephone (landline, cell) survey
advantages vs. disadvantages
- Advantages
a. Almost all households have a telephone
b. Time and money - Disadvantages
a. Unlisted phone numbers
b. Cell phones - Random-digit dialing (RDD)
- Computer-assisted telephone interviewing (CATI)
- There is more pressure for the interviewee to answer and not ignore
online survey
- Survey Monkey, Qualtrics, Survey Sampling Intl.
a. Search links, banner ads, email, online invitations, and co-registration
b. Digital fingerprinting implemented to authenticate respondents. - Low levels of interviewer and response biases
a. Impersonal; you don’t feel pressured to respond
Pretesting
i. Talking or writing on the level of your audience (children or people w/out an education) – comprehension level
f. Interview
advantages of Self-administered Questionnaires
i. Cheaper and faster than face-to-face interviews
ii. National is the same cost as local mailings
iii. Requires small staff
iv. More willingness to answer controversial items
advantages of interview surveys
i. Fewer incomplete questionnaires
ii. More effective for complicated questionnaires
iii. Face-to-face is more intimate
advantage of telephone surveys
i. Cheaper and more time efficient
Strengths and weakness of survey research
a. Strengths
i. Useful in describing large populations
ii. Surveys are flexible
iii. Standardized questions (same set of questions for each individual)
b. Weaknesses
i. Round pegs in square holes
ii. Seldom deal with context of social life
iii. Inflexible
iv. Artificial
v. Weak on validity (high in reliability)
Kansas Health Foundation
i. Purpose to improve health of children in state
ii. Tried to encourage adults to improve their social relationships with children
iii. Track and trend survey approach
Types of Evaluation Research Design
a. Experimental designs
b. Quasi-experimental designs – nonrigorous inquiries somewhat resembling controlled experiments but lacking key elements such as pre- and post-testing and/or control groups
c. Randomized controlled trails
i. Treatment
ii. Placebo
iii. Blind versus double-blind
d. Qualitative evaluations
Social indicators
measurements that reflect the quality or nature of social life. Social indicators are often monitored to determine the nature of social change in society.
i. Decrease people in the poverty line
Sampling frame
a list of units that compose a population from which a sample is selected
Experiments
i. Pros
1. Isolation of experimental variable’s impact over time
2. Replication
ii. Cons
1. Not representative
2. Artificiality of laboratory settings
Equal groups in experimental designs
a. Probability sampling rare in experimental designs
b. Instead use logic of random assignment when assigning subjects to groups
c. (Self-selection) preference must be random
d. Randomization: techniques for assigning experimental subjects
e. Matching: pairs of subjects matched on basis of their similarities on one or more variables.
One study by Albert Bandura et al
a. Goal: to study patterns of behavior associated with aggression
b. Social learning theory: aggressive behavior is learned through observing and imitating others
Terms in all types of experimental designs
a. IV → the cause
i. Part of experiment that can be controlled or changed
ii. Stimulus/intervention
b. DV → the effect
i. What is the main “measure” of the study?
c. By manipulating an independent variable (the cause), the researcher determines whether it influences the dependent variable (the effect).
d. Pre-testing & post-testing
Experiments involve:
a. Taking action
b. Observing consequences of that action
Experimental method
a. Appropriate for:
i. Limited and well-defined concept
ii. Hypothesis testing
iii. Determining causality
b. Laboratory vs. natural (of field) settings
Examples of Experimental designs in the real world
a. Randomized controlled data
b. Does drug X help alleviate depression?
i. The people getting the treatment should have the depression decreased if they take the Drug X every day.
c. Can a school-based tobacco prevention program reduce tobacco initiation among high school students?
Evaluation research
research undertaken for the purpose of determining the impact of some social intervention, such as a program aimed at solving a social problem
Need assessment studies
studies that aim to determine the existence an extent of problems, typically among a segment of the population
External validity
i. Conclusions drawn from experimental results may not be generalizable to the “real” world
ii. Pertains to the representativeness of the study, sample, setting, and procedures
Threats to internal validity
a. Testing
b. History
i. Sensational crimes reported
c. Maturation
i. Long-term experiment: subjects are aging
ii. Short-term experiment: boredom, tired
d. Experimental mortality
i. When people decide to drop out of doing an experiment
ii. People move and we lose their contact information
e. Diffusion or imitation of treatments: “Contaminated” control groups
f. Compensation
i. Researcher/assistants inadvertently compensate fort he control group
ii. Use double-blind experiment
g. Instrumentation: Use of different instrument in pretesting and post testing
Threats to External Validity
- Subjects: is sample representative of (or even similar to) population?
- Setting or context: are they similar enough to the setting/ context to which you want to generalize?
- Hawthorne Effect:
- John Henry Effect (in which control group tries harder)
- Experimenter effects (related to ethnicity, gender, personal attributes)
- Rosenthal Effect (subjects perform in accordance with the researcher’s expectations
- Pre-test sensitization (can heighten subject’s sensitivity to treatment)
- Selection-treatment interference (some trait of participants interacts with treatment)
Review of Behavioral sciences
a. Domains include visual perception, fairness, cooperation, spatial reasoning, categorization, inferential induction, moral reasoning, reasoning styles, self-concepts, motivation, and heritability of IQ
b. There is substantial variability in experimental results across populations
c. WEIRD subjects are particularly unusual compared with the rest of the species – frequent outliers.
random sampling
allow a researcher to make relatively few observations and generalize from those observations to a much wider populaion
quota sampling
is based on a knowledge of the characteristics of the population being sampled: what proportion are men, what portion aare women, what proportion are of various incomes, ages, and so on
function of stratification
organize the population into homogenous subsets (with heterogeneity between subsets) and to select
convenience sampling
relies on available participants, such as stopping people at a street corner or some other location
snowball sampling
the procedure is implemented by collecting data on the few members of the target population you can locate and then asking those individuals to provided the information needed to locate other members of that population who they happen to know.
EPSEM samples
a sample will be representative of the population from which it is selected if all members of the population have an equal chance of being selected
element
unit about which information is collected and that provides the basis of analysis
population
the theoretical specified aggregation of study elements
observation unit
or a unit of data collection, is an element or aggregation of elements from which information is collected
variable
a set of mutually exclusive attributes: sex, age, employment status, and so forth
parameter
the summary description of a given variable in a population
statistic
the summary description of a given variable in a sample
steps of survey research:
1) development of a questionnaire or survey
2) selection of a sample of participants
3) administration of the questionnaire through any of several modes, including self-administration, face-to-face interviews, telephone interviews, and use of new technologies
the challenge of asking good questions
1) factual questions sometimes elicit questionable answers
2) what people say they do, or will do, and what they actually do are not always related
3) respondents’ attitudes, beliefs, habits, and interests can appear highly unstable
4) small changes in wording sometimes result in dramatically different responses
5) respondents can misinterpret what questions are asking
6) the order in which questions are asked can affect respondents’ answers
7) changes in question format can affect respondents answers
8) respondents often answer questions when they know little about the topic.
9) questions can have cultural bias that can result in different responses from people of varying cultural backgrounds
composite measure
the use of several questions to measure a given variable
index
constructed through simple accumulation of scores assigned to individual attributes
scale
constructed through the assignment of scores to patterns of responses, recognizing that some items reflect a relatively weak degree of the variable while others reflect something stronger
- generally superior than index because it takes into consideration the intensity with which different items reflect the variable being measured
Likert-type scale
questionnaire item containing response categories of “strongly agree”, “agree”, “neither agree nor disagree”, “disagree”, and “strongly disagree”
semantic differential
a questionnaire format in which the participant is asked to rate something in terms of two, opposite adjectives
e.g: rate textbooks as “boring” or “exciting”
Thurstone equal-appearing interval scales
a type of composite measure, constructed in accord with the weights assigned by “judges” to various indicators of some variables
Guttman Scolograms
a type of composite measure used to summarize several discrete observations and to represent some more general variable. the items in a Guttman scale form a conceptual hierarchy
comparative judgements
judgements in which people are asked to evaluate two or more phenomena in a direct comparison of some kind
response rate
one guide to representativeness of the sample respondents. if a high response rate is achieved, there’s less chance of significant problem with a biased sample than in a low rate
general rules for survey interviewing
1) appearance and demeanor
2) familiarity with survey protocol
3) following question wording exactly
4) recording responses exactly
specification in survey research
prepared in conjunction with the protocol
explanatory and clarifying comments about handling difficult or confusing situations that may occur with regard to specific questions
III. In class, we discussed the evaluation of the Kansas Health Foundation. To indicate levels of campaign exposure and outcome variables, it relies on telephone interviews with unique individuals, which were conducted at multiple points in time. What’s true?
a. It’s evaluation was quantitative
b. The campaign aimed to improve youth health
c. It involves measures of social capital
d. It relied on longitudinal survey methods
IV. An increase in sample size brings about what?
a. Increase in statistical power and decrease in sampling error
I. Researchers aim to study media use among low-educated Americans. They are using mail survey and are concerned with readability levels of the audience. What should the researchers do?
a. Pretest the questions on a sample of low-educated Americans
VII. Which of the following is NOT true of survey methods?
a. Useful in describing large populations
b. Provide flexibility
c. Permit using standardized questions
d. Best method for demonstrating causation
e. With randomization, allow for creating representative samples
d. Best method for demonstrating causation
VIII. ________questions have a respondent select an from a list provided
d. Closed-ended
IX. As a general rule, a questionnaire should be____
a. Spread out
b. Uncluttered
c. Relevant
d. Clear
I. In an experimental study, we find that participants in our control group have learned about the stimulus in our treatment group. This exemplifies which of the following threats to internal validity?
b. Diffusion or imitation of treatments
XI. A study has a randomly selected sample undergrad students at Texas A&M. to whom can the study make proper generalizations
a. Undergrad students at Texas A&M
XII. When conclusions drawn from experimental results may not be generalized to the “real” world, we have an example of a threat to what?
a. External validity
I. In general, survey research is an appropriate observational method for_____
a. Describing a population too large to observe directly
b. Descriptive, exploratory, and explanatory purposes
c. The measurement of attitudes prevalent on a larger population
d. Studies that have individual people as the unit of analysis
XIV. Which question construction guideline does the following questionnaire item most clearly violate? “Would you not say that crime is not a serious problem in the U.S.?”
a. Avoid negative items
XV. After conducting a telephone survey, Dr. Erving compared the demographics of this sample to the demographics of his intended population of interest. Dr. Erving finds that the sample has 62% female respondents, while the population has only 51% female. What type of problem does this represent?
a. Sample bias
XVI. What researcher headed up to the bob doll study that we discussed in class?
a. Albert bandura
XVII. In the figure below on experimental design, the X represents what?
Experiments:
01 X02
01 02
What is the control group?
Treatment stimulus
Control group is the 2nd row with no treatment
I. A researcher finds that her random sample has a much higher mean household in come than population. This would be a problem for what?
a. Representative
II. The size of the population must be taken into account when deciding a sample size.
true or false
false
