Ch. 7 Flashcards
Survey research
A quantitative and qualitative method with two important characteristics; variables are measured using self-reports and considerable attention is paid to the issue of sampling.
respondents
Participants in a survey or study.
Survey research used in non-experimental and experimental
Most survey research is non-experimental. It is used to describe single variables (e.g., the percentage of voters who prefer one presidential candidate or another, the prevalence of schizophrenia in the general population, etc.) and also to assess statistical relationships between variables (e.g., the relationship between income and health).
But surveys can also be used within experimental research.
History and Uses of Survey Research
Survey research may have its roots in English and American “social surveys” conducted around the turn of the 20th century by researchers and reformers who wanted to document the extent of social problems such as poverty.
From market research and election polling, survey research made its way into several academic fields, including political science, sociology, and public health—where it continues to be one of the primary approaches to collecting new data.
Beginning in the 1930s, psychologists made important advances in questionnaire design, including techniques that are still used today, such as the Likert scale.
Survey research has a strong historical association with the social psychological study of attitudes, stereotypes, and prejudice.
Early attitude researchers were also among the first psychologists to seek larger and more diverse samples than the convenience samples of university students that were routinely used in psychology.
been instrumental in estimating the prevalence of various mental disorders and identifying statistical relationships among those disorders and with various other factors.
survey research can even be used as a data collection method within experimental research to test specific hypotheses about causal relationships between variables.
Constructing Surveys
The heart of any survey research project is the survey itself. Although it is easy to think of interesting questions to ask people, constructing a good survey is not easy at all.
The problem is that the answers people give can be influenced in unintended ways by the wording of the items, the order of the items, the response options provided, and many other factors.
At best, these influences add noise to the data.
At worst, they result in systematic biases and misleading results.
A Cognitive Model
Respondents must interpret the question, retrieve relevant information from memory, form a tentative judgment, convert the tentative judgment into one of the response options provided (e.g., a rating on a 1-to-7 scale), and finally edit their response as necessary.
First, they must interpret the question. For example, they must decide whether “alcoholic drinks” include beer and wine (as opposed to just hard liquor) and whether a “typical day” is a typical weekday, typical weekend day, or both.
Found that asking about “typical” behavior has been shown to be more valid than asking about “past” behavior.
Once respondents have interpreted the question, they must retrieve relevant information from memory to answer it.
But what information should they retrieve, and how should they go about retrieving it?
They might think vaguely about some recent occasions on which they drank alcohol, they might carefully try to recall and count the number of alcoholic drinks they consumed last week, or they might retrieve some existing beliefs that they have about themselves (e.g., “I am not much of a drinker”).
Then they must use this information to arrive at a tentative judgment about how many alcoholic drinks they consume in a typical day.
For example, this mental calculation might mean dividing the number of alcoholic drinks they consumed last week by seven to come up with an average number per day.
Then they must format this tentative answer in terms of the response options actually provided.
In this case, the options pose additional problems of interpretation.
For example, what does “average” mean, and what would count as “somewhat more” than average? Finally, they must decide whether they want to report the response they have come up with or whether they want to edit it in some way.
For example, if they believe that they drink a lot more than average, they might not want to report that for fear of looking bad in the eyes of the researcher, so instead, they may opt to select the “somewhat more than average” response option.
context effects
Unintended influences on respondents’ answers because they are not related to the content of the item but to the context in which the item appears.
The response options provided can also have unintended effects on people’s responses.
When people are asked how often they are “really irritated” and given response options ranging from “less than once a year” to “more than once a month,” they tend to think of major irritations and report being irritated infrequently.
But when they are given response options ranging from “less than once a day” to “several times a month,” they tend to think of minor irritations and report being irritated frequently.
People also tend to assume that middle response options represent what is normal or typical.
So if they think of themselves as normal or typical, they tend to choose middle response options.
item-order effect
When the order in which the items are presented affects people’s responses.
One item can change how participants interpret a later item or change the information that they retrieve to respond to later items.
Open-ended items
Simply ask a question and allow participants to answer in whatever way they choose.
Open-ended items are useful when researchers do not know how participants might respond or when they want to avoid influencing their responses.
Open-ended items are more qualitative in nature, so they tend to be used when researchers have more vaguely defined research questions—often in the early stages of a research project.
Open-ended items are relatively easy to write because there are no response options to worry about.
However, they take more time and effort on the part of participants, and they are more difficult for the researcher to analyze because the answers must be transcribed, coded, and submitted to some form of qualitative analysis, such as content analysis.
Another disadvantage is that respondents are more likely to skip open-ended items because they take longer to answer.
It is best to use open-ended questions when the answer is unsure or for quantities which can easily be converted to categories later in the analysis.
Closed-ended items
Questionnaire items that ask a question and provide a limited set of response options for participants to choose from.
Closed-ended items are used when researchers have a good idea of the different responses that participants might make.
They are more quantitative in nature, so they are also used when researchers are interested in a well-defined variable or construct such as participants’ level of agreement with some statement, perceptions of risk, or frequency of a particular behavior.
Closed-ended items are more difficult to write because they must include an appropriate set of response options.
However, they are relatively quick and easy for participants to complete.
They are also much easier for researchers to analyze because the responses can be easily converted to numbers and entered into a spreadsheet.
For these reasons, closed-ended items are much more common.
All closed-ended items include a set of response options from which a participant must choose.
For categorical variables like sex, race, or political party preference, the categories are usually listed and participants choose the one (or ones) to which they belong.
For quantitative variables, a rating scale is typically provided.
rating scale
An ordered set of responses that participants must choose from.
five and seven are probably most common.
Five-point scales are best for unipolar scales where only one construct is tested, such as frequency (Never, Rarely, Sometimes, Often, Always).
Seven-point scales are best for bipolar scales where there is a dichotomous spectrum, such as liking (Like very much, Like somewhat, Like slightly, Neither like nor dislike, Dislike slightly, Dislike somewhat, Dislike very much).
For closed-ended items, it is also important to create an appropriate response scale.
For categorical variables, the categories presented should generally be mutually exclusive and exhaustive.
Mutually exclusive categories do not overlap.
Exhaustive categories cover all possible responses.
In many cases, it is not feasible to include every possible category, in which case an Other category, with a space for the respondent to fill in a more specific response, is a good solution.
If respondents could belong to more than one category (e.g., race), they should be instructed to choose all categories that apply.
Branching
For bipolar questions, it is useful to offer an earlier question that branches them into an area of the scale; if asking about liking ice cream, first ask “Do you generally like or dislike ice cream?”
Once the respondent chooses like or dislike, refine it by offering them relevant choices from the seven-point scale.
Branching improves both reliability and validity.
Although you often see scales with numerical labels, it is best to only present verbal labels to the respondents but convert them to numerical values in the analyses.
Avoid partial labels or length or overly specific labels.
In some cases, the verbal labels can be supplemented with (or even replaced by) meaningful graphics.
BRUSO
An acronym that stands for “brief,” “relevant,” “unambiguous,” “specific,” and “objective,” which is used to create effective questionnaire items that are brief and to the point.
This brevity makes them easier for respondents to understand and faster for them to complete. Effective questionnaire items are also relevant to the research question.
If a respondent’s sexual orientation, marital status, or income is not relevant, then items on them should probably not be included.
Effective questionnaire items are also unambiguous; they can be interpreted in only one way.
Part of the problem with the alcohol item presented earlier in this section is that different respondents might have different ideas about what constitutes “an alcoholic drink” or “a typical day.”
Effective questionnaire items are also specific so that it is clear to respondents what their response should be about and clear to researchers what it is about.
A common problem here is closed-ended items that are “double barrelled.” They ask about two conceptually separate issues but allow only one response.
effective questionnaire items are objective in the sense that they do not reveal the researcher’s own opinions or lead participants to answer in a particular way.
The best way to know how people interpret the wording of the question is to conduct a pilot test and ask a few people to explain how they interpreted the question.
every survey should have a written or spoken introduction that serves two basic functions
• One is to encourage respondents to participate in the survey.
In many types of research, such encouragement is not necessary either because participants do not know they are in a study (as in naturalistic observation) or because they are part of a subject pool and have already shown their willingness to participate by signing up and showing up for the study.
• The second function of the introduction is to establish informed consent.
important that this part of the introduction be well documented and presented clearly and in its entirety to every respondent.
introduction should be followed by the substantive questionnaire items.
But first, it is important to present clear instructions for completing the questionnaire, including examples of how to use any unusual response scales.
Remember that the introduction is the point at which respondents are usually most interested and least fatigued, so it is good practice to start with the most important items for purposes of the research and proceed to less important items. Items should also be grouped by topic or by type.
Demographic items are often presented last because they are least interesting to participants but also easy to answer in the event respondents have become tired or bored.
Of course, any survey should end with an expression of appreciation to the respondent.
Probability sampling
Occurs when the researcher can specify the probability that each member of the population will be selected for the sample.
Non-probability sampling
Occurs when the researcher cannot specify the probability that each member of the population will be selected for the sample.
Convenience sampling
A common method of non-probability sampling in which the sample consists of individuals who happen to be easily available and willing to participate (such as introductory psychology students).
snowball sampling
A form of non-probability sampling in which existing research participants help recruit additional participants for the study.
quota sampling
A form of non-probability sampling in which subgroups in the sample are recruited to be proportional to those subgroups in the population.
self-selection sampling
A form of non-probability sampling in which individuals choose to take part in the research on their own accord, without being approached by the researcher directly.
sampling frame
A list of all the members of the population from which to select the respondents.
Sampling frames can come from a variety of sources, including telephone directories, lists of registered voters, and hospital or insurance records.
In some cases, a map can serve as a sampling frame, allowing for the selection of cities, streets, or households.
Simple random sampling
A probability sampling method in which each individual in the population has an equal probability of being selected for the sample.
stratified random sampling
A common alternative to simple random sampling in which the population is divided into different subgroups or “strata” (usually based on demographic characteristics) and then a random sample is taken from each “stratum.”
Proportionate stratified random sampling
Is used to select a sample in which the proportion of respondents in each of various subgroups matches the proportion in the population.
Disproportionate stratified random sampling
Is used to sample extra respondents from particularly small subgroups—allowing valid conclusions to be drawn about those subgroups.
cluster sampling
A type of probability sampling in which larger clusters of individuals are randomly sampled and then individuals within each cluster are randomly sampled.
This is the only probability sampling method that does not require a sampling frame.
How large does a survey sample need to be?
In general, this estimate depends on two factors.
One is the level of confidence in the result that the researcher wants.
The larger the sample, the closer any statistic based on that sample will tend to be to the corresponding value in the population.
The other factor is a practical constraint in the form of the budget of the study.
Larger samples provide greater confidence, but they take more time, effort, and money to obtain.
Taking these two factors into account, most survey research uses sample sizes that range from about 100 to about 1,000.
Sampling bias
Occurs when a sample is selected in such a way that it is not representative of the entire population and therefore produces inaccurate results.
non-response bias
Occurs when there is a systemic difference between survey non-responders from survey responders.
Although there are methods for statistically correcting for non-response bias, they are based on assumptions about the non-responders—for example, that they are more similar to late responders than to early responders—which may not be correct.
For this reason, the best approach to minimizing non-response bias is to minimize the number of non-responders—that is, to maximize the response rate.
In-person interviews have the highest response rates and provide the closest personal contact with respondents.
Some Preconceptions and Findings Pertaining to Web-based Studies
Preconception
Internet samples are not demographically diverse
Finding
Internet samples are more diverse than traditional samples in many domains, although they are not completely representative of the population
Preconception
Internet samples are maladjusted, socially isolated, or depressed
Finding
Internet users do not differs from nonusers on markers of adiustment and depression
Preconception
Internet-based findings differ from those obtained with other methods
Finding
Evidence so far suggests that internet-based findings are consistent with findings based on traditional methods (e.g., on self-esteem, personality), but more data are needed.
