Week 5: Survey Research Flashcards
1
Q
Qualitative and qualitative method with two important characteristics :
- Variables are measured using self-reports
- Considerable attention is paid to sampling
A
Survey research
2
Q
Participants in a survey or study
A
Respondents
3
Q
Where is random sampling routinely used?
A
Survey research
4
Q
Why are large and random samples preferred in survey research?
A
Most accurate estimate for what is true in population
5
Q
Why is survey research mostly non-experimental?
A
- Describe single variables
2. Assess a single relationship between variables
6
Q
What are the three ways people can be influenced in survey responses (unintended)
A
- Wording of items
- Order of items
- Response options provided
7
Q
How can survey design produce misleading results?
A
Systematic bias in the design
8
Q
Question interpretation > Information retrieval > Judgement formation > Response formatting > Response editing
A
Model of Cognitive Process Involved in Responding to a Survey Item
9
Q
Unintended influences on respondents answers because they are not related to the content of the item, but the context in which the item appears
A
Context effects
10
Q
When the order in which the items are presented affects people’s responses
A
Item Order Effect
11
Q
Two unintended effects of response options:
A
- Frequency can make people only think of major/minor instances
- Middle option assumed to be normal/typical
12
Q
How to mitigate order effect?
A
Rotate questions and response items when there is a natural order.
13
Q
Allowing particpants to answer in whichever way they choose
A
Open ended items
14
Q
What are the features of open ended items?
A
- Useful when do not know how a participant may respond
- Useful when don’t want to influence the response
- Qualitative in nature, useful for early stages of the project
- Take more time/effort for respondent
- Often skipped by respondent
15
Q
When is open ended item best used?
A
- When the answer is unsure
2. Quantities which can easily be converted to categories later in the analysis
16
Q
Ask a question and provide a limited set of response options for particpants
A
Close Ended item
17
Q
What are the features of close ended items?
A
- Used when researchers have good idea of different responses participants might use
- Quantitative in nature, used for well-defined variable or construct
- More difficult to write, but quick and easy for respondent to answers
- Easier to analyse as responses easily converted to numbers
18
Q
Which type of item is more common?
A
Close ended items is more common than open ended items
19
Q
Ordered set of responses that partipants must choose from
A
Rating scale
20
Q
What are the features of rating scales?
A
- Usually quantitative variables
- Usually 3-11 response options, 5-7 is more common
- 5 point scales best for unipolar scales where only one construct is tested (eg. Frequency).
- 7 point scales best for bipolar scales where there is a dichotomous spectrum (eg. Like to Dislike)
- Branching improves both variability and reliability
21
Q
Likert Scale
A
- 1930s developed new approach for measuring peoples attitudes
- Strongly disagree to Strongly Agree
- Numbers assigned to each response, summed to provide a score representing the attitude
22
Q
What is BRUSO?
A
Brief, Relevant, Unambiguous, Specific, Objective
- used to create effective questionnaire items that are brief and to the point
23
Q
What does leaving out the middle neutral option do?
A
Creates unbalanced survey design
24
Q
Survey introduction
A
- Need written or spoken introduction to”
1. Encourage particpation
2. Establish informed consent
25
Occurs when the researcher can specify the probability that each member of the population will be selected for the sample
Probability Sampling
26
Occurs when the researcher cannot specify the probability that each member of the population will be selected for the sample
Non-probability sampling
27
What sampling does most psychological research use?
Non-probability sampling
28
Common method of non-probability sampling in which the sample consists of individuals who happen to be easily available and willing to participate (e.g. undergrad students)
Convenience sampling
29
Existing research participants help recruit additional participants for the study
Snowball Sampling
30
Subgroups in the sample are recruited to be proportional these subgroups in the population
Quota sampling
31
Individuals choose to take part in the research of their own accord, without being approached by the researcher directly.
Self-Selection Sampling
32
Why are survey researchers more likely to use probability sampling?
So can make estimates of what is true for the population
33
A list of all the members of the population from which to select the respondents
Sampling frame
34
Sampling frame sources:
1. telephone directory
2. list of registered voters
3. hospital records
35
What are the different sampling methods:
- Simple random sampling
- Stratified random sampling
- Proportionate Stratified random sampling
- Disproportionate stratified random sampling
- Cluster sampling
36
Each individual in the population has an equal probability for being selected for the sample
Simple random sampling
37
Common alternative to simple random sampling, where the population is divided into subgroups/strata and random sample taken from each
Stratified random sampling
38
Select sample in which the proportion of respondents in each various subgroups matches the proportions in the population
Proportionate Stratified Random Sampling
39
used to sample extra respondents from particularly small subgroups - allowing valid conclusions to be drawn about these subgroups
Disproportionate stratified random sampling
40
Large clusters of individuals are randomly sampled and then individuals within each cluster are randomly sampled
Cluster sampling
41
What is the only probably sampling method that does not require a sampling frame?
Cluster sampling
42
What survey research sample sizes are most common?
100 to 1000
43
What does conducting a power analysis prior to launching the survey help the researcher do?
Guides researcher in making the sample/resources trade off
44
Why is a sample size over 1000 not considered worth the extra resources?
Only small increase in confidence interval over 1000 sample size
45
When does sampling bias in survey research occur?
When sample is selected in a way that is not representative of the entire population and thus produces inaccurate results
46
Why was probability sampling developed?
To deal with sampling bias
47
Occurs when there is a systemic difference between survey non-respondents and survey respondents
Non-Response bias
48
How do you minimise non-response bias?
Maximise the response rate (follow up reminders, simple short survey, incentives)
49
What are the four main ways to conduct a survey?
1. In person interviews
2. Telephone
3. Mail
4. Internet
50
Set of techniques for summarising and displaying data
Descriptive statistics
51
The way scores are distributed across levels of a variable (e.g. 44 have score 'male' and 56 have score 'female' for 100 sample)
Distribution
52
Display of each value of a variable and the number of participants with that value
Frequency tables
53
Features of frequency tables
- can quickly see range, most and least common and extreme scores
- includes scores only included in the data set
- can be used for category labels, with most frequent score at the top
54
Group scores into equal ranges, usually 5-15 per group
Grouped Frequency Table
55
Graphic display of a distribution
Histogram
56
When do bars in a histrogram have gaps?
When the data is categorical (not quantitative)
57
One distinct peak in a graph is called
Unimodal shape
58
Two distinct peak in a graph is called
Bimodal shape (more than two peaks in uncommon in psychology)
59
Extreme score that is much higher or lower than the rest of the scores in the distribution
Outlier
60
The middle of a distribution, the point around which the scores in the distribution tend to cluster
Central tendency (aka average)
61
Sum of scores divided by number of scores
Mean
62
The middle scores (average of two middle scores if even number of scores)
Median
63
Most frequent score, can also be used for categorical variables
Mode
64
Skewed shape central tendency
- Can be positive or negative
| - mean will differ from the median in direction of the longer tail
65
Unimodal shape central tendency
Mean, median and mode very close to each other at peak
66
Bimodal shape central tendency
Mean and median tend to be between peaks, mode will be tallest peak
67
The extent to which the scores vary around the central tendency in a distribution
Variability
68
Measure of dispersion that measures the distance between the highest and lowest scores
RANGE
- 25-15 = range of 9
- misleading when there are outliers
69
the average distance between scores and the mean in a distribution
STANDARD DEVIATION
- most common measure of variability
- "the scores in the distribution differ from the mean by about (SD ) on average"
- is always postive amount
70
A measurement of the average distance of scores from the mean
VARIANCE/MEAN OF SQUARED DIFFERENCES
| = SD Squared
71
Why use N-1 in formulas for samples?
Standard deviation of a sample tends to be a bit lower than the standard deviation of the population the sample was selected from
N-1 correct this tendency and gives a better estimate of the population standard deviation
72
For any given scores, the percentage of scores in the distribution that are lower than that score (e.g. if percentile rank on test was 40, you scored higher than 40% of the people who did the test)
Percentile rank
73
Difference between that individual's scores and the mean of the distribution, divided by the SD of the distribution
Z score
74
Why is Zscore used?
- Represents the number of standard deviations the score is from the mean
- Can help define outliers (e.g. z score is less than -3.00 or great than +3.00)
75
How are differences between groups of conditions described?
Mean and SD
76
Describes the strength of a statistical relationship
Effect Size (e.g. Cohens d or Peasron's r)
77
Difference between the two means divided by the standard deviation
Cohen's D
78
Features of Cohen's d
- Should always be positive
- Difference between the two group means, expressed in SD units
- Can combine/compare results across studies using different measures
79
Relationship between two variables whereby the points on a scatterplot fall close to a single straight line
Linear relationship
80
Relationships between two variables in which the points on a scatterplot do not fall close to a single straight line, but often fall alone a curved line
Non-liner relationship
81
Strength of a correlation between quantitative variables
Pearson's r
82
What are the features of Pearson's r?
- -1.00 to +1.00 value
- Score of 0 means there is no relationship between the variables
- Type of effect size
- Correlation does not equal causation
83
Difference between Cohen's D and Pearson's R
Cohen's d measures the SIZE of the difference between two GROUPS while Pearson's r measures the STRENGTH of the RELATIONSHIP between two VARIABLES
84
When can Pearson's r be misleading?
- Relationship in study is non-linear (no straight line)
| - When one/both variables have limited range in the sample relative to population (restriction of range issue)
85
APA Descriptive Statistics:
- use two decimal places
- Statistics written as numbers (not the word 'two')
- (M= 2.15, SD=2.27), do not write words mean or standard deviation within parenthesis
- "test restest correlation was .96, (r=-.27)"
86
Bars that represent the variability in each group/condition
Error bars (APA)
87
The SD of the group divided by the square root of the sample size of the group
Error extends one standard error in each direction usually
Standard error (APA)
88
Compare mean scores, use when the X axis is categorical (e.g. January, Feb etc)
Bar Graphs (APA)
89
When IV is measured in a continuous manner (e.g. time), use when the X axis is quantitative (e.g 1 min, 2 min)
Line graphs (APA)
90
4 Steps for conducting data analysis:
1. Do not include identifying information
2. Check raw data is complete and accurate
3. Create data file (e.g. Excel or SPSS)
4. Conduct preliminary analysis
(5. Do planned and exploratory analysis)
91
What happens in preliminary analysis of data?
- Assess internal consistency of the measure using Cronbach a/x)
- Analyse each important variable separately with histograms, central tendencies (mean etc) and variability (range, SD, variance)
- Identify outliers and decide to discard or maybe include but then use median for central tendency
92
Why do we do planned and exploratory analysis?
Answering primary research question, testing your data for the relationship expected in the hypothesis
93
How to do planned analysis?
- Investigate expected difference between means and SD of groups = use bargraph and compute Cohen's d
- Investigate expected correlation between variables = line graphs or scatterplot and computer Pearson's r
94
How to do exploratory analysis?
- Analysis undertaken without an existing hypothesis, may be relationships that you did not hypothesise
- differentiate from planned analysis when writing a report
- explain unexpected relationship maybe worthy of additional research, be skeptical as it will need to be replicated
95
A type of study designed specifically to answer the question of whether there is a CAUSAL relationship between two variables
Experiment
96
When a variable (IV) changes, it causes a change in another variable (DV)
Experiment
97
Two features of an experiment
1. Manipulate IV levels (conditions)
| 2. Control of variability in extraneous variables (control = holding constant)
98
Changing the level or condition of the IV systematically so that different groups of participants are exposed to a different level of that variable
OR
the same group of participants is exposed to different levels at different times
Manipulation of IV
99
an experiment design involving a single independent variable with two conditions
single factor two level design
100
an experimental design involving a single independent variable that is manipulated to produce more than two conditions
single factor multi level design
101
How to control extraneous variables?
- conditions of the experiment being same
| - limit participants to a specific category of persons (lowers external validity though
102
Why do we control extraneous variables?
Otherwise difficult to detect the effect of the IV
1. Adding variability or noise to data (hard to detect IV)
2. Becoming confounding variables (varies systematically with the IV)
103
Any intervention meant to change peoples behaviour for the better
TREATMENT
104
the condition in which participants receive the treatment
TREATMENT CONDITION
105
the condition in which participants do not receive the treatment
CONTROL CONDITION
106
an experiment that researches the effectiveness of psychotherapies and medical treatments
Random clinical trials
107
the condition in which participants receive no treatment whatsoever
NO-TREATMENT CONDITION
108
What does the placebo effect implicate?
The no-treatment condition (participants expecting to get better due to treatment)
109
Condition in which participants receive a placebo rather than the treatment
Placebo Control Condition
110
Condition in which participants are told they will receive treatment but must wait until the participants in the treatment condition have received it
Wait-List control condition
111
How to deal with placebo?
Give current best known treatment vs. the new treatment
112
each participant is tested on only one condition
| participants in groups are on average highly similar
Between Subject Experiment
113
each participant is tested under all conditions
Within Subject Experiment
114
Features of Within-subject experiment
- Provide minimum control of extraneous participant variables
- easier to reduce data noise and see effect of IV
115
Using a random process to decide which participants are tested in which condition (strength in design)
Random Assignment
116
Two criteria for Random Assignment
1. Each participant has equal chance of being assigned to a condition
2. Participant is assigned to a condition independently of other participants
117
All the conditions occur once in the sequence before any of them is repeated
Block Randomisation
118
Block Randomisation benefits
- Good for equal-sized groups (coin flip can't produce this)
119
Random assignment weakness
- Not guaranteed to control all extraneous variables across conditions
- However....Inferential statistics takes the liability of random assignment into account
- Confounds likely to be detected when the experiment is replicated.
120
An experiment design in which the participants in the various conditions are matched on the dependent variable or on some extraneous variable/s prior to the manipulation of the IV
(Alternative to random assignment)
Matched-groups design
121
An effect that occurs when participants responses to the various conditions are affected by the order of conditions in which they were exposed (e.g. tired, bored, practice) --- Within Subjects Design
Order Effect
122
An effect of being tested in one condition, on participants behaviour in later conditions
Carry Over Effect
123
Participants perform task better in later conditions due to the chance to practice it
Practice Effect
124
Participants perform task worse in later conditions as have become tired or bored
Fatigue Effect
125
Unintended influences on respondents answers because they are not related to the content of the item but the context in which the item appears
Contxt/Contract effect
126
Solution to Order effects (within subjects design)
Counterbalancing!
127
Testing different participants in different orders
Counterbalancing
128
Equal number of participants complete each possible order of conditions
- best method
- use random assignment to allocate to diff orders of conditions
Complete Counterbalancing
129
Partial Counterbalancing
Latin square design (randomises through equal rows and columns)
130
Order of the conditions is randomly determined for each participant
Random Counterbalancing
131
Features of random counterbalancing
- use when the number of conditions is large
- will result in more random error
- if order effects likely to be small, and number of conditions are large then it is an option
132
What does counterbalancing accomplish?
1. Controls order of conditions so that is no longer a confounding variable
2. Makes it possible to detect carry over effects (analyse data separately for each order to see whether it had an effect)
133
Single list with both conditions in it, tested altogether
Simultaneous Within-Subjects Design
134
Which subject design is usually best?
Within Subject design
- controls extraneous participant variables, reduces data noise
- easier to detect any effect of IV upon DV
- required fewer participants
135
What are features of Between Subjects Design
- Conceptually simpler with less testing time per participant
- Avoid carry-over effects without need for counterbalancing
- Possibly better when treatment produces long term change in participants behaviour
136
Refers to the degree in which we can confidently infer a causal relationship between the variables
Internal Validity
137
Why are experimental studies high in internal validity?
Because IV manipulation provides strong support for causal conclusions
138
Refers to degree to which we can generalise findings to other circumstances/settings or the population
External validity
139
When the participants and the situation studied are similar to those that the researchers want to generalise to and participants encounter everyday
Mundane Realism
140
Where the mental process is used in both the laboratory and in the real world
Psychological realism
141
Why are experiments no always low in external validity?
- Experiments need not seem artifical, or consider field experiments
- often conducted to learn about psychological processes present in a variety of people and situations
142
One of the big four validities, whereby the research question is clearly operationalised by the studys methods
- Adding more conditions does not necessarily increase construct validity
Construct Validity
143
The specification of exactly how the research question will be studied in the experiment design
Operationalisation
144
Concerns the proper statistical treatment of data and the soundness of the researchers statistical conclusions (e.g. t test, ANOVA, regression, correlation)
Statistical Validity
145
What are the features of Statistical Validity
- Must consider scale of measure of DV and study design
- Many inferential statistics carry certain assumptions, statistical validity is threatened when these assumptions are not met but the statistics are used anyway
- Small sample size is a statistical validity critique
- Important to conduct a power analysis when designing study (informs number of participants to recruit)
146
Verifying the experimental manipulation worked, by using a different measure of the construct the researcher is trying to manipulate
Manipulation Check
147
Features of Manipulation Check
- used to confirm that the IV was infact successfully manipulated (for indirect IV like stress levels)
- important when results of experiment turn our null (decide if there is a real absence of effect of IV on DV OR is null result is due to problem with manipulation of IV)
- Usually done at the end of procedure, good to include in pilot test
