Week 5: Survey Research Flashcards
Qualitative and qualitative method with two important characteristics :
- Variables are measured using self-reports
- Considerable attention is paid to sampling
Survey research
Participants in a survey or study
Respondents
Where is random sampling routinely used?
Survey research
Why are large and random samples preferred in survey research?
Most accurate estimate for what is true in population
Why is survey research mostly non-experimental?
- Describe single variables
2. Assess a single relationship between variables
What are the three ways people can be influenced in survey responses (unintended)
- Wording of items
- Order of items
- Response options provided
How can survey design produce misleading results?
Systematic bias in the design
Question interpretation > Information retrieval > Judgement formation > Response formatting > Response editing
Model of Cognitive Process Involved in Responding to a Survey Item
Unintended influences on respondents answers because they are not related to the content of the item, but the context in which the item appears
Context effects
When the order in which the items are presented affects people’s responses
Item Order Effect
Two unintended effects of response options:
- Frequency can make people only think of major/minor instances
- Middle option assumed to be normal/typical
How to mitigate order effect?
Rotate questions and response items when there is a natural order.
Allowing particpants to answer in whichever way they choose
Open ended items
What are the features of open ended items?
- Useful when do not know how a participant may respond
- Useful when don’t want to influence the response
- Qualitative in nature, useful for early stages of the project
- Take more time/effort for respondent
- Often skipped by respondent
When is open ended item best used?
- When the answer is unsure
2. Quantities which can easily be converted to categories later in the analysis
Ask a question and provide a limited set of response options for particpants
Close Ended item
What are the features of close ended items?
- Used when researchers have good idea of different responses participants might use
- Quantitative in nature, used for well-defined variable or construct
- More difficult to write, but quick and easy for respondent to answers
- Easier to analyse as responses easily converted to numbers
Which type of item is more common?
Close ended items is more common than open ended items
Ordered set of responses that partipants must choose from
Rating scale
What are the features of rating scales?
- Usually quantitative variables
- Usually 3-11 response options, 5-7 is more common
- 5 point scales best for unipolar scales where only one construct is tested (eg. Frequency).
- 7 point scales best for bipolar scales where there is a dichotomous spectrum (eg. Like to Dislike)
- Branching improves both variability and reliability
Likert Scale
- 1930s developed new approach for measuring peoples attitudes
- Strongly disagree to Strongly Agree
- Numbers assigned to each response, summed to provide a score representing the attitude
What is BRUSO?
Brief, Relevant, Unambiguous, Specific, Objective
- used to create effective questionnaire items that are brief and to the point
What does leaving out the middle neutral option do?
Creates unbalanced survey design
Survey introduction
- Need written or spoken introduction to”
1. Encourage particpation
2. Establish informed consent
Occurs when the researcher can specify the probability that each member of the population will be selected for the sample
Probability Sampling
Occurs when the researcher cannot specify the probability that each member of the population will be selected for the sample
Non-probability sampling
What sampling does most psychological research use?
Non-probability sampling
Common method of non-probability sampling in which the sample consists of individuals who happen to be easily available and willing to participate (e.g. undergrad students)
Convenience sampling
Existing research participants help recruit additional participants for the study
Snowball Sampling
Subgroups in the sample are recruited to be proportional these subgroups in the population
Quota sampling
Individuals choose to take part in the research of their own accord, without being approached by the researcher directly.
Self-Selection Sampling
Why are survey researchers more likely to use probability sampling?
So can make estimates of what is true for the population
A list of all the members of the population from which to select the respondents
Sampling frame
Sampling frame sources:
- telephone directory
- list of registered voters
- hospital records
What are the different sampling methods:
- Simple random sampling
- Stratified random sampling
- Proportionate Stratified random sampling
- Disproportionate stratified random sampling
- Cluster sampling
Each individual in the population has an equal probability for being selected for the sample
Simple random sampling
Common alternative to simple random sampling, where the population is divided into subgroups/strata and random sample taken from each
Stratified random sampling
Select sample in which the proportion of respondents in each various subgroups matches the proportions in the population
Proportionate Stratified Random Sampling
used to sample extra respondents from particularly small subgroups - allowing valid conclusions to be drawn about these subgroups
Disproportionate stratified random sampling
Large clusters of individuals are randomly sampled and then individuals within each cluster are randomly sampled
Cluster sampling
What is the only probably sampling method that does not require a sampling frame?
Cluster sampling
What survey research sample sizes are most common?
100 to 1000
What does conducting a power analysis prior to launching the survey help the researcher do?
Guides researcher in making the sample/resources trade off
Why is a sample size over 1000 not considered worth the extra resources?
Only small increase in confidence interval over 1000 sample size
When does sampling bias in survey research occur?
When sample is selected in a way that is not representative of the entire population and thus produces inaccurate results
Why was probability sampling developed?
To deal with sampling bias
Occurs when there is a systemic difference between survey non-respondents and survey respondents
Non-Response bias
How do you minimise non-response bias?
Maximise the response rate (follow up reminders, simple short survey, incentives)
What are the four main ways to conduct a survey?
- In person interviews
- Telephone
- Internet
Set of techniques for summarising and displaying data
Descriptive statistics
The way scores are distributed across levels of a variable (e.g. 44 have score ‘male’ and 56 have score ‘female’ for 100 sample)
Distribution
Display of each value of a variable and the number of participants with that value
Frequency tables
Features of frequency tables
- can quickly see range, most and least common and extreme scores
- includes scores only included in the data set
- can be used for category labels, with most frequent score at the top
Group scores into equal ranges, usually 5-15 per group
Grouped Frequency Table
Graphic display of a distribution
Histogram
When do bars in a histrogram have gaps?
When the data is categorical (not quantitative)
One distinct peak in a graph is called
Unimodal shape
Two distinct peak in a graph is called
Bimodal shape (more than two peaks in uncommon in psychology)
Extreme score that is much higher or lower than the rest of the scores in the distribution
Outlier
The middle of a distribution, the point around which the scores in the distribution tend to cluster
Central tendency (aka average)
Sum of scores divided by number of scores
Mean
The middle scores (average of two middle scores if even number of scores)
Median
Most frequent score, can also be used for categorical variables
Mode
Skewed shape central tendency
- Can be positive or negative
- mean will differ from the median in direction of the longer tail
Unimodal shape central tendency
Mean, median and mode very close to each other at peak
Bimodal shape central tendency
Mean and median tend to be between peaks, mode will be tallest peak
The extent to which the scores vary around the central tendency in a distribution
Variability
Measure of dispersion that measures the distance between the highest and lowest scores
RANGE
- 25-15 = range of 9
- misleading when there are outliers
the average distance between scores and the mean in a distribution
STANDARD DEVIATION
- most common measure of variability
- “the scores in the distribution differ from the mean by about (SD ) on average”
- is always postive amount
A measurement of the average distance of scores from the mean
VARIANCE/MEAN OF SQUARED DIFFERENCES
= SD Squared
Why use N-1 in formulas for samples?
Standard deviation of a sample tends to be a bit lower than the standard deviation of the population the sample was selected from
N-1 correct this tendency and gives a better estimate of the population standard deviation
For any given scores, the percentage of scores in the distribution that are lower than that score (e.g. if percentile rank on test was 40, you scored higher than 40% of the people who did the test)
Percentile rank
Difference between that individual’s scores and the mean of the distribution, divided by the SD of the distribution
Z score
Why is Zscore used?
- Represents the number of standard deviations the score is from the mean
- Can help define outliers (e.g. z score is less than -3.00 or great than +3.00)
How are differences between groups of conditions described?
Mean and SD
Describes the strength of a statistical relationship
Effect Size (e.g. Cohens d or Peasron’s r)
Difference between the two means divided by the standard deviation
Cohen’s D
Features of Cohen’s d
- Should always be positive
- Difference between the two group means, expressed in SD units
- Can combine/compare results across studies using different measures
Relationship between two variables whereby the points on a scatterplot fall close to a single straight line
Linear relationship
Relationships between two variables in which the points on a scatterplot do not fall close to a single straight line, but often fall alone a curved line
Non-liner relationship
Strength of a correlation between quantitative variables
Pearson’s r
What are the features of Pearson’s r?
- -1.00 to +1.00 value
- Score of 0 means there is no relationship between the variables
- Type of effect size
- Correlation does not equal causation
Difference between Cohen’s D and Pearson’s R
Cohen’s d measures the SIZE of the difference between two GROUPS while Pearson’s r measures the STRENGTH of the RELATIONSHIP between two VARIABLES
When can Pearson’s r be misleading?
- Relationship in study is non-linear (no straight line)
- When one/both variables have limited range in the sample relative to population (restriction of range issue)
APA Descriptive Statistics:
- use two decimal places
- Statistics written as numbers (not the word ‘two’)
- (M= 2.15, SD=2.27), do not write words mean or standard deviation within parenthesis
- “test restest correlation was .96, (r=-.27)”
Bars that represent the variability in each group/condition
Error bars (APA)
The SD of the group divided by the square root of the sample size of the group
Error extends one standard error in each direction usually
Standard error (APA)
Compare mean scores, use when the X axis is categorical (e.g. January, Feb etc)
Bar Graphs (APA)
When IV is measured in a continuous manner (e.g. time), use when the X axis is quantitative (e.g 1 min, 2 min)
Line graphs (APA)
4 Steps for conducting data analysis:
- Do not include identifying information
- Check raw data is complete and accurate
- Create data file (e.g. Excel or SPSS)
- Conduct preliminary analysis
(5. Do planned and exploratory analysis)
What happens in preliminary analysis of data?
- Assess internal consistency of the measure using Cronbach a/x)
- Analyse each important variable separately with histograms, central tendencies (mean etc) and variability (range, SD, variance)
- Identify outliers and decide to discard or maybe include but then use median for central tendency
Why do we do planned and exploratory analysis?
Answering primary research question, testing your data for the relationship expected in the hypothesis
How to do planned analysis?
- Investigate expected difference between means and SD of groups = use bargraph and compute Cohen’s d
- Investigate expected correlation between variables = line graphs or scatterplot and computer Pearson’s r
How to do exploratory analysis?
- Analysis undertaken without an existing hypothesis, may be relationships that you did not hypothesise
- differentiate from planned analysis when writing a report
- explain unexpected relationship maybe worthy of additional research, be skeptical as it will need to be replicated
A type of study designed specifically to answer the question of whether there is a CAUSAL relationship between two variables
Experiment
When a variable (IV) changes, it causes a change in another variable (DV)
Experiment
Two features of an experiment
- Manipulate IV levels (conditions)
2. Control of variability in extraneous variables (control = holding constant)
Changing the level or condition of the IV systematically so that different groups of participants are exposed to a different level of that variable
OR
the same group of participants is exposed to different levels at different times
Manipulation of IV
an experiment design involving a single independent variable with two conditions
single factor two level design
an experimental design involving a single independent variable that is manipulated to produce more than two conditions
single factor multi level design
How to control extraneous variables?
- conditions of the experiment being same
- limit participants to a specific category of persons (lowers external validity though
Why do we control extraneous variables?
Otherwise difficult to detect the effect of the IV
- Adding variability or noise to data (hard to detect IV)
- Becoming confounding variables (varies systematically with the IV)
Any intervention meant to change peoples behaviour for the better
TREATMENT
the condition in which participants receive the treatment
TREATMENT CONDITION
the condition in which participants do not receive the treatment
CONTROL CONDITION
an experiment that researches the effectiveness of psychotherapies and medical treatments
Random clinical trials
the condition in which participants receive no treatment whatsoever
NO-TREATMENT CONDITION
What does the placebo effect implicate?
The no-treatment condition (participants expecting to get better due to treatment)
Condition in which participants receive a placebo rather than the treatment
Placebo Control Condition
Condition in which participants are told they will receive treatment but must wait until the participants in the treatment condition have received it
Wait-List control condition
How to deal with placebo?
Give current best known treatment vs. the new treatment
each participant is tested on only one condition
participants in groups are on average highly similar
Between Subject Experiment
each participant is tested under all conditions
Within Subject Experiment
Features of Within-subject experiment
- Provide minimum control of extraneous participant variables
- easier to reduce data noise and see effect of IV
Using a random process to decide which participants are tested in which condition (strength in design)
Random Assignment
Two criteria for Random Assignment
- Each participant has equal chance of being assigned to a condition
- Participant is assigned to a condition independently of other participants
All the conditions occur once in the sequence before any of them is repeated
Block Randomisation
Block Randomisation benefits
- Good for equal-sized groups (coin flip can’t produce this)
Random assignment weakness
- Not guaranteed to control all extraneous variables across conditions
- However….Inferential statistics takes the liability of random assignment into account
- Confounds likely to be detected when the experiment is replicated.
An experiment design in which the participants in the various conditions are matched on the dependent variable or on some extraneous variable/s prior to the manipulation of the IV
(Alternative to random assignment)
Matched-groups design
An effect that occurs when participants responses to the various conditions are affected by the order of conditions in which they were exposed (e.g. tired, bored, practice) — Within Subjects Design
Order Effect
An effect of being tested in one condition, on participants behaviour in later conditions
Carry Over Effect
Participants perform task better in later conditions due to the chance to practice it
Practice Effect
Participants perform task worse in later conditions as have become tired or bored
Fatigue Effect
Unintended influences on respondents answers because they are not related to the content of the item but the context in which the item appears
Contxt/Contract effect
Solution to Order effects (within subjects design)
Counterbalancing!
Testing different participants in different orders
Counterbalancing
Equal number of participants complete each possible order of conditions
- best method
- use random assignment to allocate to diff orders of conditions
Complete Counterbalancing
Partial Counterbalancing
Latin square design (randomises through equal rows and columns)
Order of the conditions is randomly determined for each participant
Random Counterbalancing
Features of random counterbalancing
- use when the number of conditions is large
- will result in more random error
- if order effects likely to be small, and number of conditions are large then it is an option
What does counterbalancing accomplish?
- Controls order of conditions so that is no longer a confounding variable
- Makes it possible to detect carry over effects (analyse data separately for each order to see whether it had an effect)
Single list with both conditions in it, tested altogether
Simultaneous Within-Subjects Design
Which subject design is usually best?
Within Subject design
- controls extraneous participant variables, reduces data noise
- easier to detect any effect of IV upon DV
- required fewer participants
What are features of Between Subjects Design
- Conceptually simpler with less testing time per participant
- Avoid carry-over effects without need for counterbalancing
- Possibly better when treatment produces long term change in participants behaviour
Refers to the degree in which we can confidently infer a causal relationship between the variables
Internal Validity
Why are experimental studies high in internal validity?
Because IV manipulation provides strong support for causal conclusions
Refers to degree to which we can generalise findings to other circumstances/settings or the population
External validity
When the participants and the situation studied are similar to those that the researchers want to generalise to and participants encounter everyday
Mundane Realism
Where the mental process is used in both the laboratory and in the real world
Psychological realism
Why are experiments no always low in external validity?
- Experiments need not seem artifical, or consider field experiments
- often conducted to learn about psychological processes present in a variety of people and situations
One of the big four validities, whereby the research question is clearly operationalised by the studys methods
- Adding more conditions does not necessarily increase construct validity
Construct Validity
The specification of exactly how the research question will be studied in the experiment design
Operationalisation
Concerns the proper statistical treatment of data and the soundness of the researchers statistical conclusions (e.g. t test, ANOVA, regression, correlation)
Statistical Validity
What are the features of Statistical Validity
- Must consider scale of measure of DV and study design
- Many inferential statistics carry certain assumptions, statistical validity is threatened when these assumptions are not met but the statistics are used anyway
- Small sample size is a statistical validity critique
- Important to conduct a power analysis when designing study (informs number of participants to recruit)
Verifying the experimental manipulation worked, by using a different measure of the construct the researcher is trying to manipulate
Manipulation Check
Features of Manipulation Check
- used to confirm that the IV was infact successfully manipulated (for indirect IV like stress levels)
- important when results of experiment turn our null (decide if there is a real absence of effect of IV on DV OR is null result is due to problem with manipulation of IV)
- Usually done at the end of procedure, good to include in pilot test
