Research Methods Final Flashcards
What creates false positives
Incentives to publish - academics are rewarded for publishing which can motivate people to take shortcuts
Questionable research practices - slightly adjusting design, analysis, and reporting to produce p value above .05
Specific questionable practices
Measure the dependent variable in multiple ways
Gradually add more observations
Add and drop covariates
Add or drop experimental conditions
Combining these QRP’s gives you a 61% chance of getting p over 0.05 for an effect that isn’t real
File drawer problem
Studies showing null effects often wind up in a file drawer instead of in a journal
The published literature is thus heavily biased towards studies that “worked” (publication bias)
What can help false positives
Pre Registration: mitigates QRP’s: reporting exactly how you will conduct your experiment beforehand so you can be fact checked when you go to publish their results
Open materials and data: reader can see and test all dependent variables, test analyses with and without covariates, see and test all experimental conditions
Make unsuccessful studies searchable
Journals publishing null results: studies that answer important questions no matter the result are worth publishing
Credibility revolution
The movement has expanded to many other ways we can improve our research practices
Credibility is not just about statistical results
What is a construct
variables that cannot be observed directly
Defining variables
Conceptually vs operationally
Must conceptually define constructs
Must operationally define constructs
Every variable in study must be operationalized, must operationalize based on conceptual definition
Types of measurement
Self-report,
Behavioral: could be naturally occurring or lab induced
Physiological: assessment of bodily states (fmri, heart rate, pet)
How to choose which type of measurement to use
Previous research; how was this variable measured in previous studies
theory
methodological advances: new technology means new ways to measure
Feasibility: resource limitation like time and money may affect your choices
Test-retest reliability
same test given twice with some time in between, good for stable qualities like personality, not good for temporary states like mood
Parallel forms reliability
different forms of the same test used
Internal consistency
split half correlation ( top half questionnaire is compared to bottom half, Chronbach’s Alpha tests how items are intercorrelated
True score
the real score on the variable
Obtained
the score measures give
Measurement error
difference between true score and obtained score
Face validity
does measurement look like it measures the thing it’s meant to measure?
Survey research
uses self report
Tries to obtain generalizable samples - ideally random and large
Interviews
Structured or unstructured
Costly
Interviewer bias
Social desirability concerns
Phone surveys
Structured or unstructured
Used to be easy to get random samples, now no one picks up their phone
Cheaper
Less social desirability
Questionnaires
Paper or electronic
Cheapest
Fewest social desirability concerns
Survey advantages
Can access non-observable variables as well as variables you cannot ethically or practically manipulate
Demographic info
Attitudes and benefits
Past behaviour
Current behaviour that cannot be observed
Motivation and emotion
Personality traits
Easy to administer
Quick and easy way to gather lots of information that requires few resources
Survey disadvantages
Accuracy may be low
Participants may lack insight
May forget previous behaviour
May respond in socially desirable way
Not manipulating IV, thus cannot demonstrate causation
True of all correlation / non experimental research
Good survey
BRUSO
Is brief - avoid long sentences and jargon
Relevant - avoid temptation to include extra items that stray from question - avoid personal/ nosy questions
Unambiguous - don’t be vague or use negative wording
Specific - avoid questions with multiple ideas packed in
Objective - questions shouldn’t have emotionally charged words
The hidden curriculum
norms and opportunities within academic culture that are rarely explicitly taught
Employers want
communication skills, clear and precise writing, persuasive speaker, careful listener
Strong work ethic: high ethical standards, effective time management
Sense of initiative: persistence in face of challenges, can plan and carry out projects
Team work skills
Interpersonal skills: deals with a wide variety of people, good at handling conflict
Strategies for academic success
Reading for comprehension:
read in distraction free setting
Take notes that summarize reading
Look up unknown terms
Studying effectively:
Connect the concepts to each other and to things you care about
Organized ideas are easier to remember then a list of unrelated facts
Familiarity is not knowledge; test yourself, flashcards good
Spaced practice is better than cramming
Time management:
Break large tasks into small ones
Block more time then needed for each deadline
Be aware of planning fallacy
Figure out best time of day for you a d plan most difficult tasks for that time
Install a website blocker
Caring for yourself:
University is demanding job
Set aside and protect the time when youre not working - guilt free
When you’re tired, take a break
Building and academic network
Office hours - You can just go to office hours to chat with prof
Class discussions - Voice your opinions in class discussions, Listen to classmates - build on others points
Attending departmental events
Joining research labs
Attending conferences
Joining a research lab
email profs whose research interests you, independent study, apply to work in labs as work/study student
Emailing prof about ab involvement: use proper salutation, express interest, give availability (which sem) attach resume and grade report
Research assistant: 6-12 hours per week of research based tasks, responsibility and independence increases over time, regular lab/ team meetings
Advantages of academic network
Opportunities: scholarships, internships
Advocacy: reference letters
Mentoring and support
Getting reference letters
pick profs who know you well, make it easy for them, ask if they’d be willing, give package of all relevant info
Personal statement
: academic, professional, word doc that communicates as forcefully as possible why you’re a good fit for the program of your choice, goal is to sell yourself to grad program
Statement should explain:
What you want to study at this program
Why you want to study it
What relevant experience you have
What you plan to do with your degree once you have it
Show strengths by tying in experiences and accomplishments that demonstrate them
Use active voice and dynamic sentences, don’t be boring but don’t be too theatrical
Tailor each statement
Measurement
assignment of scores to individuals so that scores represent some characteristic
Psychometrics
measurement in psych
Constructs
ideas that cannot easily or accurately be assessed (traits, emotional states, attitudes)
Operational definition
definition of variable about how it will be measured, multiple for any given construct
Converging operations
psychologists use multiple different definitions of same construct - within or accoress studies
Nominal level of measurement
measurement used for categorical variables and involves assigning scores that are category labels
Ordinal measurement
ranking and ordering of qualitative data (very satisfied to very dissatisfied) - cannot assume variance
Interval level of measurement
assigning score using numerical scales where intervals have the same interpretation through (temperature scale, IQ) - can never be 0
Ratio level of measurement
assigning scores where there is true zero point(# of siblings, score on exam, how much money you have rn)
Cronbach’s a:
a is mean of all possible split half correlations for set of items, best test for internal reliability
Sensitive to # of items in scale
Higher then .90 = excellent
.90-.80 - good
.70-.60 - questionable
Lover then .60 - poor
Validity
extent to which scores from measurement represent the variable they are intended to
Content validity
extent to which measure covers construct of interest
Criterion validity
people’s scores on a measure are correlated with other criteria that one would expect them to be correlated with (test anxiety should be negatively correlated with performance on exam, if this is found it would be evidence that scores truly represent people anxiety)
Predictive validity: does it predict expected outcomes
Criterion can be any variable that should be correlated with the construct being measured - usually lots of them
Convergent validity
measures how well constructs that are theoretically related correlate (test measuring extraversion should be correlated with test measuring self-esteem)
Discriminant validity
Measure should NOT correlated with theoretically different variables. Ex. Loneliness and security - if scores are not correlated, test has high discriminant validity
Divergent validity - same thing
Of measure doesn’t have discriminatory validity it is too broad
Demanded characteristics
subtle cues that reveal how researcher expects participants to behave
APA style
Genre of writing appropriate for presenting results of psychological research
Title page, abstract, intro, method, results, discussion, references
High level style APA
formal; adopts tone appropriate for communicating with professional colleagues, straightforward; communicates ideas as simply and clearly as possible.
Avoids language biased against certain groups - to avoid offense and scientific objectivity and accuracy
Reliability
consistency of measure
Abstract
summary of study - 200 words
Introduction: introduces research question and explains why it’s interesting, lit review discusses relevant previous research and closing restates research question and methods used to answer it
Opening; 1-2 paragraphs, introduces research question, its importance or interest
Lit review: describes relevant previous research
Closing: final paragraph of intro, clear statement of question and hypothesis, brief overview of method
Method section of paper
describe how you conducted your study - clear and detailed enough so someone could replicate it exactly
Discussion
summary of research, theoretical implications, practical implications, limitations, suggestions for future research
Appendix
tables figures, supplementary material, stimulus words, questionnaire items
Review and theoretical articles
Review articles summarize research on topic without presenting new results
Theoretical articles are when these articles present new theory based on past research
Final manuscripts
published not in journal, dissertations, theses, other student papers, easier to read
Other way to share research that isn’t papers
Professional conferences, Oral presentations, poster, image description
Survey research
Qualitative and quantitative method using self-reports with careful attention payed to sampling
Context effects
effects not related to content of item but to context in which item appears
Item-order effects
order items presented in affects response
Open bs closed ended items
Open-ended items: ask a question and allow interpretation/ any response
Close ended items: provide set of options
Rating scale
ordered set of responses, ex. Frequency: never, rarely, sometimes, often, always
Probability sampling
researcher can specify the prob that each member of pop will be selected for sample
Non probability sampling
Convenience, snowball sampling, quota sampling, and self selection sampling
Sampling frame
list of all members of pop that can be selected for sample
Proportionate stratified random sample
to select random sample with equal amount from each strata
Disproportionate stratified random sample
can be used to sample extra respondents from small subgroups
Cluster sampling
larger clusters of individuals are randomly sampled and then individuals within each cluster are sampled ex. Select several small towns and then select several residents of each small town
Nonsense measures
Convergent validity and reliable
Graph formatting
Title - summarizes graph, explains what c and y represent
axis labels - x(horizontal) y(vertical) what do they represent? Includes units of measurement
legend - key to data plotted - what do colours represent
footnotes - further explain data and source
Appropriate representation of axes, scale, and error
Clutter-free visual
Indicating range of error
Only necessary when reporting from samples or where there is uncertainty
95% confidence = range of values where we are 95% sure true population value lies
Represented by error bars - if they do not overlap, there is significance
Visual style of graphs
Reduce clutter
Highlight what is important
Data ink: numbers and vital points representing data
Colours and patterns: can be useful but should not be overused and distracting
Dimension: should be 2d
Types of bar graphs
Bar graphs - categorical data
Vertical bar graphs - comparing estimates
Horizontal bar graphs - combined of above
Cluster bar graphs - two or more cathodes
100% stacked column graphs - compare percentages when total is 100% across categories - no more then 3 components
Other graphs (line,histo,scatter,box,pie, time)
Line graphs - illustrate trends/ different variables overtime - x-axis usually discrete, y usually continuous
Histogram: each bar represents range of data - categorical, can show normalcy, binning matters
Scatterplot: shows relationship between two variables, all data represented in dot - work best for continuous variables
Box plot - shows range, outliers quartiles
Pie - show percentages or parts of a whole
Time series graph - shows change over time x= discrete time, y= variable (price)
Categorical data
Each value represents discrete category - order does not matter
Numerical data
Each value represents either a real number (age) or a place on continuum (rating scale) - order matters
Importance of y axis
Must start at baseline so difference isn’t exaggerated
Too broad range may minimize differences
Can’t be upside down
Can only be 1
