Applied Research Methods Flashcards
What is research
- Investigation undertaken to gain knowledge and understanding
- A detailed study of a subject especially in order to discover new information or reach a new understanding
- Research methods developed in academia are applied in many real world domains
○ Eg commercial marketing
○ Government/social
Ogranisations
What is the applied research process?
Gap in knowledge -> question -> design + data -> need -> context -> insight -> needs met
What are the types of applied research objectives?
- Exploratory
○ Aims to discover the nature of a topic that is not clearly understood or defined
○ Qualitative or miced methods - Descriptive
○ Aims to describe or define the topics
○ Quantitative methods or miced methods
○ Analyses like correlations and between-groups and within-groups/repeated measures comparisons - Explanatory/causal
○ Aims to explain why or how things work the way they do
○ Quantitative methods
○ Experimental designs like A/B testing and
What are some important applied research design considerations?
- Population/s of interest
○ How specific is it? Do you need to look at different groups within the sample?- Factor/s of interest
○ What are you measuring? - Practical considerations
○ Budget
○ Timeframe
- Factor/s of interest
What are some mixed methods designs?
- Qualitative insights used to inform design of quantitative phase
- Quantitative insights raise questions that are best understood through qualitative examination
- Qualitative insights used to design quantitative evaluation, then quantitative findings are explored with qualitative methods
How is applied research reporting different from regular research?
- Slide decks vs report format
- Story telling approach to communicate findings
- Design for your audience - what is most important to them?
- Keep it short - put detailed results in appendix
- Include Overview/executive summary at the start - help orient people to what they’re about to hear/read
- More visuals and less text - show don’t tell
- Insights are the golden egg
What are the 8 principles covering how to conduct research, how to treat others, and how to behave in a professional manner?
-Honesty
-Rigour
-Transparency
-Fairness
-Respect
-Recognition
-Accountability
-Promotion
How to design research for inclusion?
-Develop cultural competence
-Design for accessibility
-Consider potential biases
-Consider impact of cultural norms
-Involve specific participant groups in end-to-end process
-Build neurodiversity into methodology
How to ensure psychological safety in applied research?
-Wellbeing of participants is always more important than the research
-Follow trauma-informed practice principles
-May be at risk of vicarious trauma from unexpected or expected disclosures
What is a survey?
- The most popular data collection tool
- Commonly used to assess opinions, attitudes, and preferences, and enable self-report of behaviours and intentions
- Different to psychological assessment tools which objectively measure constructs such as personality traits and knowledge or assess psychopathology or mood
- Questions are most often closed-ended producing quantitative data but can be open-ended producing qualitative data
What is the Net Promoter Score (NPS)?
○ *The most used user feedback score
○ Kinda like the Myers Briggs Inventory
§ Very popular, used in industry etc but is very love or hate
Often have a comment section after the scale to get users to discuss why they responded the way they did - this is where the ‘gold’ comes from
Why is Net Promoter Score (NPS) divisive?
- Negatives
○ A lot of professional researchers think the psychometric qualities don’t stand up, low replicability
○ People don’t use them consistently (in terms of labels etc)
○ We are so bombarded with so many survey feedbacks, so you have to have had a really intense experience to want to respond
○ Intention-behaviour gap
○ They are often used for services that people just don’t recommend- Positives
○ Asking people one scale response - easy to do,
Then ask for their comments on why they responded that way
- Positives
What are some customer experience (CX) measures other than NPS?
-Customer Satisfaction Score (CSAT): gets your to rate how satisfied you are, has face validity
-Customer Effort Score (CES): face validity, bipolar scale
-Star rating
What are the limitations of surveys?
urveys are prone to biases
○ Social desirability
§ Effects accuracy
○ Intention-behaviour gap
§ The size of the gap may be dependent on factors such as
□ Whether the intention based on personal attitudes or social pressure to act (social norms) - former smaller gap than the latter
□ How much effort the behaviour requires - a verbal recommendation requires relatively little effort compared to changing to a keto diet or buying a house
§ What to do?
□ Minimis the use of behavioural intention questions and make sure your client knows their limitations
□ Consider using big data on actual consumer behaviour as well or instead
○ Acquiescence/agreement
§ Tendency to just agree with things
○ Question order
§ Priming
§ Primacy
§ Recency
□ We have a tendency to be influenced by how recently we heard the information
○ Recall bias
How to design a good survey?
- Design is for optimising, not satisficing - Krosnick
- Optimising
○ Interpreting and responding to a survey question using a careful and considered process - Satisficing
○ Taking shortcuts when responding to a survey question
- Optimising
What are some optimising strategies when designing a survey?
- Reduce the task difficulty
○ Make questions easy to understand
○ Keep the survey short
§ No more than 30 mins (~ 30 responses, but you’ll need a pre-test)
§ For mobile-first survey, 7 mins max
○ Minimise distractions- Increase respondent motivation
○ Use incentives and gratitude
○ Ask respondents to commit to provide their best responses
○ Emphasise the importance of of the survey and their responses
- Increase respondent motivation
What is the conventional wisdom for survey question order?
- Start with questions on the topic described to respondents
○ Easy questions early
○ Group by topic
○ General to specific
○ Sensitive topics at the end
○ Use filters to avoid asking unnecessary questions
How can you guide your participants through a survey?
- Include introductory text that clearly informs respondents what the survey is about, why you’re asking them to do it, how their answers will be used and how long it should take
- At the start of each topic section, consider starting with a sentence saying what the questions are about, for example
○ For demographic questions: ‘first, we’d like to find out a bit about you’
○ For a block of rating questions: ‘In the following section, you will be shown
pictures of different foods and you will be asked your opinions of these foods. - At the end, make sure you thank them
- Consider including a progress bar that shows how far along in the survey they are
- At the start of each topic section, consider starting with a sentence saying what the questions are about, for example
What is the conventional wisdom on survey question wording?
- Use simple, familiar language
- Use simple syntax
- Specific and concrete (as opposed to general and abstract)
- Make response options exhaustive and mutually exclusive
- Common mistakes
○ Using ambiguous words
○ Leading or loaded questions
○ Double-barreled questions
○ Double negative wording
○ Emotionally charged words
What are rating questions in surveys?
- Most used question format
- Obtains a judgement on an object (described by question wording) along a dimension (provided by a response scale)
- Choosing the number of response options is a choice between having enough to differentiate between respondents as much as (validly) possible while still maintaining high reliability in responses (which comes with fewer options)
- According to Krosnick the ideal number of options is 5 points for unipolar scales (not at all satisfactory - very satisfactory); 7 points for bipolar (extremely dissatisfied - extremely satisfied), but consider 5 points if you have mobile first data collection
- Ideally label all points on your response scales and use words, do not use (only) numbers
Describe multiple choice questions in surveys
- Enables respondents to indicate one or more responses from the list eg preferences, behaviours etc
- Allow you to apply a pre-existing structure to your data eg groupings or other categories like demographics (except age)
Describe ranking questions in surveys
- Enable comparisons between multiple things at once
- Useful when wanting to measure comparison or choice-relative value
- May be more reliable than rating questions, particularly for items at the ends of the ranking scale
Describe open-ended questions in surveys
- Enable you to ask exploratory questions and gather qualitative data
- Often good to add Other (please specify) option to you rmultiple choice questions
- But
○ they can increase task difficulty
○ More time-consuming to analyse
How to minimise bias in survey responses
- Social desirability
○ Remind people of anonymity
○ Use the wording to make the less socially desirable response ok- Acquiescence/agreement
○ Avoid communicating the intent of the research
○ Keep it short
○ Vary response scales
○ Add attention checks - Order effects
○ Randomise question order and/or response order (where appropriate)
- Acquiescence/agreement
What is benchmarking/baselining in surveys?
- Sometimes you will need to create a standard to measure your results against and this can influence your choice of research design and the questions you use
Describe the survey testing process
- Pilot testing
○ At least 5-10 people from your population of interest. You can add a question at the end of asking for any feedback on the survey on the survey. Look at the data to check for completeness or any unusual patterns- For larger surveys, developers sometimes use cognitive interviewing for testing. This involves people doing the survey while an interviewer prompts them to ‘think aloud’ and asks questions to explore their comprehension, information retrieval, judgement and response
- Factors to consider
○ Comprehension - respondents understand the question wording and any instructions
○ Logic and flow - questions follow a logical order, nothing seems out of place or creates biases for what follows
○ Acceptability - none of the questions could be considered offensive or inappropriate
○ Length - most respondents finish without losing interest
○ Technical quality - the survey operates well on any device
How to identify your population and sampling approach for surveys?
- Census designs - in which the whole of the population participates - are uncommon
- The remainder of studies use a sample of the population of interest
- Your approach to sampling, as part of your research design, is informed by the objectives of your research as well as practical constraints (eg money and time)
- The two broad categories of sampling are random (aka probability) and non-random (aka non-probability)
- How you sample your participants affects the generalisability of your findings (external validity) to your population of interest
- The size of your sample affects your statistical power - a consideration if you want to do hypothesis testing (correlations, between groups differences, pre- and post-differences)
Describe random sampling
- A random sample is a sub-group of your target population that has been selected randomly such that each person in the population has an equal chance of being selected for the sample
- This process reduces the risk of bias that comes from selection methods that systematically increase or decrease the occurrence of particular characteristics in the sample
- However, as research participation is voluntary, most samples are not truly random because they opt-in/self-select
- Random samples are best practice for evaluative and experimental research, where null hypothesis statistical testing is used for making inferences about relationships between variables in the populations of interest (so systematic bias needs to be avoided)
What are non-random samples?
- Members of the population of interest do not have an equal chance of being selected for the sample, thus there is a higher risk of bias in the data they produce
- Types of non-random samples
○ Convenience
§ People who are readily available in a non-random way (eg online panels, Researcher Experience Program)
○ Quota
§ Often used with convenience samples, researcher selects sub-sets of sample based on characteristics (usually demographics) to increase representativeness of sample
○ Purposive
§ Selecting people because they have characteristics of interest for the research
○ Snowballing
§ Finding participants who then refer other potential participants to you - To help control the risk of bias created by non-random sampling, researchers use quota sampling and weighting of data to make the sample findings better represent the population
- Types of non-random samples
Describe online panels
- Online panels are now the most common source of participants for applied research
- Panel providers have a large database of people who sign up to do research regularly for a small payment
- Different types
○ General population
§ Can use random and non-random recruitment from within the panel
○ Specialist
§ Industry or population sectors
○ Proprietary
§ Research organisations collect their own data, create products, and sell - Those that specialise in supplying to research agencies are much better than those who supply directly to consumers
- For general population studies look for large size (eg an Australian panel of 1 million is good) with a large range of attributes
- Data quality issues
○ Some panels have major issues with bots/server farms producing significant proportions of junk data
What are types of data collection
- Online
○ Most common, cheap and fast- Telephone
○ CATI - computer-assisted telephone interviewing
○ Survey is a structured interview with instructions
○ Best with multiple-choice questions, bad for open ended - Face to face
○ Most expensive
○ Sometimes referred to as CAPI - computer-assisted personal interviewing
Useful for in-field/contextual data collection
- Telephone
Describe the process of translation in surveys
- The goal of survey translation should be to achieve functionally equivalent version in the target language
- Usually in survey research, this means that one follows an ask-the-same question approach, where the questions are translated so the same concept is measured on the same measurement across languages
- Ideally, use at least two translators with a background in survey research to separately draft a full translation of the questions. Then these are reviewed and integrated with another person
What are the different types of validity?
- Construct validity
○ Are you measuring what you say you’re measuring?- Internal validity
○ Are your causal claims valid? - External validity
○ Are your claims about generalisability valid?
§ Population validity
§ Ecological validity - Statistical validity
○ Are you statistical conclusions valid?
- Internal validity
What are different aims of research
-Testing for differences (between existing groups or manipualted groups): internal validity is v important
-Generalising to a population (external validity is v important)
When are manipulated groups not better than existing groups in research
○ Some things can’t be manipulated
○ Some things can only be manipulated by modifying groups so much that we can no longer generalise to the situations we want to generalise to, or making other compromises
○ Sometimes we choose an outcome first (eg depression), and we want to know its causes. Experimentation can’t help with the search for candidate causes
○ Experiments are good for getting at ‘descriptive causation’ but not very good for getting at ‘explanatory causation
○ Waiting on an experiment can mean delyaing evidence-based solutions
○ Experiments favour the testing of light-touch interventions rather than structural changes (bc they’re hard to manipulate )
○ People willing to be randomised may be unrepresentative
○ Knowing you’re part of an experiment may alter the results
What are some possible threats to the research aim of testing for differences (in terms of threatening internal validity)
○ Differences between existing groups
§ Cause-and-effect relationship (‘X causes Y’) is very difficult to establish
§ Possible threats
□ Y causes X - reverse causality
□ Z causes X and Y - third variable
○ Differences between manipulated groups
§ Cause-and-effect relationship can be established
§ Possible threats - smaller but still exist
□ Manipulation of X can also affect other variables - confound
□ Participants drop out in non-random ways - selective attrition
In terms a research aim seeking to test for differences, what are the key types of external validity?
○ Population validity
§ Getting a representative sample is often quite difficult
§ Convenience samples are very common, often accepted
§ Representative sample are always better, all else equal
□ Convenience samples always come with a risk that results were due to an unrepresentative sample
○ Ecological validity
§ Your study should not be too contrived, or too far from the real-world context that you want to draw conclusions about
* Essentially there is a trade-off between internal and external validity, because to have good internal validity you need to have a highly controlled and manipulated experiment, whereas external validity benefits from observational
What is a Type I error?
False positive
What is a Type II error?
False negative
What is a p value?
- Null hypothesis significance testing (NHST)
- Starts by assuming the null hypothesis (no difference) is true (we don’t really believe this, but we pretend we do)
- Asks: how unlikely is the difference I observed if I assume the null hypothesis is true?
- Smaller p-value =the more extreme result. If p < .05, we conclude that null hypothesis is false. But that’s not sound logic!
What are the statistical conclusions when testing for differences?
- We didn’t detect a difference (null hypothesis couldn’t be rejected)
- There is a difference between groups (null hypothesis is false)
- Chance of error: when null hypothesis is true, 5% chance
○ For this to be the case, we need to play by the ‘rules’ (below), and we often don’t - How big is the effect? (Effect size and 95% confidence interval)
- Chance of error: depends
- We didn’t detect a difference between groups (null hypothesis couldn’t be rejected)
What are the rules of Null Hypothesis Significance Testing to keep the p-value honest?
- A p-value is valid if:
- You do only one significance test (or correct for multiple tests)
- You choose your test ahead of time - can’t change your analysis after you see the data (eg exclude outliers, transform variables)
- You don’t peek at your data - must collect all data, then analyse once
- Solutions
- Pre-register your data collection and analysis plan
○ Be specific - what is your key hypothesis test?
○ Report any unexpected or unplanned result as exploratory (new hypothesis generated), confirm if a replication is possible
○ Report all studies you ran, and all statistical tests you conducted. Pay attention to the entire set of results, not just the ‘best’ ones
When would it be appropriate for a research aim to be generalising to a population, and what are the most important factors?
- When you want to know frequencies, proportions, levels (eg How many people commute to work by bus?)
- Most important factors:
○ Valid measures (construct validity)
○ Representative sample (external validity)
○ Large sample (statistical validity)
○ A small but representative sample is better than A large but unrepresentative sample
Describe the difference between noise and bias
- Noise = random error, imprecision
- Can reduce noise by collecting more data - aggregation cancels out random error
- Bias = systematic error, not random
- Can reduce bias by collecting a representative sample
- Large samples reduce noise but not bias
- A large, unrepresentative sample gives you a precise estimate, but may be inaccurate
- Representative samples reduce bias
- A small, representative sample gives you an imprecise estimate, but likely in the right ballpark
What are some important considerations for systematic reviews?
- Important consideration: Good idea to mix together both quantitative and qualitative methods to allow your own judgement to be inserted
- Another Important consideration: If you leave things too open, biases can come in and it can mean the conclusion/interpretations of the data can be misguided
○ Note that this is quite contradictory to the above consideration in terms of one pressing the importance of your own judgement, and the other stressing making things predetermined to make it as objective as possible - this is a tricky weigh-up
- Another Important consideration: If you leave things too open, biases can come in and it can mean the conclusion/interpretations of the data can be misguided
What step in the systematic review stage is most subject to biases?
- Data extraction and appraisal is the step where bias most commonly comes into play
Most important thing is to be transparent about what you did so that the reader can assess whether it is clouded by bias
What to look out for when evaluating research quality?
- Something to look out for: the closer a p-value is to .05 (eg .03), the less strong the effect and the more likely it is to be influenced by inadvertent p-hacking
○ If they preregistered their plan - eg recruitment methods, data collection methods etc - would reduce the reasonable scepticism- Many literature reviews do little or no quality evaluation
Even academic papers (meta-analyses, systematic reviews, research syntheses) often do not evaluate the quality or validity of individual studies
- Many literature reviews do little or no quality evaluation
What are the three components to quality research?
-Transparency
-Strong methods (valid)
-Calibration
What is transparency and why is it critical to quality research?
What is transparency?
* Procedures, materials, and data are reported in full
○ Would be easy to replicate the study
○ Would be easy to reanalyse the data/reproduce the results
* All relevant studies and all relevant results are reported
* Unplanned analyses and results are clearly marked
○ ^Pre-registration lets readers see what was planned and what wasn’t
○ Registered reports provide protection against p-hacking and HARKing
* Conflicts of interest are declared
○ Common in med but not as much in psych
○ It’s a bit of a grey area what a conflict of interest is within psychology
* Author contributions are reported
* Peer review history is public
* Transparency is necessary for evaluating quality
○ Without transparency, you can’t assess quality, but it doesn’t make a paper credible
Why transparency?
* Transparency is necessary for credible science
* ‘Nellius in verba’ - take no one’s word
* HARKing = Hypothesising After the Results are Known
○ Changing your hypothesis and pretending it was planned
* P-Hacking = analysing your data many different ways and only reporting the significant results
○ Changing your analysis and pretending it was planned
* File drawering = not publishing the unsatisfactory findings
* Publication bias = journals not publishing findings because they are not significant, or not the finding they want etc
What is pre-registration?
- Decide your design (sample size, conditions, measures), analysis plan, and key hypothesis tests
2. Write it down & time stamp it
3. Collect & analyse your data
4. Share your plan when you share your paper
What are registered reports?
§ ^Another way to block these biases from coming in, which is stronger than pre-registration
§ Authors will write their intro, lit review, method etc without having done the study yet, then they submit it to a journal and gets peer-reviewed
§ Authors make a commitment to take on the edits, and the editors make the commitment to publish the paper if the edits were taken on
§ After data has been collected and analysed, and the paper is written it gets reviewed again before publishing
□ Makes the HARKing, p-hacking, file-drawing and publication bias very very unlikely
What is calibration with regards to research quality?
- If the study can speak to the research question, but the conclusion was much more extreme - that is an issue of calibration
What are the red flags to look for when evaluating evidence quality?
○ Research isn’t reported transparently
§ You can’t tell what they did, wouldn’t be able to repeat it
§ The data aren’t available (and there’s no good reason given)
§ You can’t tell what was planned and what wasn’t
○ The methods are weak or not well-suited to the research question
§ Bad measures or manipulations (or mismatched to the aims/interpretations)
§ Bad sample (or mismatched to the aims/interpretations)
§ Causal inference
§ Etc
○ Authors make grandiose or exaggerated claims
What are some qualitative approaches in research theory, method, and analysis?
- Ethnographic
○ Observations of people in situations, to capture external influences (socio-cultural, environmental)- Phenomenological
○ Exploring what emerges, why they act and react the way that they do (beyond unreliable self-reports), ie ie interpretive, phenomenological analysis
○ Extracting meanings from observation - not trying to explain but looking for a picture of what’s going on - similar to ethnographic, but ethnographic you are more situating yourself in someone else’s context and taking in what role their environmental factors could play - Grounded theory
○ Builds on existing theory, question, or data which gives structure to data collection and analysis - Case study
○ Deep exploration of one thing - eg person, group, organisation - Narrative
○ Done over time with the aim of developing a comprehensive story of the phenomenon of interest
○ Looking for a story not to explain but to build a picture - Historical
○ Use of past events/instances as models to explore current situations
○ Using events of the past to help explain what’s going on now - A lot of these methods are not necessarily used very often - a lot use thematic analysis
- Phenomenological
What are some popular qualitative methods?
- Interviews/In-depth interviews (IDIs
○ By far the most popular- Focus groups
- Content analysis (text, visual)
○ Can sometimes be used interchangeably with thematic analysis - Online/Insight communities
- Observation/ethnography
- Open-ended survey questions
When to use interviews
- Use when
○ you’re interested in individual perspectives and experiences
○ When the topic is sensitive - so confidentiality is important, and disclosure will require some degree of trust and rapport
○ There are concerns about fear of reprisal
○ You want to avoid group effects- Budget is a consideration - interviews are resource intensive/expensive
- Sampling for interviews
○ Will have smaller sample sizes in qual than quant because we are looking for theoretical saturation (not hearing any new ideas) rather than statistical power)
○ Key informants - sometimes you need people will really specific knowledge on the matter
§ All part of the sample - not considered separately
What are some interviewing tips?
○ Rapport is critical because people will only talk candidly if they
§ Feel comfortable
§ Feel secure about confidentiality
§ Trust they will be understood
§ Trust they won’t be judged
○ Beware of
§ Influencing by leading questions conveying your own view (implicitly or explicitly) or giving examples
§ Moving too quickly from one topic to another
§ Moving off topic
§ Interrupting the person or speaking over them to ask a question
□ Majority of interview types in qual are semistructured
What is an appropriate interview guide for how to begin and commence the interview process?
○ Good to actually start with the warm-up questions like ‘how has your day been’, then
○ Begin by asking factual, socio-demographical questions, but if there are a lot or some are sensitive (eg income, age, and sexual orientation), it might be better to ask them at the end
○ Your first question/s about the research topic should ask for relatively neutral, descriptive information (eg Tell me about your career so far)
○ Aim to use open-ended questions, eg
§ Tell me about a time when…
