CHD4630 Midterm Flashcards
The four elements of research design
Variables, Sample, Location, and Method
Element of Research Design: Variable
what kind of data to collect
Element of Research Design: Sample
from whom to collect data
Element of Research Design: Location
where to collect your data
Element of Research Design: Method
how to collect and analyze your data
Clarifying your research focus
Clarify & focus in definitions help your research question be understood by readers
-What you aim to achieve
Sub-questions provide direction and structure for your research
-Ex: What role do family and friends play in motivating Facebook use? Extent of particular software applications motivate?
Strengths of Qualitative research Design
Emergent research design
-info. explore may influence future data gathering decisions (flexibility)
Exploration
-useful as first step in understanding ideas, perspectives and phenomena
In-depth understanding
-can enable deeper understanding of individuals’ perspectives and experiences
Weaknesses of Qualitative research
Lack of Breadth
-hamper ability to gather data from a broad range or large number of cases
-possible skewed understanding
Time-consuming
-voluminous data
-higher costs and longer time to prep data for analysis
Limited ability to generalize to population
-experiences or perspectives of small numbers
Strengths of quantitative research
Easier to generalize results
Large scale
-collecting data from large numbers, greater confidence not biased
Statistical analysis
-complex and powerful analysis that can identify subtle statistical patterns in the data
Visualizations
-easily translate data
Weaknesses of Quantitative data
Loss of depth and context
-individual-level patterns can be obscured, important contextual factors may be neglected
Limitations in accuracy
-limited depth of measurement or respondents not fully understanding or engaging with survey questions
Disconnected
-“Operationalizing” concepts and turning into Qs can create measures that are removed from real-world phenomenon
-may measure what not intended
Elements of Action research
Cycle of taking action
-Ex: Implementing intervention
Evaluating an action’s effectiveness
-Ex: Did intervention address research question effectively
Using that knowledge to inform future action
-Ex: modifying intervention
Action research does what?
Enables researchers to intervene with aims of improving or affecting something about your sample while creating new research knowledge
-may be specific system, service, product or outcome
-refined product & new knowledge
-usually continue until resources dry up or evaluation results become satisfactory
Evaluation research
= focus on assessing whether objectives or intended outcomes have been achieved by a particular intervention or initiative
-real life events or contexts
-Social, policy, learning, or business
-aims to provide useful information on how and why particular interventions or processes work or fail
-goal is to inform future decision-making
Process of writing literature review includes
- Focus you literature review while accounting for the big picture
- Search for books and articles
- Organize your sources
- Continually concept map
- Begin-in-depth reviews of categories
- Write your literature review
Process of writing literature review: Focus you literature review while accounting for the big picture
Place your topic within a relatively broad research and policy context to establish why your topic is interesting and orient your readers
Save time by defining your focus as narrowly as possible
The more tightly focused your topic, the more efficient and effective your research can be
If you don’t yet have a specific research question, try to sharpen your focus by talking to your colleagues/supervisor or doing initial reading about general topic
Keep making adjustments as you develop your literature review
Process of writing literature review: Search for books and articles
Investigate a lot of different approaches and ideas (open and curious, while maintaining goal)
Develop a search plan (words and phrases likely to lead to information)
-Enables you to maintain good coverage in your search and to explain how you went about your search in your research report
Evaluate & access potential sources
Weigh strengths and weaknesses
Peer review is gold standard for source credibility
-Widely accepted quality control mechanism in social research (can vary, so be critical)
Start with electronically accessible, saving difficult access for a later detailed search
Ensure relevance by first reading the title, then read the abstract or summary, and then by downloading or recording the details of the publication and its citation
Process of writing literature review: Organize your sources
Organize them into categories, with a folder on your computer for each category
Place downloaded publications into these folders along with a document containing copied and pasted details of publications you couldn’t download
Once you’ve created a preliminary set of categories, take stock to see if you can combine categories, if you need to expand your search or if you should re-focus away from any categories
Process of writing literature review: Begin-in-depth reviews of categories
Shouldn’t just report, but critically evaluate your sources’ claims
Evaluate whether the methods used in the study provide sufficient evidence for its claims
Don’t assume that established researchers are producing good research
Take 2 different sets of detailed notes on your sources
First set summarize the methods, findings and a few quotes that effectively describe the research results or conclusions
Second, start identifying connections (similarities or divergences) with other literature
Process of writing literature review: Write your literature review
Clear organization
Include introduction that emphasizes its importance, as well as stating what research has been conducted and how the section will be structured
Next, body section being the main part where you present your summaries, evaluation and application of the literature to your topic
Conclude by summarizing the gaps your research will be addressing
Reviewing your literature includes
Evaluation of core content
Assessment of evidence
Coverage of the topic
Reviewing your literature: Evaluation of core content
Has the author clearly defined the topic? How significant?
Whats author’s theoretical framework?
Does the information come across as opinion or propaganda?
Are assumptions reasonable?
Does the author take a radically different approach from others? Could it have been better if approached from a different perspective?
Reviewing your literature: Assessment of evidence
Are there methodological shortcomings? If so, how fundamental to the author’s claims are they? What are the implications for the author’s claims? Is the chosen research method leading the author to make certain claims?
Are claims supported with sufficient and credible evidence? Is the author overreaching with the publication’s claims?
Does the article rest on primary or secondary research (or just the author’s arguments or application of the theory)?
Reviewing your literature: Coverage of the topic
Does the source confirm or support other sources you’ve read, update other material, or provide new insights?
Does the publication appear to be of good quality?
What does it mean to be analytical?
Summarize publications, presenting any major patterns in the literature and relationships between studies
Use quotations only when the author says something in a particularly concise way, or when you are going to be disagreeing with the author’s claims
Keep your own ‘voice’ as you write
-may feel pressure to mimic the style or terminology of your sources, but staying authentic helps ensure the review presents your analysis
Identifying key gaps in the literature
Finding a research gap that you can address is a good way to demonstrate the value of your research
Helpful to ask self:
Are there areas of worthwhile study relating to your topic that have not been covered by other researchers?
Are relevant studies neglecting a particular type of evidence?
Is there a relevant theoretical concept that has not yet been applied to the topic?
What is research ethics?
= values or principles that are integral to your research objectives and define your responsibilities to your participants, your institution and yourself
-Provides guiding principles during the research process
-Essential in order to maintain a positive relationship with your research participants that will stand the test of time
-Participants need to feel they are protected and not taken advantage of
Purpose of research ethics is to
Maintain integrity
Protect the welfare of others
Build support for you and your research
Give you direction when facing challenging situations
Purpose of research ethics: Maintain integrity
Promotes good research by establishing that you should be consistently truthful, seek to gain knowledge and avoid error
Established a set of expectations for your conduct that prohibits falsifying information, deception or other acts that might bring disrepute to you, the quality of your research or your institution
Purpose of research ethics: Protect the welfare of others
A fundamental principle of research ethics is to do no harm to others
You must ensure your research does not put at risk the welfare or reputation of those directly or indirectly involved in your research
Purpose of research ethics: Build support for you and your research
Those who participate in or view your research will have a more positive view of both you and your research
Improve likelihood of recruiting participants for your future research and gain help from colleagues
In regards to using data for purposes other than specified
Always make clear to participants how you will use the data collected from them
The person giving you permission to use their information has a right to know how it is going to be used and then decide whether this is an acceptable purpose or not
You can only use the information gained from participants for the specific purpose that you declare to them and they authorize
2 different methods of achieving informed consent
Writtent and Oral conset
Written consent
Generally viewed as gold standard
Offers the most protection from a legal standpoint
Oral consent
Generally considered a weaker form of consent, although we contest this view
In some situations, gaining consent verbally is superior to using a written form, such as when a signature can be a threat to maintaining a participant’s anonymity
How to achieve informed consent
Carefully weigh the relative strength of the form of informed consent vs privacy needs to your respondents
When seeking informed consent, make clear:
Participation is voluntary
They have a right to refuse to participate
They can withdraw from participation at any point for any reason (they don’t have to tell you what that reason is)
Should specify early in the process whether the data will be anonymized or held confidentially
When obtaining, you may need to make it clear that there are limits to the confidentiality you can offer (ex: report violent behavior to authorities)
Permission to electronically record a research interaction must be explicitly granted
Ethics an vulnerable populations
Additional ethical implications
Carefully consider whether you are prepared and qualified to tackle
Potential for causing harm to your participants is greater
Consider that you may need additional training before embarking with vulnerable population
Assess whether the vulnerable participants are capable of giving informed consent
Must be able to understand what participating in your research would mean, as well as the nature of the risks associated with it
If there are guardians responsible for the well-being of your prospective participants, you should also seek their informed consent
When working with children, you generally need the consent of their parent, guardian or school (if it has the delegated authority to grant consent)
Also important to gain the direct consent of the child if possible
Participants of all ages may also be vulnerable
Difficult decision about whether to engage someone in research that will likely be personally upsetting (e.g., victim of abuse)
Employ neutral phrasing
Investigate phrasing that may upset or insult your participants (Useful resources:)
Speaking to gatekeepers or those with knowledge of the participants’ social group
Viewing your research from the perspective of your participants
Creating a list for yourself of verbal ‘no-go’ terms or topics
Be familiar with your participants’ preferred phrasing and mirror it where possible
Employing the phrase ‘so called’ can help distance yourself from controversial term
Stress and decision making
Stressful emotions can distort your research perspective
Your emotional response can be instructive, pointing you towards important insights
But it can also override your objectivity, making you vulnerable to bias
Repeated exposure to content can make you think that upsetting incidents are more common than they really are
May cause you to lose perspective and overestimate a phenomenon’s importance or relevance to your research
Raw emotional experiences can make you identify too closely with your participants
While you certainly want to be able to empathize with your participants’ perspectives, you must maintain emotional boundaries
Getting too emotionally involved in your participants’ problems can get in the way of successfully completing your research
Indicators of emotional risk in research: lower risk
Grant-funded
Familiar environment
Risk to physical health, security and property are better understood
Familiar participant community or topic
Team-based research
Allows you to share the workload and responsibilities
Strong social support network
Indicators of emotional risk in research: higher risk
Self-funded
Language barriers
Unknown environment
Risk to physical health, security and property are poorly understood
Solidarity research
Emotionally-charged or unfamiliar topic
Research requires developing new skills, such as a new statistical test
What is sampling?
Used when your population is too large for you to study everyone (almost always)
Pick a sub-set of population; a sample
Can be used for both qualitative and quantitative research
Quantitative goal:
make accurate generalizations about the population’s characteristics
Qualitative goal:
get beneath surface of phenomenon
Supporting your knowledge claims
Always include a methods section which allows your readers to assess the validity of the claims you make, and to identify any flaws in your research design
A good methods section includes:
Why you opted for a partiular sampling strategy and who you collected data from
How you gathered your data, identify and operationalize the variables you investigated
How you analyzed the data
Probability Sampling includes
Simple Random Sampling
Stratified Random Sampling
Cluster Sampling
Multistage Sampling
Systematic Sampling
Non-Probability Sampling
Don’t meet the standard of every member of equal probability of selection for all members of your target population
Generally much cheaper and easier than probability sampling, but should be used with caution
Can’t use inferential statistics to generalize from your sample to the population as a whole
Your conclusions should therefore be more tentative
Non-probability Sampling includes
Convenience Sampling
Quota Sampling
Snowball Sampling
Purposive/Theoretical Sampling
Simple Random Sampling
Ideal to aspire to as any of your individuals in your target population have an equal chance of being selected to be part of your sample
Members of your population are selected using random numbers for inclusion in sample
Requires little or no information about your population’s characteristics an is easily the most robust for of sampling
Becomes exponentially harder as the population size increases and the accessibility of your population decreases
Stratified Random Sampling
Involves dividing the population you are researching into exclusive sub-groups, or strata (age, gender, country, ethnicity, etc.)
Useful if you are looking to compare the attitudes of particular sub-groups (e.g. men to women)
Also useful if you are trying to compare sub-groups with drastically different sizes
Requires significantly more information about your population and is much more complicated to perform than simple random sampling
Cluster Sampling
Useful when your population is dispersed across a large geographical area
Divide the population into geographical groups, or clusters, often based on pre-existing boundaries
Then randomly select clusters: your sample consists of everyone in the cluster you selected
Can also be used to gather a representative sample when you have a large group of people in a space (e.g. professors, festivalgoers, etc.)
Physical space can be divided into imaginary grids
Multistage Sampling
Combination of sampling techniques
Example:
First randomly select clusters (cut down on cost by limiting the geographic disbursement of your sample)
Then stratify the individuals in those clusters (to ensure your sample is representative)
Then randomly select participants from the stratified groupings (to avoid sampling bias)
Systematic Sampling
Occasionally you will find that it’s just not practical, or possible, to use a truly random sampling method
Use ‘selection rules’ to mimic the objectivity of the random selection
Often, the selection rule is as simple as picking every Xth person to meet some criteria (like entering a room)
Not as good as strict random sampling, it can come relatively close to producing a random sample
Convenience Sampling
Selects the members of a population who are the easiest for the researcher to access
Can involve surveying your family or people you stop on the street, your Facebook friends, or anyone else who is easy to access
Unlikely that the people sampled through this method are representative of a wider population
Can be an effective way to test that survey questions, or conduct other preliminary pilot testing activities
Quota Sampling
Selection of a sample according to a predetermined quota (or number) on a non-random basis
Two ways:
Proportional quota sampling: selecting participants for your sample in proportion to some characteristic of the population as a whole
Non-proportional quota sampling: having minimum number of people you wish to sample within each category, rather than a maximum
Snowball Sampling
A technique where participants in your research are used to find additional participants
Almost guarantees your sample won’t be representative
However, it’s used when there may be no other way to access your population other than personal introductions
Ex: Those with an illness
Purposive/Theoretical Sampling
Uses researcher’s judgment to select participants who are likely to offer particularly useful insights
Need to use ‘special knowledge or expertise about some group to select subjects who represent this population
Selection process will be guided by earlier data collection within the project, prior research or theory, as well as the researcher’s instincts
Response rate
percentage of those you invite to complete your surveys that actually do so
Should keep a refusal log where you write down basic information including
Data/Time of refusal
Apparent age of non-respondent
Apparent gender of non-respondent
Apparent ethnicity of non-respondent
Explicit reasons given for the refusal
Other possible reasons for the refusal
Sampling frame
a summary list of all the members of a population who can be sampled (i.e. those who can be identified, located and contacted)
Type 1 error
alternatively known as ‘false positives’; the rejection of a null hypothesis that is actually true
Example: conclude that a new medical treatment helps to cure cancer, when it does not, the results would be tragic
Type 2 error
‘false negatives’; failure to reject a null hypothesis that is actually false
Ex: serious error if you were to dismiss a potential cure for cancer as ineffective when it actually cures the disease
The Incentive categories are
Individual, Social, and Ethical
Individual Incentives
Direct compensation
-Vouchers
-Tickets
-Gift certificates
-Prize draw
-Cash (fair market value)
Reimbursing expenses
-Travel costs
-Stipend for meals or alternatively provide refreshments
Social Incentives
Charitable
-Donation to charity of researcher’s choice
-Donation to charity of participant’s choice
Community/Society
-Details of how findings will benefit the participant’s community or society
Ethical Incentives
Access
Giving respondents access to the research findings
Manage concerns over privacy
Help assuage concerns about privacy by indicating to your participant that:
Their information will not be passed to third parties under any circumstances
There is no reason to believe that anyone would be specifically hacking your system
Every reasonable precaution (such as using password protected computers) will be used to maintain privacy
Collected data will be assigned a unique identifying number and the correlating respondent names will be stored on a medium not connected to the internet
Establish your credibility as a researcher
Take time to build up trust between you and the participant, especially if the topic is particularly personal
Soliciting participants: Decision tree includes either
Face-to-Face or Online
Soliciting participants: Decision tree for Face-to-face
Sufficient resources to access a big enough sample
Potential participant locations are clear
Credibility more likely to be gained through face-to-face interaction (depending on topic area)
Participants will feel comfortable discussing the topic in public (even if just an introduction)
Soliciting participants: Decision tree for Online
Small budget but large sample required
Large geographic area needs to be covered
Uncertainty around where participants may be located
Topic area is sensitive- participants will seek anonymity
Conclusion: Strategies and incentives
You should be prepared to demonstrate basic mastery of information about your research topic to assure your participants of your credibility as a researcher
You should also avoid terms and phrases that may sound condescending, pompous or unapproachable
The greater the time commitment, inconvenience and risk faced by your respondent, the more you will need to offer incentives for participation
You may need to work through gatekeepers in order to reach your participants effectively
What are surveys a good method for?
Can be used for describing patterns in a large population
Can determine individual’s characteristics
Can be used to assess social/political conditions from individuals’ perspectives
Layout and sequencing
Don’t include too many questions as it may be intimidating and harm your response rate
The question order affects the readability of the survey
Questions should go from general to specific and from easy to hard
Gradual introduction to the topic area
Save the demographics
Types of survey questions include
Open-ended, close-ended, classification and demographic questions, ranked responses, multiple choice answers: select one, multiple choice answers: all that apply
Strengths of Open-ended questions
Respondents can answer in their own terms
Limited researcher bias
Useful for exploring areas of limited researcher knowledge
Weaknesses of Open-ended questions
Time-consuming analysis (coding)
Require greater effort from respondents
Strengths of Close-ended questions
Easy to administer and complete
Easy to process and analyze
Enhance comparability of responses
Weaknesses of close-ended questions
Generate limited detail
Sensitive to design errors
Time-consuming to design
Classification and Demographic questions
Objective characteristics about respondents
Biological, sex, ethnicity, religious affiliation
Can be asked in different forms
E.g. Age can be an open-ended question, a multiple-choice question or by enquiring about the date or birth
Depends on the researcher’s preference
Recommended to collect demographics according to Census data questions for consistency (in US)
Ranked responses
Used to find out the respondents’ relative preferences about a list of options
This can be a good question type for researching comparisons
Multiple choice answers: select one answer
This question type provides pre-determined response options for a respondent to choose one answer to a given question
This criteria for Multiple choice answers: select one answer is that all response options should be
Exhaustive, Exclusive, Unambigous
Exhaustive
everyone fits into at least one category
Exclusive
everyone fits into only one category
Unambiguous
response categories mean the same to everyone
Multiple choice answers: all that apply
This type requires more time to analyze then types where only one answer is selected
This can be very useful question type in some cases
Ex: How did you hear about this event? Check all that apply/
If this is only used for descriptive statistics, the time taken to analyze the data is reduced
Types of survey
Cross-sectional, longitudinal, self-administered, verbally administered
Cross-sectional survey
Surveys are done at a single point in time
Most commonly used as they are a simple, low-cost option
Longitudinal survey
Studies use 2+ linked studies measuring change over a period of time
Lapse between studies can be hours, days, weeks, or years
Often seen as impractical: time-consuming, costly
Self-administered survey
Where respondents read and complete the survey themselves
Usually online, but can be distributed face-to-face
Verbally administered survey
Where researcher or trained assistants go through the respondents providing their answers verbally or pointing to a chosen category
Types of survey bias
Researcher Expectancy effect, Acquiescence bias, Demand characteristics, Social Desirability Bias,
Researcher Expectancy Effect
= researchers unintentionally introduce bias by designing survey questions and response options around their existing assumptions
Ex: a business’s customer service team expecting positive feedback might unintentionally bias their survey by asking leading questions
Acquiescence Bias
= respondents tend to agree with Likert scale (level of agreement) statements
Ex: If all such Likert scale statements are framed positively, the results may skew towards agreement
Demand Characteristics
= respondents may alter their answers based on what they think the researcher’s preferred result
Ex: Being asked to give feedback about a hospital by a uniformed hospital worker may result in more positive responses
Social Desirability Bias
= respondents may over-report views and behaviors that are widely praised in society and to make themselves look better
Ex: inaccurately reporting higher levels of recycling or charitable donations in order to appear more caring is typical of this bias
