Research Methods B Flashcards
What is an experiment?
- Manipulation of one or more variables
- Determine the effect of this manipulation on another variable
- To test the cause-effect relationship between variables (test of causality)
What are hypotheses?
- A hypothesis is a testable prediction
- Hypotheses are derived from theories
- Science is about testing hypotheses
What is an experimental/alternative hypothesis, and give an example?
- Treatment DOES leads to an effect
- ‘Learning with background music DOES lead to lower marks’
What is a null hypothesis, and give an example?
- Treatment does NOT lead to an effect
- ‘Learning with background music does NOT lead to lower marks’
Explain dependent and independent variables
- Manipulating the independent variable changes the value of the dependent variable
- Independent is changed, dependent is measured
What is a nuisance variable, and give an example?
- An additional factor that affects the dependent variable
- Background music (independent variable) affecting mark (dependent variable)
- Place/time of testing (nuisance variables) affecting mark (dependent variable)
Explain the difference between an experimental group and a control group, and give examples?
Experimental group:
- Group receiving the important level of the independent variable
- e.g. students listening to music as they study
Control group:
- Group that serves as the untreated comparison groups
- Group receives comparison level of the independent variable
- e.g. students not listening to music as they study
What are parametric tests?
- Are based on assumptions about the distribution of measures in the population; a normal distribution is usually assumed
- If comparing more than one group, the groups should have equal variance (homogeneity of variance, e.g. are the variances similar)
- Parametric tests are powerful but can be used (e.g. when data doesn’t meet the underlying assumptions of tests)
What are non-parametric tests?
- They don’t make assumptions about population distributions/distribution free tests
- Lower in power and less flexible than parametric tests
Should parametric tests be used whenever possible?
- Yes!
- More are quite robust and limitations well documented
- Possibly use transformations (e.g. logs) to normalise data distributions
How do you test for parametric assumptions, and why?
- Kolmogorov-Smirnov test
- The determines the likelihood of the data belonging to a normal distribution (given as a p value)
To test for homogeneity of variance:
- Levene’s test
- This determines the data sets are from the same population
Explain homogeneity of variance
- If the data isn’t significantly different from a normal distribution and there is no significant difference between the variances of the samples - there is homogeneity of variance
- Therefore, we can go ahead and perform a parametric test
If there isn’t homogeneity of variance, we need to perform the equivalent non-parametric test
Explain parametric and non-parametric test in terms of testing for significant differences
- Both types of tests are available to test for significant differences between data sets
- Parametric tests make assumptions about population parametric (e.g. are distribution dependent)
- Parametric tests require interval or ration scale data
- Violation of test assumptions lead to erroneous interpretations of the data
- Non-parametric tests can be used as alternative to parametric tests
- Non-parametric tests make no assumptions about population
- Non-parametric tests can use data at nominal level
- Non-parametric tests aren’t as powerful as parametric statistical tests (can fail to detect differences)
Explain the Chi-Square Test for goodness of fit (X2)
- Used on unrelated data (every participant/case yields data for only one category)
- Used to answer questions about the proportions of a population distribution (e.g. gender bias in the psychology department)
- Used to compare different levels of ONE variable
- Compared the same proportions to population proportions as specified by the null hypothesis
What are observed frequencies?
- The observed frequencies are the number of participants measured in individual categories
- These frequencies are then compared to frequencies predicted by the null hypothesis (the expected frequencies)
Explain working out expected frequencies
- The exact form of these frequencies changes according to what the null hypothesis is
No difference between the specified categories (e.g. the number of men and women is equal)
No difference between the frequency distribution for the observed categories and existing population (e.g. the number of men and women in the computing department reflects the gender balance in the whole university)
Explain interviews
Aim to find out as much as possible about the participants’ experiences and meanings
Explain structured interviews
Often used when a questionnaire is being administered verbally, and may not be useful in exploring the experiences of the participant as fully as using other methods
Explain semi-structured interviews
Allows flexibility on both the participant and researcher
Explain loosely structured interviews
- Loosely structured schedules usually have fewer specific questions and topics
- Can be more useful in focus groups or when other activities are used
Explain unstructured interviews
Unstructured interview schedule is often considered to be a misnomer - how ‘unstructured’ can you be on paper and in your intentions?
What are focus groups?
- Aim to find out as much as possible abut the participants’ understandings and meanings, with more than one participant
- Individuals come together to discuss a topic
- Involves sharing of experiences, ideas, views, etc.
Why do we use focus groups?
- Contextualises collective understandings and sense-making
- Useful in considering peoples’ shared understandings
- Sensitive to points of consensus and disparity
Explain face to face focus groups
- Effort from the researcher: you must act as a facilitator to your participants
- Ensure the topic is followed
- Focus group schedule used - interview schedule, list of questions, topics and prompts for discussion
- Lead discussion - but more them than you
- Needs attention to interaction in ‘the room’
- Acknowledge agreements and disagreements
- Ensure people are respected and heard
Explain online focus groups
- The format may be different, but the content seems relatively stable between face to face and online focus groups
Asynchronous Online Focus Groups
+ More time to think about responses
- Could be technological issues associated with them
Synchronous Focus Groups
+ Technology can provide different types of environments for participants to engage with
- Requires a good and consistent bandwidth, and reliant on individual schedules
Groups in the “virtual world”
+ Avatars may lead to greater engagement and co-creation activities
- Assumes a certain level of skill/ability is needed
What are ‘alternative’ qualitative data collection methods?
- Interviews and focus groups are largely seen as the key type of qualitative data collection
- There are several other ways of gaining insight into people’s experiences, sense-and-meaning-making practices, and perceptions/constructions/views
What are some prompt methods?
Use videos/vignettes/activities/audio to start discussions on a given topic
What is story completion?
- Projective test, completing a story stem
- Allows for participant creativity
Explain how to develop story stems
- Story stems are the beginning of the ‘story’ that participants need to respond to (it needs to be meaningful, but ambiguous enough for participants to respond in a way that means they will draw on their assumptions)
Things to consider:
- Topic: what is it you want to explore? Is story completion appropriate?
- Length: shorter for familiar and straight-forward topics (one sentence)
- Characters: engaging and authentic
- Detail: too much (limits responses), too little (unsure where to take it)
- Ambiguity: can be good, especially when thinking about taken-for-granted knowledge
- First or third person: usually in third person (better for tapping into socially undesirable responses)
- Instructions: need to be clear, depending on what you want to focus on it could be broad or very specific, e.g. ‘Tell us what happens next/how the story unfolds…’ versus ‘What is Maria saying to her friends about her reasons for not attending lectures?’
Explain qualitative surveys
- Predetermined, open-ended questions
- There is scope to combine this data with quantitative responses, purely qualitative surveys are less common
- Less reliant on researcher craft-skills
- Participants can have more control, and consideration over their responses
- Slightly more scope for possible anonymisation in recruitment
- Can suit broad and specific topics of interest
- Usually suits realist, critical realist, or essentialist perspectives
- However, no interaction with subject, nuances of emotion or environment lost
Explain solicited diaries
- Diary writing within pre-defined guidelines, intended for research purposes
- Can be more stringent (strict), e.g. travel practices/food consumption
- (Partial) access to thoughts/feelings of the participants
- More participant control can be useful for sensitive subjects
- Can feel cathartic for the participants-giving voice
- Can be done by writing, through apps, can include photos
Explain media data
- e.g. newspapers, magazines, tv, films, and reader comments
- Ubiquitous and easily accessible
- Highlights common messages about populations/issues
- Taps into our mediated lives, practices, and beliefs
- Pervasive and accessible (but not necessarily easy)
- Need to focus on sampling strategy and justification
Explain online data
- Can (sometimes) use pre-existing ‘naturalistic’ data from online sources in qualitative research (e.g. forums, blogs, social media, etc.)
- Data ‘harvesting’ - using existing forums, chats, blogs, tweets, etc. and analysing that text
- Use existing external cites to host purposive research (e.g. new thread on a forum like Reddit specific to that topic)
- The data is naturalistic, but may not be fit for purpose)
- ETHICS is a concern
Is it personal or public data? Can you make some recognisable by quoting them?
Should you ask moderator/administrator/owner permission?
READ the terms and conditions - not all websites allow researchers to use them
What is thematic analysis?
- Foundation for other types of qualitative analyses
- Process of identifying meaningful patterns (themes)
- A way of ordering and understanding participants’ social world
- Turning the mess of everyday talk into something meaningful
- Researcher is actively involves in the generation and organisation of these themes
- Good for looking across a data set
- Useful for identifying patterns
- Flexible
Theoretical approach
Topic
Data collection method
Size of data set
Good starting point for those new to qualitative analysis
What constitutes a theme?
- Recurrent ideas, topics, statements, etc. that generate a pattern which mat explain or add meaning to a person’s (or group of people’s) experiences
- These patterns (themes) are then brought together into a category which is then labelled by the researcher
- No set rules are available to determine a theme
- A theme must answer the research question
- Can’t quantitatively measure a ‘key’ theme - if a theme contributed to answering the research question, then it’s up to the researcher to decide if it’s a key theme
Explain epistemology (positivism, contextualism, social constructionism)
Positivism:
- Human experience is knowable, universal and objective
- Research as a form of investigation for THE truth
- Direct correspondence between perception and things
- Knowledge is inert and impartial
Contextualism:
- Sits in-between positivism and constructionism
- Akin to critical realism
- Contextualisation of human acts
- No single reality
- Knowledge comes for contexts
- Provisional
- Interest in THE TRUTH, despite this being inaccessible, knowledge can be the TRUTHFUL
Social Constructionism:
- Historically and culturally contextualised account
- Research as a form of investigation of AN ACCOUNT
- Questions tacit, taken for granted knowledge
- Knowledge is (re)constructed through language
- Knowledge is active, and has power
Explain ontology (realism, critical realism, relativism)
Realism:
- A pre-social reality exists that we can access through research
Critical Realism:
- A pre-social reality exists but we can only ever partially know it
Relativism:
- ‘Reality’ is dependent on the ways we come to know it
What are the problematic uses of thematic analysis?
- Was developed as a way to systematise a general approach to interpreting qualitative data
- Have been good examples of thematic analysis, but there’s also evidence of poor practice, characterised by:
A mashing of other approaches, e.g. grounded theory techniques
Use of coding reliability measures
Treating thematic analysis as one approach
Confusing summaries of data domains or topics with fully realised theme - Procedure is often prioritised over reflexive thought and decision-making (e.g. how many codes should I have? Is my coding accurate? Are my themes right?)
- Instead…move towards reflexive thematic analysis
Centrality of researcher subjectivity and reflexivity
Focus on deliberate, and well-thought-out methodological decisions that allowed for exploration, rather than recipe-following
What is reflexive analysis?
- Associated with reflexivity in qualitative research
- Researcher is active, and embedded, in the results
- Reflecting on, and understanding, your position as a researcher in relation to the topic of study
- Thinking about how you think about the object of investigation and understanding the impact you have on how the topic is investigated
- Methodologically, theoretically, and epistemologically and ontologically transparent
- Being embedded in the decision-making of the project, avoiding recipe-like approaches
- Draws on informed judgement calls, rather than a recipe!
Explain the common qualitative analyses (thematic , interpretative phenomenological , discourse, conversational, grounded theory, content)
Thematic Analysis
- Identification of common themes
Interpretative Phenomenological Analysis
- Attempting to understand participants’ experiences from their perspective (through themes which include descriptive, linguistic and conceptual comments)
Discourse Analysis
- Talk as social action - people convey their social position through their languages and language itself is an interaction
Conversational Analysis
- Focus on how interactions are represented via talk and what action the talk represents in naturally occurring conversations (the process of interaction - how it’s managed, constructed, etc.)
Grounded Theory
- Identification of a model/theory generated from the data (no preconceived ideas on what might be found)
Content Analysis
- Count frequency of pre-defined behaviour
What are the qualities of Interpretative Phenomenological Analysis, in comparison to the qualities of Thematic Analysis (e.g. what are the differences between the two?)?
- IPA is a methodology in its own right and adheres to a set of philosophical assumptions; TA is flexible to researcher positionality
- IPA and TA both embrace researcher subjectivity; in IPA, this is explained by the double hermeneutic
First hermeneutic - participant making sense of their experiences
Second hermeneutic - researcher making sense of the participants’ sense-making - Similarities in coding, but they tend to be more detailed and may draw on metaphor, psychological processes, and language use (e.g. pronouns)
- Similarities in thematic structure, but they tend to be more formalised, detailed, and individualised in IPA
- Interviews are usually used in IPA projects as they allow exploration of personal accounting and sense-making
~ TA can be used to help interpret a wider range of data that doesn’t have to be focused on the individual (e.g. focus group data) - IPA assumes that language reflects, to some extent, people’s thoughts, feelings, and beliefs
~ TA is flexible to researcher positionality, so may/may not assume that through language we can understand internal processes - IPA, while interested in personal social contexts, isn’t as focused on broader social structures that act as constructive forces
~ TA can take a broader approach and explore how and why people take up and use broader social discourses - In-built philosophical assumptions (phenomenology, hermeneutics, critical realism)
~ TA is more flexible to researcher-orientation - Focus on personal experience and meaning-making. Data is looked at both thematically (across the data) as well as ideographically (on a case-by-case basis)
~ TA tends to look across the data-set, not a focus on the individual - IPA tends to rely on small, homogeneous (similar) samples to allow for depth of interpretation
~ TA can be applied to larger or more varied samples, as the interpretation tends to not be as in-depth or focused on personal meaning-making
When should Reflexive Thematic Analysis be used instead of Interpretative Phenomenological Analysis?
- When the research question is interested in exploring something other than personal experiences and meaning/sense-making
- When data isn’t personal enough
- If the sample is larger or heterogeneous (varied)
- When there is a focus on themes across the data (no idiographic focus/approach)
- Focus is on the individual social contexts, rather than broader social structures
What are the qualities of Discourse Analysis, in comparison to the qualities of Thematic Analysis (e.g. what are the differences between the two?)?
- DA tends to be heavily influenced by theory, and as such the process of analysis tends to be more conceptual and theory-driven
~ TA coding tends to follow a more practice-based, rather than theory-based, approach - DA has multiple iterations ranging from the specific focus on language-use, to taking a broader approach where language is considered to represent broader social discourses
~ TA isn’t sensitive to the specific functions of language use, but similar theories can be drawn on to consider the taking up of broader social discourses - Associated with philosophical assumptions (e.g. social constructionism/postmodernism)
~ TA is flexible and can draw on similar ‘critical’ theories in interpreting data. This is often referred to as a constructionist TA - Language is considered to have a social function, that individuals are active in using to serve a social/performative function
~ TA can draw on similar principles, but may be more inductive (data-drivers), and not draw as heavily on theory
When should Reflexive Thematic Analysis be used instead of Discourse Analysis?
- If the researcher is new to qualitative methods
- When wanting a less theory-dense approach
- When the research question isn’t solely focused on discourses, and particularly social constructionist approaches
- When there is an interest in something other than the constructive power of language
Instructions on Chi-Squared formula (Goodness of Fit and X2 Test of Independence)
For each of the categories:
1. Subtract the number of cases expected from the number of cases observed
2. Square this difference
3. Divide the results by the number of expected cases
4. Add all the values from all the categories
What is the formula for calculating degrees of freedom?
df = (R - 1) x (C - 1)
R - row
C - column
What is a t-test?
- A test that compares 2 means
- A test that involves 2 groups, or one group and a population
- Looks at differences between 2 groups
What are the different types of t-tests?
- Unrelated/independent/between groups
e.g. is there a difference between male and female scores on a numerical ability test? - Related/dependent/paired sample/within groups/repeated measures
e.g. does recall improve after attending “Hypnotic Memory Training” with Paul McKenna?
What type of data is needed for t-tests?
- Interval or ratio
- Normally distributed
Independent test
- Scores are independent
- Homogeneity of variance
What are the necessary features for completing a one sample Wilcoxon signed rank test?
- Looking for differences between data
- Interval and ratio data
- Independent groups
- Non-parametric data
What is a Wilcoxon test used for?
- Used to test a hypothesis about one population median
The median is the midpoint of the distribution: 50% below, 50% above
Explain how to complete a Wilcoxon signed rank test
- To calculate the difference between each observation and the interest value, 100 mg/dl
- You should exclude any differences that equal 0
- To classify and order (ranking) differences by magnitude, not taking into account the sign (+ or -)
- Sum the ranking of positive differences
- Sum the ranking of negative differences
- Select the smallest number of these sums and call it T
What are the necessary features for completing an independent t-test?
- Looking for differences between two groups of data
- Ratio data
- Repeated measures/independent groups
- Parametric data
What are the necessary features for completing a Mann Whitney test?
- Looking for differences between two groups of data
- Ordinal data
- Repeated measures/independent groups
- Non-parametric data (should check all of these)
What are the necessary features for completing a dependent t-test
- Looking for differences between groups of data
- Interval or ratio data
- Repeated measures/independent groups
- Parametric data
What are the necessary features for completing a Wilcoxon matched pairs test?
- Looking for differences between groups of data
- Ordinal data
- Repeated measures/independent groups
- Non-parametric data
What is effect size?
- Simple way of quantifying the difference between two groups
- Emphasises the size of the difference rather than confounding this with sample size. It’s easy to calculate, readily understood and can be applied to any measured outcome in psychology
- It’s particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison
Explain Pearson’s r correlation (type of effect size)
- Introduced by Karl Pearson
- Can vary in magnitude from -1 to 1, with -1 indicating a perfect negative relationship, 1 indicated a perfect positive relationship, and 0 indicating no relationship between two variables
- Cohen gives the following guidelines for the social sciences:
Small effect size, r = 0,1
Medium effect size, r = 0.3
Large effect size, r = 0.5
Explain Pearson’s r correlation (type of effect size)
- Introduced by Karl Pearson, is one of the most
widely used effect sizes - Pearson’s r can vary in magnitude from -1 to 1, with -1 indicating a perfect negative relationship, 1 indicating a perfect positive relationship, and 0 indicating no relationship between two variables
- Cohen gives the following guidelines for the social
sciences:
Small effect size, r = 0.1
Medium, r = 0.3;
Large, r = 0.5
Explain Cohen’s d (type of effect size)
- Is the appropriate effect size measure to use in the context of a t-test of means
- d is defined as the difference between two means divided by the pooled standard deviation for those means
- Cohen argues that:
Small effect size = 0.2
Medium effect size = 0.5
Large effect size = 0.8
What is the formula for calculating effect size?
R (effect size) = Z / square root of N
N = total of all observations
(Z value is listed on the test statistics table on the SPSS output)
(N score is the total multiplied by 2, since the test used a repeated measures design)
What is nominal data?
- Data that can only be categorised
- e.g. names of people
What is ordinal data?
- Data that can be categorised and ranked
- e.g. competition places (1st place, 2nd place, 3rd place, etc.)
What is interval data?
- Data that can be categorised, ranked, and is evenly spaced
- e.g. temperature
What is ratio data?
- Data that can be categorised, ranked, is evenly spaced, and has a natural zero
- e.g. height in metres
What is the equation for calculating z-score?
z = (x - mean) / SD
