RMC, W9

Question 1

Quality in quantitative research: Reliability

Answer 1

• There is a need for reliability in quantitative research > reliability essentially looks at whether the same results can be achieved with different researches and on different populations.
○ The more replications you have, the more reliable the method is considered to be (this is linked to the replication crisis in psychology that a lot of classic, more modern studies are no longer replicable.)
○ The researcher should be able to be removed from the research process
- But the problem with using this in qualitative research is that we focus on individual meanings which can be understood in several different ways thus interpretations may not always replicate or match

Question 2

Quality in quantitative research: Validity

Answer 2

• Validity looks to identify whether the researcher has measured what it intends to measure.
○ Different types of validity > e.g. internal (if the concept has been measured and not some additional variable > e.g. making sure depression measure is measuring depression + not anxiety)
• This would be an important means of assessing quality and quantitative research.
• However, the core problem with using this as a means to assess quality in qualitative research is that individual meanings can differ + qualitative data collection largely relies on semi-structured interview guides meaning that there is a lesser standard of standardisation.
- Qualitative research is also sensitive to the cultural context of participants, meaning that often other aspects are considered within the interpretation.

Question 3

Quality in quantitative research: Generalisability

Answer 3

• Generalisability looks to establish the findings can be applied to other contexts outside of the research environment
• In quantitative research, the more generalisable the findings, the more that it can tell us about the phenomenon.
• However, problem w/ using this in qual research is mainly because qualitative research understands help us participant accounts which are contextually and historically bound. > There is little scope for findings, then, to apply readily outside the research context.
- Focus is on the detail of the phenomena

Question 4

Quality in qualitative research (Yardley, 2008)

Sensitivity

Answer 4

• Because qualitative and quantitative methods are different. They tell us different things + explore issues in different ways > would be inappropriate to expect the same standards of quality to be applied to.
  • Qualitative research needs to be sensitive to the context in which it is being produced and to the accounts of participants + ensure that they are being reported on in a sensitive way without misleading.
  • We also need to ensure that we act as ethical practitioners and ensure that participants are aware of the true nature of the study and their rights as participants.
• An inductive approach is needed for data interpretation

Question 5

Quality in qualitative research (Yardley, 2008)

Commitment and rigour

Answer 5

• We also need to appropriately report on the accounts of participants in a faithful way without misrepresenting their voices.
• The data collection also needs to be appropriate to the study.
• So, for example, if you’re interested in exploring personal experiences of sexual assault, perhaps the focus group may not be the best place to explore participant accounts
• Also need to make sure that the analytic technique is appropriate to the types of questions you want answered and that the process is conducted in a competent way.
- There needs to be a clear engagement with an understanding of the topic

Question 6

Quality in qualitative research (Yardley, 2008) Transparency

Answer 6

• process of interpretation needs to be clear on the analytic process and decision making needs to be retained clear and explicit discussion in the process > this is to account for the particular way that you do your analysis.
• The quote selected need to be appropriate means that they match the point and the interpretation being outlined in the analysis.
- There needs to be a clear discussion of how data fits with the research question and an embedding and addressing of any points of reflexivity.

Question 7

Quality in qualitative research (Yardley, 2008)

Impact

Answer 7

• This is where qualitative researchers need to think about the impact that they have had on those around them and in the field of study
• In applied settings, the needs to be a consideration of how these insights can help address and improve current practises and in theoretical considerations.
• In theoretical considerations, need to think about how these results push theory forward
• Are there issues that could be taken away from this research that have a lasting impact on others lives or inform future work that may lead to this?
• Research as a political act is usually done for a purpose (whether it’s genuine interest in the topic and wanting to bring about change) > the impact of research stretches beyond the person who’s conducting it and there needs to be a consideration as to whether the research would likely be used politically to help, but also to hinder populations.

Question 8

Addressing quality (Cresswell & Miller, 2001)
Triangulation

Answer 8

• Can be done in two ways > methodological or researcher
• Methodological triangulation: use different data collection methods or analytic techniques to improve the quality of the insights you can achieve about a phenomenon or population.
○ Consistency in themes > greater trustworthiness of findings
• Researcher triangulation: where multiple researchers would analyse the same dataset in the hopes of getting a high rate of agreement between two or more researchers.
○ Consistency in themes > greater trustworthiness of findings
- There are problems with this however > if you’re more inclined towards critical method, then this can actually measure some of the sentiment of reliability as a marker of quality and may not address the importance of researcher interpretation, especially when using more critical methods or looking at the latent level of themes.

Question 9

Addressing quality (Cresswell & Miller, 2001)
Audit trail

Answer 9

• This is the process of producing a clear overview of the research decisions made > e.g. field notes
• Field notes involve taking notes after the interviews/focus groups + identifying issues you felt were particularly pertinent to the participants > e.g. commenting in how the interview went etc…
• Also need a clear + evidenced analytic process
• Audit trail: the process of providing a clear account of your procedure, analytic process and any decisions made > e.g. why you used semi-structured interviews, why it matters for your topic, what will it contribute?
• Deviant cases: not all data will fit together on systematically, there will be cases that don’t fit and don’t make sense. And there needs to be an appreciation of these what we call deviant cases.
May perhaps offer an alternative account of experiences or perceptions.

Question 10

Addressing quality (Cresswell & Miller, 2001)
Reflexive practice

Answer 10

• need to ensure that you have provided a clear discussion of the reflexive processes of doing research, and you need to be thinking about whether or not the design was well considered and justified
• Example of reflexive practice include considering: are concepts meaningful? Are the methods appropriate?
○ have you analysed your data carefully, faithfully and well?
○ are your conclusions while supported and are they applicable?
- Reflexivity happens throughout the research process + also think about what you’ve brought to the research as well other assumptions that you had of the topic the population or have you also learnt something about the topic + population, thinking about how it’s changed us as researchers

Question 11

Addressing quality (Cresswell & Miller, 2001)
Collaboration

Answer 11

• This is where the voices of participants and other experts are used to develop the research insights.
○ for example, some researchers will ask participants to check their transcripts or themes to make sure that they are happy that the researcher has been faithful to their voice
○ this could lead to censoring of the participants voice > they may not like some of the interpretation and ask for it to not be included.
○ this could have ramifications for the quality of the final product, but also it may further allow participants to be happy with their contribution
• researchers may also call on experts in the fields, such as practitioners, to help and understand and work through data.

Question 12

Critiques: Lacking objectivity

Answer 12

The critique
• Qualitative research is critiqued for lacking objectivity
○ Objectivity is associated with limited researcher-influence
○ Objectivity is considered to be a marker of ‘good’ scientific enquiry
• Thus suggesting qualitative research sits outside of good scientific enquiry
• This may be evidence of social constructionism in action > objectivity = truth, fact, neutrality, and reality
• Whereas subjectivity = tentative, less-than-real, biased, non-useable things in psychology
• qualitative research is subjective in nature and therefore biased and unhelpful in telling us things about human experience according to the notions of objectivity.
The response
• In qualitative research the methods are subjective but this is never denied but rather embraced.
• Subjectivity is acknowledging and exploring the interwoven role of the researcher, the studied, and the socio-historical-political context in the research process
• Subjectivity is an accepted aspect of qualitative research
○ It is not seen as a limitation
○ It is an acceptable and essential aspect of qualitative research
Addressed through methodology & reflexivity which acknowledge the personal subjectivities of the researcher

Question 13

Critiques: Lacking generalisability

Answer 13

• Importantly, research is always limited. It can never be truly generalisable across all populations, cultures and situations > there are natural limits to the applicability of the research.
• E.g. a study looking at conscientiousness in final year students may not relate to the rest of the student population
• Instead, it is about thinking about how a body of evidence can build up a picture of the phenomenon over time > So using qualitative methods, we can add to that growing body of evidence.
• Even if a qual study is not generalisable (even in quantitative), instead, we can think about the weight of evidence and what the weight of evidence is telling us about phenomenon in populations.
- We also need to think about the idea of science as something that is continual progression and is not necessarily about reaching “the answer”, is about reaching an answer that provides an insights and offers of opportunities for development.

Question 14

Critiques: Sample size

Answer 14

• Sample sizes in qualitative research tend to be smaller than in quantitative research
• The criticism levied at qualitative research is primarily focused on the notion of generalised ability and reliability. The more data a researcher collects, the more reliable the data tends to be, and the more generalisable it is therefore likely to be in qualitative research
• There needs to be an adequate and detailed justification for sample sizes
• However, developing a justification for sample sizes is often difficult and an unwieldy task for researchers to explain and justify

Question 15

When to stop data collection

Saturation

Answer 15

• Movement by researchers to use an ocean of saturation
• Saturation is concerned with stopping data collection based on the premise that no new ideas were being generated > themes were saturated
• some researchers went so far as to provide numerical or statistical explanations for why and when projects projects would stop recruited based on expectations, when data would be saturated
• This is a highly criticised concept and not aligned to more critical approaches in qualitative research. This premise moves quite strongly towards statistical remits of knowledge, and some would say that the concept of saturation is inappropriate for qualitative research.
• Saturation focus on no new insights being developed.
○ Vague and impractical (O’Reiley & Parker, 2012) > no guidance for researchers
○ Not consistent with critical qualitative approaches (Braun & Clarke, 2019) > assumes we can know something absolutely about data (data exists + is up to us to remove it)
Fails to address issues like diversity, theoretical context, and specific qualities of the study (Sim, Saunders, Waterfield, & Kingstone, 2018)

Question 16

When to stop data collection

Priori sample size

Answer 16

• Involves rules of thumb: following similar studies in how many people were recruited
• Numerical guidelines: point of saturation. E.g. 12 interviews reaches saturation
• Statistical formulae: determining the point of saturation through statistical formulae and probability.

Question 17

Issue with priori sample size

Answer 17

• Many issues with this approach:

1. Ontological status of the theme: There is an assumption, particularly within the statistical approaches, that themes reside in the data ready to be mined > isn’t always the case > If you’re coming from a critical perspective especially, this is certainly not how you think about yourself as a researcher. You may see yourself of more as somebody who goes on a journey with the participants or is even cultivating at these generative ideas and insights not just taking themes out of their interview readily
2. Themes as instances: there is an assumption that themes are consistently defined throughout the data and that they cannot change due to the iterative process of the research > can be quite considerable variation in participants accounts.
3. Analytic context: assumes more of a deductive analysis, rather than inductive + some models assume that the analysis process is fixed and collection and analysis techniques follow a consistent shared approach.
4. Diversity of ppts: there’s an assumption of the homogeneity of participants, which can be the case > But there can be quite a range of diverse opinions and experiences being talked about.
5. Interdependent determinants: An assumption especially in conceptual models that determinants work separately on a uninfluenced by each other.
6. Statistical assumptions: assumptions that themes are independent and inflexible, so that built on the assumptions of statistical distributions and that this level of control is then reflected in participant recruitment (but this isn’t the case in qual research)
7. Assumes generality: an assumption that calculations made can be put into another study, and the specifics of the study is then lost. Most studies are all very different with differing focus points + requirements > assumptions of generality ignore the specific nuances of each individual qualitative research project.

Question 18

Data adequacy

Answer 18

• Focusses more on whether the data is adequate to meaningfully address the aim(s) and research question(s)
• Researcher judgement may be most important in choosing sample size > Not necessarily about rules of thumbs but using reflexivity to decide
• Researcher feels there is enough data to answer the research question (Morse, 1995, 2000)
• Preference to address issues like the nature of the topic, the intended analysis, and the quality of the data.
• Reflexive process wherein the researcher needs to make the decision (Braun & Clarke, 2019)
○ Meaning is generated through analysis, so judgements about stopping are part of the research process
Usually a decision made in-situ during the analysis process

Question 19

Methodolatry (Chamberlain, 2000; Willig & Stainton-Rogers, 2013)

Answer 19

• Methodolatry is a concept based on the idea that researchers sometimes revere or worship a particular method above all others.
• Whilst this is usually reserved for quantitative approaches, qualitative researchers are not immune.
• Under this concept, the method would be prioritised over the phenomenon or populations of study.
• Methods are assumed to be in the realm of experts so meaning methods can only be conducted by them
• There are strands within this where qual research is seen as a political act such as social empowerment through participatory action research.
- Others would argue that qualitative research should be approach agnostically, meaning that there is a focus on increasing knowledge without the trappings of theory and approach within qualitative research.

Question 20

The qualitative toolbox

Answer 20

• Under this assumption, methodology, theory, ontology under epistemologies exist as creative toolboxes and the researchers should select the most appropriate to answer the questions.
• Known as bricolage, known in art or literature as a construction or creation from a diverse range of available things.
○ E.g. IPA comes with its own set of assumptions. To see IPA is part of this tool box may mean a few things like using idiography + focusing on experience specific to IPA in analysis w/o feeling constrained to theoretical assumptions of this approach
• the idea is that we think of qualitative approaches as tools and we literally select the most appropriate to answer the questions.

Question 21

Mixed/Multi-method design

Answer 21

• Mixed method designs usually have two or more methods, but one will take precedence, while others are subsidiary to it.
• Multiple methods, however, looks to present methods as equally informative as each other + equally weighted in importance.
• Qualitative methods used alongside/to support quantitative methods (Lambert & Loiselle, 2008) > e.g. quantitative methods identify something which are investigated later qualitatively or vice versa
• Multiple methods has also been used as a means of triangulation > So to tell us if our results are consistent across methods of data collection or analysis, in most cases, these multiple methods of data collection and analysis form one singular result section, which means that sometimes the nuances of these individual forms of data collection and interpretation are somewhat lost.
- Issues relevant to using multiple qualitative methods, i.e. epistemological and ontological consistency (e.g. Chamberlain et al., 2011) + how do they aim to account for these?

Question 22

Multiple methods in qualitative research

Answer 22

• Qualitative methods are often considered to ‘fit together’ unproblematically (Barbour, 1998; Lambert & Loiselle, 2008) > integrate well together
○ qualitative methods are varied and include varied assumptions of the participants accounts + what analysis tells us
• What you can be left with is studies which may all have come from a different array of epistemological positions or very different data has been produced.
• Qualitative research encompasses several, sometimes conflicting theoretical, epistemological, ontological and analytical frameworks (Barbour, 1998)
• Multiple methods endorse the combination of different methods within one research project (Morse, 2012) but things need to be considered before starting
○ that each method will be, if appropriate, treated with equal weight that when interpreting and reporting that not one method, participant group, or analysis takes a fore > can be difficult to do + at times certain methods/ppts may be heard louder if appropriate
○ also needs to be a consistent theoretical orientation > a consistent epistemological and ontological position
○ There also needs to be sensitivity to the limits of the data and an awareness that sometimes they fit together well. (So an interview may leave gaps in ideas which are addressed through a focus group + can fit together)
• However, a caveat against this is bricolage > bricolage sees qualitative methods more as a tool kit and that you and what you have in this toolkit is used for different purposes and at different times.
in these cases, you then forgo your concerns over theoretical issues and then some actually go as far to say that theoretical concerns hold qualitative researchers back from being as innovative as it could be.

Question 23

Why should researchers use multiple methods in qualitative research?

Answer 23

• Lambert & Loiselle (2008) argue there are 3 main reasons for this
• Pragmatism: Some ppts may only wish to participate in certain forms data collection. So, for example, a preference may be shown for interviews over focus groups.
• Compare perspectives: Researchers wish to understand the differing experiences of different groups about the same phenomenon.
• Completeness: Researchers predominately conducted multiple method research projects in order to generate a more comprehensive understanding of the topic.
• Through use in multiple methods, design researchers may aim to establish a more comprehensive understanding of a given phenomenon further.
• The different methods produce difference kinds of data and knowledge and are useful for answering different kinds of questions (e.g. Heary & Hennessy, 2006) > e.g. focus groups are appropriate for exploring a range of views and generating consensual understandings whereas interviews are more appropriate for research questions focussing on individual experience
○ rather than the data tell a different stories, different datasets may cover areas that others have failed to consider or are not sensitive to
• Sensitivity to the types of data collection and analysis
• Analyses were appropriately associated with the aims of the studies
most effective way to ensure compatibility across different datasets is through epistemological consistency, wherein the researcher would hold the same epistemological, ontological and theoretical position across all types of data collection and analysis.

Question 24

Person-first

Answer 24

• This approach originates from autism research and has been used to address stigma and mental health and addresses the issue of individuals with a mental or physical health condition being defined by that condition.
• Largely has been a push for “put the person before the condition” (Braun & Clarke, 2013, p.300) > person first approach
• Represented a paradigm shift towards inclusivity, representation, and consultation (Crocker & Smith, 2019)
• Person-first approach involves encouraging practitioners in attempt to decrease stigma and increase compassion with the ultimate goal of improving the efficacy of care through more positive relationships with clients > but also ensuring that access to care and support is equitable.
• However, the person first approach whereby the individual is not defined or identified by their condition is not universally accepted

Question 25

Identity first

Answer 25

• “Merely being associated with a disorder, condition, or socially abnormal behaviour means that potential stigma is always present” (Sainsbury, 2009 > has ASD herself)
• Sainsbury suggested that merely being associated with a disorder condition or socially abnormal behaviour using her terms means the potential stigma is always present.
• Furthermore, she argued that putting the person first maybe makes a disorder an addition and enhances the shame and stigma.
• Person-first language is used more frequently with those who have a physical or mental health condition (Gernsbacher, 2017)
○ They argue that the use of person first language in scholarly writing may actually accentuate stigma rather than attenuate it.

Question 26

What approach is appropriate?

Answer 26

• Braun and Clarke encourage researchers use a terminology that socially marginalised groups use to describe themselves.
○ essentially asking people what term they’d prefer to be referred by.
• Gernsbacher (2017) recommended that scholars and practitioners should be educated to the use and ethos behind person first language, but also suggested referring to all persons with those with and without physical or mental health conditions.
○ Refer to all persons, those with and without physical or mental health conditions, with person-first language (Gernsbacher, 2017) > e.g. a child with or without a developmental delay
Can be difficult to balance + should be carefully considered > If you feel that this topical issues manifested in your study, you should reflect on it, will impact on how you talk about that phenomenon, how you represent your participants talk, and how you operationalise your terms.

Question 27

Open science

Answer 27

• Open science was a movement in response to poorly conducted research and primarily quantitative research.
• Open data allows for the accessing, use, modification, and sharing of data > greater integrity and rigour, innovation, and collaboration
• Often is now an expected element of reach of the research process from funders that researchers essentially make their data open access so they will upload their data to a repository for other people to scrutinise and to assess the validity of the claims that have been made as a consequence of analysing the data and then having robust and replicable methods sections and documenting the stages of analysis.
• hoped that this would mean that that same analysis can be conducted with the same results resulting from it.
- Open science is well received by quantitative researchers but not as much by qualitative researchers
• Chauvette et al (2019) explored three key areas where qualitative research may not be as amenable to open data as quantitative research.

Question 28

Open science: Chauvette et al (2019)

Epistemological issues:

Answer 28

• researchers question how appropriate is to use qualitative research outside of its original context and the researchers epistemological position and the specific cultural and historical context to which this research developed, from which the researcher also had a role in in the generation of that data as well
○ researchers have conducted secondary analysis of their own data, felt less intellectually engaged with the data and felt unable to recapture the context within which that data was generated.
○ As we value subjectivity in qual research, is it really appropriate for those sitting outside of the research project to re-analyse it > what happens w/ this analysis? Whose is considered to be more robust?
○ Does this extend critique of qual research in lacking rigour? ? Are we just adhering to quantitative prescriptions of quality in research

Question 29

Open science: Chauvette et al (2019)

Methodological issues:

Answer 29

• Not all approaches are amenable to secondary analysis. For example, IPA, where the researchers inextricably linked to the data itself through the double hermeneutic or approaches which rely on field notes amongst participants, are involved in the study through collaborative approaches such as Participatory Action Research.
○ reflexivity is an integral part of qualitative research, which helps to bring meaning and understanding to our interpretation. It is difficult, therefore, to consolidate individual accounting and reflexivity with secondary analysis.

Question 30

Open science: Chauvette et al (2019)

Ethical and legal issues:

Answer 30

• Harm to participants, confidentiality, and anonymity > individuals accounts and sense making and sometimes incredibly personal and sensitive stories. Researchers need to think about the appropriateness of why they would share these accounts. Often, participants are sharing accounts of their personal stories for one particular purpose > you also need to be incredibly careful around anonymity and confidentiality (harder to anonymise than quantitative because detailed accounts are obtained)
  ○ Anonymous using a transcript for snippets to be used in analysis is very different to anonymizing and unti transcript full of personal stories.
  • open science is seen as a very positive thing, but primarily from a quantitative perspective, we perhaps need to think about it a little more carefully when thinking about qualitative projects.