Week 6 - Using Questionnaires in Research Flashcards
It is generally easier to test more participants using questionnaires than using objective measures - Why?
- Questionnaires are cost-effective, scalable, and simple to distribute, making them an efficient way to gather data from a large number of participants.
- They can be administered online, on paper, or through other accessible formats, which reduces logistical challenges compared to objective measures (e.g., lab tests or clinical assessments
Questionnaires can offer the chance to…
Assess the success of clinical interventions, and
set more realistic goals for intervention
- Questionnaires can collect subjective feedback, monitor changes over time, and evaluate the outcomes of clinical
interventions. - The data obtained can help in setting more realistic and
tailored goals for future interventions based on participants’ responses.
If you want to use a questionnaire to assess the suiccess of a clinical intervention, you should…..
Administer the questionnaire to a group of
participants, then administer the intervention, then administer the same questionnaire to the same participants at a later time point to see if performance has changed
- This sequence ensures that the questionnaire is used as a pre-test (before the intervention) to establish a baseline, and a post-test (after the intervention) to assess whether the clinical intervention led to changes in performance, attitudes, or other outcomes.
- By comparing responses from before and after the intervention, you can evaluate its effectiveness
For a questionnaire to have content validity it should be…
assessed by experts in the field to ensure it contains questions that cover all aspects of the psychological construct that is to be tested.
- Content validity refers to the extent to which a questionnaire, test, or assessment adequately covers all aspects or components of the psychological construct or concept it is intended to measure.
- For example, if a questionnaire is designed to assess “anxiety,” it should include questions that cover all relevant aspects of anxiety, such as physical symptoms, cognitive effects, and emotional experiences.
- A questionnaire with strong content validity ensures that it fully captures the concept being measured, without omitting critical elements.
- The criteria for assessment should be clearly and systematically set out. The boundaries and definition of the construct needs to be clarified. The criteria should refer to current theory and empirical articles and be assessed by experts,
For questionnaires, test-retest reliability is….
administering the questionnaire to a group
of participants, then repeat the questionnaire with the same group at a later point in time. Compare the responses at the two time points.
- Test-retest reliability refers to the consistency of a
questionnaire or test over time. In other words, it evaluates how stable the responses are when the same participants complete the questionnaire on two different occasions under similar conditions. - To calculate test-retest reliability, researchers compare the responses from two time points. A high level of similarity between the responses suggests that the questionnaire is reliable over time
for questionnaires, inter-rater reliability is…
comparing how two individuals raters score judgements of another subject, to check if they give similar ratings or if one is more “harsh” than the other.
- Inter-rater reliability refers to the degree of agreement or consistency between two or more raters (or observers) when they independently assess or score a particular subject, behavior, or response.
- It is a measure of how much raters are in sync when evaluating the same thing.
- High inter-rater reliability means that the scores given by different raters are similar, while low inter-rater reliability suggests discrepancies between their judgments.
- Examples include as clinicians assessing anxiety in a patient, or teachers watching video footage and assessing a child’s performance in a task.
- Inter-rater reliability can be checked by comparing the scores given by the two raters for the same subject.
- This can assess if one gives systematically higher (better) or lower (poorer) scores (is easier-going or harsher in their assessments)
What is back-translation used for?
to check that the meaning of the questions is not altered during translation.
When using a questionnaire in other languages, care must be taken that the meaning of the questions is not altered during translation
- Back translation is a technique used to
ensure that the meaning of a questionnaire remains accurate and consistent after being translated into another language. - It involves translating the text from the original language to the target language, and then independently translating it back into the original language.
- The two versions in the original language are then compared to identify any discrepancies or changes in meaning that may have occurred during translation.
- This method is particularly useful when creating multilingual questionnaires to ensure that all versions convey the same intent and meaning.
why might we want to reverse-score participant responses?
to make subsequent analyses more intuitive, so that higher scores denotes greater perceived ability.
- If we have questionnaires where some questions are phrased negatively or in the opposite direction of the construct being measured, we would need to reverse score those questions.
- This ensures that all items contribute to the final score in a consistent way, making interpretation and subsequent analyses more intuitive.
- After reverse-scoring, higher scores on the questionnaire consistently reflect a greater degree of the
measured construct (e.g., higher perceived ability, greater satisfaction, etc.)
Why might we use questions worded in different directions?
this can be done to improve the quality and reliability of the data collected
Capturing the Full Spectrum of the Construct:
Psychological constructs often have complex dimensions. For example, a measure of confidence might include both “I feel confident in my abilities” (positive wording) and “I doubt my abilities” (negative wording). Reverse-worded items help capture these nuances of the construct being tested and ensure that all aspects of the construct are being assessed.
Avoiding Response Bias:
Some participants might fall into patterns of responding, like always agreeing (“acquiescence bias”) or always disagreeing, without carefully considering each question. Including reverse-worded items forces participants to think more critically about each question, reducing this type of bias
Balancing Questionnaire Tone (related to response bias) above:
Including reverse-worded items can make a questionnaire feel more balanced. A one-sided questionnaire (e.g., all positively worded
statements) might unintentionally bias participants into perceiving the questionnaire as leading them toward a specific response.
Checking Attention and Validity:
Reverse-worded items can serve as an internal check to identify inattentive or careless responding. If participants answer inconsistently across positively and negatively worded items, it could indicate that they aren’t paying close attention, but is only one reason we might
word questions in different directions.
You are deciding whether to conduct face-to-face interviews to fill out a questionnaire, or to send out a mass email and ask participants to fill out the questionnaire themselves. Your colleague says it does not matter what method you use. Are they right?
No. The two methods have advantages and disadvantages e.g. the presence of an interviewer can help clarify questions, but may also bring their own biases into the interview situation
Face-to-Face Interviews (Advantages and Disadvantages):
Advantages:
- The presence of an interviewer allows participants to ask for clarification if they don’t understand a question, leading to more accurate responses.
- Interviews can capture non-verbal cues, such as body language, which may provide additional insights.
Disadvantages:
- The interviewer may unintentionally introduce bias, such as leading participants to answer a certain way (interviewer bias).
- Interviews are time-consuming, expensive, and less practical for large-scale studies. Self-Administered
- Questionnaires (Advantages and Disadvantages):
Advantages:
- They are more scalable and cost-effective, allowing researchers to reach a larger number of participants
quickly.
- They reduce interviewer bias since participants complete the questionnaire independently.
Disadvantages:
- Participants cannot seek clarification if they don’t understand a question, which may lead to inaccurate
or incomplete responses.
- Participants might be less motivated to complete the questionnaire thoroughly, leading to missing or
careless responses
One of the potential drawbacks of using questionnaires is that….
Different participants might have different subjective ideas of the situation being described in the questions.
- One of the potential drawbacks of questionnaires is that participants may interpret the same question differently based on their subjective understanding, experiences,
or ideas. - This variability can lead to inconsistencies in the data collected and reduce the reliability and validity of the results. Clear and precise question design is critical
to minimize this issue, but subjective differences can still occur.
Difference between quantitative and qualitative research
- Quantitative research can test associations, relationships and differences.
- Qualitative research can explore the nuanced and complex issues associated with human behaviour
Among Young Adults, do Men and Women have
different types of sexual relationships?
This can be quantitatively researched (and has been) – using socialsurveys and Likert scales.
Furman and Shaffer (2011):
- Women report having more sex/sexual contact with committed romantic partners than men.
- Men report having more sex/sexual contact with casual acquaintances than women.
- Among both men and women, sex most frequently occurs in a romantic committed relationships.
- These are interesting findings, but they don’t reveal the
complexities of such relationships, how people feel about them, how they evolve or initiate.
Understanding Sexual Relationships in Young Adulthood
Qualitative Narrative Analysis – Banker et al., (2010):
Participants used 70 different terms to refer to relationship categories.Highlighted that there is a hierarchy to relationship types:
- Ideal relationship = Sexual and Romantic relationship. “When the two are combined I would call it perfect…if you find a relationship where there is a healthy balance between the romantic and sexual aspects it is a relationship worth holding onto”
- Second place = Romantic relationships. “The difference is that romantic can be those cute
little “lovey dovey” and emotional attachments without sex being involved. For me, a sexual partner is the next step from a romantic relationship; I guess I consider it as a ‘prerequisite’.”
- Third place = Not yet romantic or sexual relationships. “This would be the stage where the term ‘talking to’ would be appropriate. From here the relationship would grow and most likely sexual intercourse would become a factor. A ‘what are we’ discussion would then pursue.”
- Fourth place = Sexual partnerships. “I could have a sexual partnership with girls that are just looking for sex from me. A lot of guys in college maintain this kind of relationship with multiple girls so that they can just hook up whenever they please and still be free to look for the girl that they want to be involved in relationship with.”
