Seminar 2 Flashcards
Which of the following components are necessary for something to be an experiment? Mark all correct options.
a) Observation of outcome
b) Control of background variables
c) Elimination of disturbing variables
d) Intervention on target variable
e) A control group
Answer: a), b), d)
Randomization by itself does not guarantee that the division of relevant background variables are evenly distributed between test and control group. Why is that? Mark all correct options.
a) After one has randomized a sample into control and treatment group, there is the possibility that new background variables randomly enter the experimental system in an uncontrolled way.
b) Even though the probability might be very small, there is always the mathematical possibility that a completely random distributor might distribute objects into control and test group too unevenly. This is because a sample is always a finite set of objects (especially if the sample is small).
c) Assuming there are no problems when randomizing the sampled objects, it is still not guaranteed that the sample was unbiased towards some relevant background variable to begin with, since the sampling might have been non-random.
Answer: b), c)
What is the purpose of randomizing one’s distribution of individuals to the test- and control group? Mark all correct options.
a) It reduces the effect of selection bias.
b) It allows for a better representation of the population, from which the sample of individuals is extracted.
c) It is a means to achieve a constant distribution of background variables between the groups.
d) It facilitates blinding.
Answer: a), d)
What is confirmation bias (i.e., the interpretation problem)? Mark the correct option.
a) It is when two or more observers interpret the same observation in different ways.
b) It is when an observer influences the observation and changes the result.
c) It is when the observation, results of some test, or data is read incorrectly because of some psychological properties of the observer.
d) It is when an observer makes interpretations of what is being observed, for instance social behavior between people.
Answer: c)
Which of these are examples of confirmation bias? Mark the correct options.
a) The doctor uses a test whose operationalization has a low quality that makes the result seem more significant than it is.
b) The doctor has done research on the drug for many years, has an expectancy that it will work as intended, and interprets the drug’s effects to be greater than they actually are.
c) The doctor purposely picks evidence that confirms that the drug works as it should, while disregarding contradicting evidence.
d) The doctor knows from previous tests that the medicine does not affect chronic pain, and therefore does not test the medicine.
Answer: b)
What is the observer effect (i.e. the influence problem)? Mark the correct option.
a) It is when an experimenter intentionally changes the independent variables in an experiment.
b) It is when an operationalization is too theory-dependent.
c) It is when some observers are influenced by previous knowledge that might be false.
d) It is when the observation, in some sense, changes the object of study.
Answer: d)
Which of these are examples of observer effect (i.e., influence problem)? Mark all correct options.
a) When using improper statistical tools make an insignificant result appear to be significant.
b) When an experimenter exaggerates their results in order to make people think the results are more groundbreaking than they actually are.
c) When the act of using a speedometer slows down the vehicle’s speed.
d) When asking the question “How do you feel?” affects the subject’s mood.
Answer: c), d)
What is single and double blinding? Mark all correct options.
a) Double blinding reduces the interpretation problem.
b) Single blinding is a way to reduce theory dependence.
c) The conditions for single blinding are fulfilled if the test subjects in a medical trial do not know whether they’re given the real treatment or a placebo.
d) Single blinding is when only the test subjects know which group (test or control) they belong to.
e) Single blinding is an elimination method.
Answer: a), c), e)
Suppose that you are designing an experiment to measure the effects of Yoga exercise on flexibility in arms and legs. Which of the following are correct applications of some of the steps in Mill’s method of difference? Mark all correct options.
a) Dividing test subjects into two groups, making sure that training background and other relevant factors are evenly distributed.
b) Seeing that both groups increase their flexibility equally and concluding that “Yoga increases flexibility in arms and legs”.
c) Doing an intervention by having both groups do the same amount of Yoga for one month.
d) Formulating the hypothesis that Yoga increases flexibility in arms and legs.
Answer: a), d)
Which of the following are true statements about internal validity? Mark all correct options.
a) Control is a precondition for internally valid inferences.
b) If the conclusion I infer is a causal effect from, or heavily correlates with, the independent variable in my setup (and not some uncontrolled background variable(s)), then the inference is internally valid.
c) Internal validity is a property belonging to an inference between the independent variable and the dependent variable.
d) Internal validity is a property belonging to experimental systems; they have this property if enough background variables are controlled.
Answer: a), b), c)
Feedback:
a) Without control, it is difficult to make internally valid inferences because you cannot be certain that the observed effects are genuinely caused by the independent variable and not influenced by other factors.
b) An inference in a scientific study (an experiment, an observational study or a model) is internally valid, if the relation between the studied factor and observed effect inferred from the study is indeed true and is not confounded by uncontrolled background factors.
c) Internal validity is primarily concerned with the causal relationship or inference made between the independent variable and the dependent variable in an experiment.
Which of these options would (most likely) have a negative effect on the internal validity of the inferences drawn from an experiment? Mark all correct options.
a) An influence problem.
b) Interpretation problems.
c) Not using single blinding when testing the opinions of people.
d) A large amount of experimental errors.
e) A large amount of controlled variables.
Answer: a), b), c), d)
It is useful to distinguish between problems relating to the operationalizations used in a study, and problems relating to the design of the study. Mark those options that are examples of poor experimental design and not a problem with the operationalization.
a) Failing to randomize between test group and control group in an experiment where people are asked to give their opinions on a survey.
b) Using an untested and non-standardized questionnaire for measuring peoples opinions.
c) Failing to make sure that both the test group and control group in an experiment have the same and constant temperature before performing an intervention, resulting in a misleading temperature change when the intervention is performed.
d) Failing to calibrate the thermometer before measuring the temperature of the test and control samples after performing an intervention, resulting in a temperature change.
Answer: a), c)
General Feedback
Remember - operationalizations are ways to measure something and generates data. For instance, a scale used in an experiment might have a systematic error, but is used in a (otherwise) well designed experiment. You can also have a top-notch scale in a poorly designed experiment. These are different types of problems, and this question is about separating them.
What is external validity? Mark the correct option.
a) Justified inferences drawn from experimental data to an experimental sample.
b) Justified inferences drawn from the result of a scientific study to the target of the study.
c) Justified inferences drawn from a population to a sample.
d) Justified inferences drawn from a population to some other population.
Answer: b)
A researcher is aiming to improve an existing exercise app by developing a new version of the app. The hypothesis to be tested is: People using the new app version will exercise and enjoy their exercise more. The research process will start by locating people who use the current version of the app, ask and receive their informed consent about participation, and then let half of the consenting participants use the new version of the app. Statistical data will be collected from participants’ smartphones, and the participants will also be asked to answer a survey asking for a qualitative assessment of the app.
Which of the following are possible method choices whose relevance is related to internal validity (as compared to external validity)? Mark all correct options.
a) Using randomization for the selection of participants of the study into treatment and control group.
b) Checking that the chosen participants do not have any upcoming plans that obviously risk invalidating their participation (e.g. someone is planning to run a marathon and will consequently train substantially more the upcoming months).
c) Only doing the intervention on the treatment group.
d) Making sure that the survey questions are understood unanimously among the participants (e.g. the questions are not too ambiguous and/or vague).
e) Conducting the study in 2 countries each from Europe, Africa, Asia, North- and South America, and Oceania.
f) Making sure that they have a sufficiently large sample that is sufficiently similar to the real world population.
Answer: a), b), c), d)
