Midterm Lec 6a Flashcards
Quantitative research
Data are numbers and the people are subjects
Randomized control trials, chart studies
Qualitative research
Field research, case studies, participant observation and the people they study are called participants
3 key characteristics of experimental research design
1) independent variable that is manipulated
2) control of all other variables (dependent)
3) observation of the effect of the independent variable on the dependent variable
Advantages of experimental design
CRAE
Convergence,
replication,
adjustment of variables,
establishment of cause and effect relationship
3 criteria for establishing cause and effect
1) the cause must precede the effect in time
2) the cause and effect must be correlated with eachother
3) the correlation between cause and effect cannot be explained by another variable
Disadvantages of experimental design
cost
inability to generalize
securing cooperation
depending on the type you choose, it can be quite complicated to design and implement
Control or situational variables
Variables that the researcher may not be able to manipulate, exclude, remove, alter
—>Preferred way to overcome this is to hold the variable constant by the researcher
An auto manufacturer wants to know now bright brake lights should be in order to minimize the time required for the driver of a following car to realize that the car infront is stopping
Independent? Dependent? Control?
Independent, intensity brightness of brake light (manipulated)
Dependent, time of onset brake light until depression of brake pedal ( observed)
Control, colour of brake light, shape, force needed to depress pedal (hold constant)
A pigeon is trained to peck a key if a green light is illuminated but not if the light is red. Correct pecks get rewarded by access to grain. Independent? Dependent? Control?
Independent, colour of light red or green.
Dependent, number of pecks
Control, size of key and intensity of light.
Experimental group
The group that receives the treatment
Control group
The group that does not receive the treatment
Subjects
The name for the individuals in experimental research
Advantages of subjects/groups
Advantages; no chance that one treatment con contaminate another since the subject/group only receive one treatment
- each subject is compared to himself/ herself therefore differences observed are not because of differences between subject.
Disadvantages of subjects/groups
Disadvantages; concern that there is a possibility that the subjects or groups are different enough to influence the effectsof treatment.
- carryover effect: testing subjects in one condition has an effect on their testing in another condition.
How to overcome disadvantages in subjects and groups.
- Randomization of treatment levels
- counterbalancing the treatment levels, systematically vary the order of the conditions to distribute the effects of time eg practice effects, fatigue.
Random sample
A sample from the population that has been selected in an unbiased way.
Each person has an equal chance of being selected.
Random assignment
Subjects are assigned to conditions in an unbiased fashion.
Random groups design
Subjects are randomly assigned to conditions in the between subjects1 group design
Matching
- Treatment, experimental subjects are matched with a central subject
- common “matching” items: age, gender, weight, height, IQ, years of schooling
Matched group designs
Subjects are matched on some variables assumed to be correlated with the dependent variable and then randomly assigned to conditions.
Matching cautions
What characteristics or items do you match on?
Can you locate exact matches?
- true matching becomes almost impossible because there will be personal, genetic, envioumental, influences that the researcher simply can’t control.
Blind assignment
-The subject does not know whether they are in the treatment or control group
-the researcher knows what group the subject is in.
Double-blind assignment
Neither the researcher collecting the data or the subject know which group the subject belongs to.
Pre-experimental design types
-Lacks random assignment
- uses short-cuts that are much weaker than a classic experimental design
-substituted for when a researcher cannot use all the parts of the classical experimental design
- has weaker internal validity
Classic/true experimental design has
- Random assignment
- control group
-experimental group - pre-test and a post-test for each group (control + experiment)
Quasi-experimental designs
- This design is stranger than pre-experimental
- variations on classical experimental design
- used when the researcher has limited control over the independent variables
- some have… Randomization but lack pre-test, more than 2 groups
Randomized contra trial
- An experimental design for medical and pharmacutical investigation
- RCT is viewed as the best type of evidence on which to base decisions and establish guidelines
- 2 types of trials within RCT
Parallel group trial (rct)
- Only one set of patients receives the new drug
- comparison is between the two groups
Cross over trial (rct)
- All participants receive new drug
- patients are randomized into an intervention group or a control group, with the intervention group taking the medication for a selected period of time.
- after that period of time, the OG intervention group becomes the control group taking the placebo while the OG control group serves as the intervention group receiving new drug.
Populations
- The total number of people things, or events of interest
Samples
- A subset of the population
-small collection of units from a much larger collection or population - smaller set of cases selected from a larger pool
Representative sample
-the tendency to display variations among its members that are proportional to the variations that exist in the actual population of interest
-the tendency does not exist, the sample is said to be biased
Variability
- Now little, or now much the sample varies when compared to the population from which it was selected
-if there’s lots of variability, you will want a large sample in order to ensure that the sample is reasonable representative of the population - little variability then small sample size will be sufficient.
Sampling techniques
Probability and non-probability
Non-probability sampling
Aka non-random sampling
-used by qualitative researches
-seven types
Probability sampling
- Used by qualitative researches
- sampling is based on theories of probability from mathematics
-often used in experimental studies
-random is the key to sampling
Random (sampling)
- Lacks predictability, without any systematic pattern
-random samples are must likely to yield a sample that truly represents the population - lets researcher statistically calculate the relationship between the same population
Sampling error
- Now much a sample deviates from being representative of the population
Sampling element
- Unit of analysis or case in a population
- “what” are you going to study? People? Animals? Objects? Organization?
Target population
- Specific pool of cases that are to be studied
- a representation of a certain segment
- “who” or “what” specifically is the researcher interested in studying? People with 6 toes? Cats with no whiskers?
Sampling ratio
- ratio of size of sample to the size of target population
- can be expressed as a decimal value or %
- sample: target population
Example: 50,000 people in the target population researcher selects (samples) 150 people from the target population to study
150/ 50,000 = 0.003 or 3%
Sampling frame
- List of cases in a population, or the best approx. Of it.
- the list from where the researcher will attempt to draw his/her sample from
- examples can include: taxation records, telephone directors, drivers license records, voters list, membership lists, etc
- potential problems with sampling frames
Parameter
- A characteristic of the entire population that is climate from the sample
- a quantifiable characteristic or feature of a population
Probability sampling key words
- Sampling element1 target population, sampling ratio, sampling frame, parameter
Measurement
- A repeatable, objective procedure for generating a measure
Scale of measurement
- The set of possible numbers that may be obtained by the measurement process
- all scales have certain characteristics or ‘properties’
1Magnitude
2 Equal intervals
3 Absolute 0
Magnitude
- When a scale has a magnitude, one aspect of the attribute being measured can be judged greater than, less than, or equal to, another aspect of the attribute
Eg if you have an aggression scale of 0-10 and you assign Jane a value of 8 and Joe a value of 5, you would say that Jane is more aggressive
Equal intervals
- The unit of measurement on the scale is the same regardless of where on the scale the unit falls
Eg the difference between 2 inches and 3 inches on a measuring tape would be the same difference between 82 inches and 83 inches
Absolute zero
-a value that indicates that nothing at all is being measured
- eg if the height is measured with a tape measure 0 inches of height is a scale value that indicates no night whatsoever
It is absolute zero
Factors that will affect measurement
Continuous variable and discrete variable
Continuous variable
- Have an infinite number of values or attributes that flow aloes a continuum
- values can be divided into many smaller increments
Es temps ages income, crime rate.
-Discrete variable
-Have a finite number of values
-Limited number of distinct, separate categories
E.g., gender, religion, marital status
Levels of measurement (LOM)
-Suggests that some measures are higher or more refined, while others are crude, less precise.
LOM will:
-Depend largely on how a concept has been defined
-Affect the kinds of indicators chose
-Depend on the limits of the variables
Nominal level LOM (discrete)
-The lowest, least precise LOM for which there is a difference in type only among the categories of a variable
E.g., racial heritage – African, Asian, Caucasian
Ordinal level LOM (also discrete)
-Identifies a difference among categories of a variable AND allows for the categories to be rank ordered as well
E.g., grades – A,B,C,D
Interval level LOM (continuous)
-A LOM that identifies differences among variable attributes, ranks categories, and measures the distance between categories, BUT there is no true zero
E.g., temp – 5, 45, or 90 degrees celsius
E.g., water freezes at 32 degrees fahrenheit, 0 degrees Celcius and 273 Kelvin
Ratio level
-Highest, most precise LOM
-Variable attributes can be rank ordered, the distances between them precisely measured and there is an absolute zero
-The existence of a true zero allows for statements of proportions or ratios
E.g., money income – $0, $10, $100, $500
E.g., 0 means 0
-Discrete variables tend to be nominal and ordinal
-Continuous variables tend to be interval or ratio
-Interval level can always be turned into ordinal or nominal, but not vice versa
-Ratio level can be turned into interval, ordinal or nominal, but not vice versa
Internal Validity
-Ability of experimenters to strengthen the logical rigor of a causal explanation by eliminating potential alternative explanations for an association between the IV and DV
-IV relates to how well a study is conducted
Threats to Internal Validity
-Selection bias
-History
-Maturation
-Testing
-Instrumentation
-Experimental mortality
-Statistical regression
-Diffusion of treatment
-Compensatory behaviour
-Experimenter expectancy
-Sequence effects
Selection Bias
-Groups in an experiment are not equivalent at the beginning of the experiment with regard to the dependent variable
- E.g., hot peppers / heartburn – some subjects may already have a history of heartburn before starting the study
History
-An event or situation that occurs and affects the dependent variable during an experiment
-It is unplanned and outside the control of the experimenter
-More likely to occur in experiments of long duration (e.g., longitudinal studies)
- E.g., Hurricane Katrina, The Great Depression
Maturation
-Natural processses of growth, boredom, etc. that occur during an experiment and affects the DV
- E.g., study measuring cartoon preference
- E.g., study wherein the subjects must complete several written tests over the course of a day
Testing
-The very process of measuring in the pre-test can have an impact on the DV
- E.g., study for hypertension
Instrumentation
Changes in the characteristics of the measurement instrument over time
- E.g., wear and tear of equipment
Experiemntal mortality
-Subjects failing to participate through the entire experiment
-Why is it a problem?
-Very problematic if there are a lot of subjects who withdraw (w/d)
- E.g., study starts with 100 subjects and then 75 w/d
-Very problematic if there are a lot of subjects within a specific group
- E.g., three groups of 20 subjects; 10 people from ⅔ groups w/d
Statistical regression
-Subjects who are at the extreme end of the spectrum on a measurement will tend to move towards the middle
-Aka regression towards the mean (or average)
- E.g., scores on an exam – when there is little room to go up, you tend to come back down due to random errors
Diffusion of treatment / contamination
-Treatment “spills over” from the experimental group into the control group and then the control group subjects modify their behaviour because they learn of the treatment
Compensatory behaviour
-Subjects in the control group modify their behaviour to make up for not getting the treatment
Experimenter expectancy
-Can happen whenever the experimenter indirectly makes subjects aware of the hypothesis or desired results
Sequence effects
-Seen in w/in subject designs
-Response or performance in a later condition may be the result of the subjects’ role in a previous condition
Controlling the Threats to Internal Validity
-Randomization: Real randomization
Matched pairs (not matched groups)
Randomizing treatments or counterbalancing
-Placebos
-Blind setups: Single blind vs double blind
-Reactive effects of testing: Eliminate pretest
-Instrumentation: Calibration and test reliability, Halo effects.
-Experimental mortality: Keep participants
External Validity
-The ability to generalize the findings beyond the study
Location of experiment
-Laboratory experiment: Takes place in a specialized setting or laboratory
-Field experiment: Experimental research that takes place in a natural setting
Population validity
-Refers to the question of whether the responses or behaviours of sx in the research sample can be generalized to the target population
E.g., can you generalize the reaction time of healthy university students crossing the street to an elderly population?
Ecological validity
-Refers to the question of whether the research findings can be generalized to all of the environmental contexts of interest
E.g., can testing sunscreen in a Canadian summer be generalized to the caribbean environment
Experimental realism
Not always a bad thing…
-The experiment is made to feel so realistic that experimental events have a real impact on subjects
-Becomes a problem when the design is weak and the events do not impact the sx
Reactivity
-Arises because sx are aware they are in an experiment and may react differently
Hawthorne effect
-A reactivity effect named after a famous study at an American electrical company
-Subjects responded to the fact they were in a study more than they did to the treatment
Demand characteristics
A type of reactivity in which subjects in an experiment pick up clues about the hypothesis and alter their behaviour accordingly
Placebo effect
-When subjects who do not receive the real treatment and instead receive a placebo or imitation treatment but respond as though they have received the real treatment
Threats to External Validity
Location of experiment
Population validity
Ecological validity
Experimental realism
Reactivity
Hawthorne effect
Demand Characteristics
Placebo effect
Controlling the Threats to External Validity
Selecting from larger population:
- Participants
- Treatments
- Situations
Ecological validity : Does the setting capture the essence of the real world?
Measurement Validity
-How well an empirical indicator (measurement) and the conceptual definition of the construct that the indicator is supposed to measure “fit” together
Reliability
-Reliability means dependability or consistency
-Reliability is consistency across time (test-reliability) across items (internal consistency) and across researchers (interrater reliability)
-Suggests that when the same things are repeated or recur under identical or very similar conditions, the outcome or results should be identical or very similar
-Researchers will often compute statistical analysis to identify / ensure more “specific” types of reliability-based on their research question, their type of research (quantitative vs qualitative) and their discipline of study (e.g., science, arts, humanities, engineering etc)
Stability reliability
-A measure that yields consistent results at different time points assuming what is being measured does not itself change
E.g., if standardized intelligence tests are used to measure IQ, then they will have stability-reliability over the years; however, if starting in 2010 IQ is measured by how many toes you have, the standardized test will no longer have stability-reliability
Representative reliability
-A measure that yields consistent results for various social groups
-Do you think that IQ tests have ‘representative reliability’? Why?
Equivalence reliability
-Reliability across indicators
-A measure that yields consistent results using different specific indicators, assuming that all indicators measure the same construct
E.g., culture specific IQ tests
How to Improve Reliability
-Clearly conceptualize all concepts: Develop clear, unambiguous definitions
-Increase: Try to measure at the most precise level possible -Use refined categories
E.g., high or low (not so good)
E.g., on a scale of 1-10 (much better)
-Use pre-tests, pilot studies, and replication: Test out your measurements before actually implementing them – use “drafts” -Do it once, do it again, and then again.
Use multiple indicators of a variable:
-Several different tests or measures to provide empirical evidence
Probability
-The odds that a certain event will occur
-The likelihood that something will happen
-Given that all things are equal, what are the chances that an event will happen
Probability in Research
-What are the odds that the results we found were not due to chance?
-When researchers state a level of probability, they are indicating how confident they are that their findings are not simply due to chance
-The extent to which the researcher is sure that changes seen in the outcome were because of the intervention, not chance
Probability Math
Probability is expressed as p
-The numerical value that follows is referred to as alpha
-Often when you are reading, you will come across values like:
- p < .05
- p < .01
- p < .001
- p < .05
- 5/100 = five times out of 100, the results you see are because of chance and not the intervention
- p < .001
- .1/100 = one times out of 1000, the results you see are because of chance and not the intervention
- Findings with this alpha level are considered to be very significant
Probability / Alpha Level Example
- If the researcher say their statistical analyses revealed significant results with a p < .01, they are saying:
- There is a less than 1 chance in 100 the results are because of chance
- To have a probability statement with this alpha level, they are very confident about the statistical findings
- Consequently, there is a very good chance that the intervention has an effect
Example
Dr. Neversleep would like to conduct a study on exam performance and levels of sleep. He believes that students need at least 8-10 hours of sleep each night to perform well on exams. He is going to randomly sample 1000 university students and follow them throughout the December final exam period
-State the hypotheses for the research question:
Ho: Lack of sleep has no effect on the outcome of exame
Ha: Lack of sleep does affect the outcome of exams
-The results for Dr. Neversleep’s study are very interesting. It would appear that with a p <.01, he can conclude that lack of sleep can have an impact on exam outcomes
Why do Type I Errors Happen?
Most often, subjects with extreme scores affect the data analyses
Consequences of Type I errors
-The researcher claims there is an effect and there is not – can lead to change in clinical practice, which in turn could be harmful
-Researcher may try to replicate the findings but does not get the same results because there was no effects the first time around
-False results can lead to other researchers to develop theories based on false information – leads to a dead end for everyone
Why do Type II Errors Occur?
-Treatment effect may have been too small to have a noticeable effect on the sample
-It is possible by chance that the sample was extreme in some way before the study started
Consequences of Type II errors:
-The experimental design may not be good enough
-Experiment may be abandoned when in fact it has potential (provided modifications are made)
-Could mean giving up a good line of research that could lead to important findings
