Step 4: Analyze the Data Flashcards
What are the 2 types of measurement errors in quantitative data?
Random errors - variable, unpredictable chance errors
Systematic - error occurs consistently
Who is the person contributing the information that will be analyzed in a quantitative and a qualitative study?
Quantitative
- study participant
- respondent
Qualitative
- study participant
- informant
- key informant
Who is the person undertaking the study in a quantitative and a qualitative study?
Researcher/investigator for both
What is being studied in a quantitative and a qualitative study?
Quantitative
- Concepts
- Constructs
- Variables
Qualitative
- Phenomena
- Concepts
What information is gathered in a quantitative and qualitative study?
Quantitative
- data
- numerical values
Qualitative
- data
- narrative descriptions
What are the connections between concepts that are being analyzed in a quantitative and qualitative study?
Quantitative
- Relationships (cause and effect, associative)
Qualitative
- patterns of associations
What is the logical reasoning process used during analysis of a quantitative and qualitative study?
Quantitative
- deductive reasoning (top down)
Qualitative
- inductive reasoning (bottom up)
What is the quality of evidence being analyzed in a quantitative and qualitative study?
Quantitative
- reliability
- validity
- generalizability
Qualitative
- trustworthiness
What is involved in the analysis of a mixed methodology study?
What is the focus?
Integration of the strands
- the strands are the qualitative and quantitative components
Meta-inference
- The insights derived from integrating qualitative and quantitative inferences at the end of the study
- Interpretative level after separate analyses have been completed
Focus
- congruence
- complementarity
What are strands in the mixed methodological study?
Parts of the mixed method design, typically either the qualitative or quantitative component
What is inference quality as it refers to a mixed methodological study?
The believability and accuracy of inductively and deductively derived conclusions
What is inference transferability as it refers to a mixed methodological study?
The degree to which conclusions can be applied to other similar people or contexts
What does the integrative framework for inference quality of a mixed methodological study consist of?
Design quality
Interpretive rigour
What is skewed data?
A long tail on one side or the other of a curve of a data set
What is a negative skew?
- Long tail on the negative side of the peak
- Also called “skewed to the left”
- Mean is shifted to the left of the peak
What is a positive skew?
- Long tail on the positive side of the peak
- Also called “skewed to the right”
- Mean shifted to the right of the peak
What type of skew shows the peak to the left of the mean?
Right or positive skew
What type of skew shows the peak to the right of the mean?
Left or negative skew
What is the extreme thoroughness and accuracy in research that is achieved through strict methods, processes, or procedures?
How is this expressed in quantitative and qualitative studies?
Rigour
Quantitative research, rigour is expressed as
o Validity
o Reliability
Qualitative research, rigour is expressed as
o Trustworthiness
o It may just be called rigour
Regardless of the paradigm used in a research study, what are the goals?
What is a study considered if it meets these goals?
MEMORY: CANT do it
C - consistency
A - applicability
N - neutrality
T - truth value
Rigorous if it meets these goals
How is the goal of truth value expressed in quantitative and qualitative studies?
Quantitative
- internal validity
Qualitative
- credibility
How is the goal of applicability expressed in quantitative and qualitative studies?
Quantitative
- external validity
Qualitative
- fittingness/transferability
How is the goal of consistency expressed in quantitative and qualitative studies?
Quantitative
- reliability
Qualitative
- auditability/dependability
How is the goal of neutrality expressed in quantitative and qualitative studies?
Quantitative
- objectivity
Qualitative
- confirmability
What does the statistics used in a statistical analysis depend on?
o Type of data to be analyzed
o Level of measurement of variables
o Assumptions about population distribution
o Study sample size
Where is the statistical analysis located in a paper?
The methods and results section
What type of statistical analysis may be used in a paper?
Descriptive - describes data
Inferential - tells about the greater population
What is the index that describes a characteristic of an entire population such as an average when performing a statistical analysis?
Parameter
What is a parameter in a statistical analysis?
o Index that describes a characteristic of an entire population
o i.e. averages and percentages
What is an index calculated from sample data as an estimate of a population parameter when performing a statistical analysis?
Statistic
What is a statistic?
o Index calculated from sample data as an estimate of a population parameter
What are the 4 levels of measurement?
What do they help us determine?
nominal
ordinal
interval
ratio
Determines which summary statistics, graphs, and analysis are possible/sensible
What is the process of assigning numbers to objects, where the number represents the quantity of the attribute under study?
Measurement
Describe the nominal level of measurement
- Most basic level
- Also called categorical or qualitative
- Data is classified
- Variables are named
o Examples are sex, colour, preferred type of chocolate - Values can be stored as a word, text, or given a numerical code but the numbers to not imply order
- To summarize we use a frequency or percentage
- Cannot calculate mean or average value
- Visual representations – pie, column/bar, stacked column/bar charts
What level of measurement is data classified but does not use rank ordered and used strictly for non-numerical data such as sex?
nominal
Which type of data can be represented by pie charts, bar/column charts or stacked column/bar charts?
nominal
Describe the ordinal level of measurement
- Data is ordered
- Meaningful order but intervals between each value on the scale may not be equal
o Think like the always to never scale, satisfaction scales, or ranking things from first to tenth (even runners in a race) - Summarized as frequencies
- Mean may be calculated, just ensure the calculation is justified
- Visual representations – column/bar chart
Which level of measurement does not allow for calculating the mean or average?
nominal
Which level of measurement may sometimes have a mean calculated, but other times, it does not make sense to calculate a mean?
Ordinal
Which level of measurement is summarized as a frequency?
nominal
ordinal
Describe the interval/ratio level of measurement
- Data is measurable
- Meaningful order but intervals between each value on the scale have consistent and quantifiable differences between them
o Interval has no true zero, the zero is arbitrary and does not mean a complete absence of the variable
i.e. temperature in Celsius – 0C doesn’t mean an absence of temperature
i.e. test score – 0% doesn’t mean you didn’t take the test
o Ratio has a true zero
Cannot have negative numbers
i.e. weight, height, number of kids - Also called scale, quantitative, or parametric
- Examples include age, weight, number of customers etc
- Discrete – uses whole numbers
- Continuous – uses fractional numbers
- Mean, median and standard deviation can be calculated
- Visual representations – bar chart, histogram, box plots, line charts
In which level of measurement is there meaningful order, but the intervals between each value on the scale have consistent and quantifiable differences between them?
Interval/ratio
What is the difference between the interval and ratio levels of measurement?
Interval has no true zero, the zero is arbitrary and does not mean a complete absence of the variable
i.e. temperature in Celsius – 0C doesn’t mean an absence of temperature
i.e. test score – 0% doesn’t mean you didn’t take the test
Ratio has a true zero
Cannot have negative numbers
i.e. weight, height, number of kids
What is the use of whole numbers in statistical analysis called? What level of measurement uses whole numbers?
Discrete
Interval/ratio
What is the use of fractional numbers in statistical analysis called? What level of measurement uses whole numbers?
Continuous
Interval/ratio
Which level of measurement allows for calculating the mean or average?
Interval/ratio
What are non-parametric statistics?
- No rigorous assumptions avoid distributions of the variables
- Used with small samples
- Nominal or ordinal data
- When distribution is severely skewed
What are parametric statistics?
- Assumes normal distribution of the variables
- Requires interval/ratio measures
- Participants should be randomly assigned
- Homogeneity of variance
What are the non-parametric tests of differences?
chi square
What are the parametric tests of differences?
- t-tests (independent or dependent)
- ANOVA
- MANOVA
What are the non-parametric tests of association?
- Phi coefficient
- Spearman rho
- Kendall tau
What are the parametric tests of association?
Pearson coefficient - r
Multiple regression - R
Describe the Chi Square test
Tests of Differences – Non-Parametric
* Used with nominal/ordinal data
* Compares observed frequencies with expected frequencies
* Expected frequencies are the number of cases in each case if null is true
* Fischer’s exact probability test
o Used for small sample sizes < 6 in each cell
* i.e. test for differences in study groups on marital status, racial makeup, and education level
Describe the independent group t-tests
Tests of Differences – Parametric
o Subjects in 2 groups are not the same people and not connected in any systematic way
i.e. caffeinated coffee and intraocular pressure (N=100)
Group 1 – regular coffee n=50
Group 2 – decaffeinated coffee n=50
IV = group – nominal level
DV = intraocular pressure – ratio level
Describe the dependent group t-tests
Tests of Differences – Parametric
- Paired, correlated groups t-test
- Some group of subjects is measured on more than one occasion
- Sample fluctuation is lower
- i.e. diabetics on a weight loss program
Describe ANOVA tests
Tests of Differences – Parametric
- Analysis of variance
- When the means of 3 of more groups are compared
o Assumes independence of groups - Used to analyze all groups simultaneously by dividing the (alpha) between all tests
- Dependent variables should be continuous and normally distributed
- Statistic calculated is the F ratio
o MSB – means between the groups
o MWW – Variation of individual scores within each of the groups - F = MSB / MWW
Describe MANOVA tests
Tests of Differences – Parametric
- Multivariate analysis of variance
- Examines the difference between mean scores of two or more groups on two or more dependent variables that are examined at the same time
- Repeated measures ANOVA – group means are compared at multiple points
Describe the Spearmen’s Rho and Kendall Tau tests
Tests of Association/Relationship – Non-Parametric
- Used for ordinal data
- Varies from -1.0 to +1.0
Describe the Pearson Coefficient - r test
Tests of Association/Relationship – Parametric
- Indicates the magnitude and direction of a linear relationship between two variables
- Varies from -1.0 to +1.0
Describe the multiple regression - R test
Tests of Association/Relationship – Parametric
- Relationship between interval dependent variable and several independent variables
- What independent variable contributes to explain dependent variable
- Varies from -1.0 to +1.0
What tests use nominal or ordinal data? What are examples of these tests?
Non-parametric tests
Tests of differences - chi square
Tests of association - phi coefficient, Spearman rho, Kendall Tau
What type of tests uses nominal data and compares the observed frequencies with the expected frequencies?
Chi square test
You are running a trial for a new blood pressure medication. One group receives the new medication while the other group uses standard therapy. What type of statistical analysis should be run on the resulting data?
Independent group t-tests
o Subjects in 2 groups are not the same people and not connected in any systematic way
You are a nurse working in a hospital and you notice that many of your patients seem to have COPD. You want to know if your city actually has higher than expected rates of COPD for your country or if it’s just your anecdotal evidence that there are more patients with COPD. What test would help you determine this?
Chi Squared test
What tests use interval/ratio data and has 2 groups that are not the same people and they are not connected in any systematic way?
Independent group t-tests
You are running a study to determine if patient education will improve the lives of your obese patients. At the beginning of the study, the 50 participants came to see you to have their weight recorded. For the next 8 weeks, all 50 people participate in a 1 hour meeting each week. You cover a different topic each week regarding obesity (risks, diet, exercise etc.). At the end of the study, the participants come back to be measured a second time.
What type of statistical analysis would tell you if your patients lost a statistically significant amount of weight over the 8 week program?
Dependent group t-tests
(same group measured on more than one occasion)
You are testing out a few different interventions commonly used in your hospital to treat pain. You divide the patients into 4 groups.
Group 1 - meds only
Group 2 - meds and massage
Group 3 - meds and regular ambulation/movement
Group 4 - meds and meditation
You have each participant rate their pain at the end of the study. What statistical analysis would tell you which was the most effective?
ANOVA test
(checks differences between at least 3 groups)
You are attempting to determine the effectiveness of several non-pharmaceutical treatment modalities for your patients. You randomly assign your patients into the following groups
Group 1 - mindfulness/meditation
Group 2 - foot massages
Group 3 - CBT therapy
Group 4 - traditional treatment plans only
You are going to measure the effects of these modalities on their stress levels, blood pressure, heart rate and pain levels.
What type of statistical analysis would tell you which was the most effective?
MANOVA
100 nursing students have all ranked their preferred placements for Year 4. The manager of the program wants to find out if there is any correlation between the students about their top 5 picks. What test should he use?
Spearman’s Rho, Kendall Tau
Tests of Association/Relationship – Non-Parametric
* Used for ordinal data
* Varies from -1.0 to +1.0
A nurse knows that COPD is related to smoking. The nurse would like to know if there is also a relationship between the number of years someone smoked at the time of their diagnosis and their age at diagnosis. What test should the nurse use?
Pearson coefficient r
- Indicates the magnitude and direction of a linear relationship between two variables
- Varies from -1.0 to +1.0
You are a public health researcher interested in social factors that influence heart disease. You survey 500 towns and gather data on the percentage of people in each town who smoke, the percentage of people in each town who bike to work, and the percentage of people in each town who have heart disease.
What test can be used to analyze the relationship between these variables?
Multiple regression R
What is a type 1 error?
- Alpha error
- Rejects the null hypothesis incorrectly
- Finds a difference when there is no differences
What is a type 2 error?
- Beta error
- There is a difference, but the study does not find it
- Occurs when the sample is too small
What type of error often occurs because the sample size was too small?
Type 2 or beta error
What type of error occurs when the null hypothesis is incorrectly rejected?
Type 1/alpha
What type of error occurs when there is a difference and the null hypothesis is not rejected?
Type 2/beta
What is the power in a statistical analysis?
- 1 minus beta
- Probability of correctly rejecting the null when it is false
- The risk of not making a type II error – the probability of avoiding the type II error
- Used to estimate the sample size, effect size, and desired power level
What are the steps of the data analysis process?
Step 1: Preparing Data for Analysis
Step 2: Describing the Sample
Step 3: Testing the Reliability of Measurement Methods
Step 4: Conducting – Exploratory Analysis of Data
Step 5: Conducting – Confirmatory Analysis Guided by the Hypothesis, Questions, or Objective
Step 6: Conducting – Post Hoc/Supplementary Analysis
Describe what occurs in step 1 of the data analysis process: preparing data for analysis
- data handling and entry
- coding
- data handling - statistical packages for computer systems
Describe what occurs in step 2 of the data analysis process: Describing the Sample
- Estimates of central tendency and dispersion
- Relevant variables
o i.e. age, education level, health status, gender, ethnicity, etc. - Determine equivalence of groups
- Identify statistical procedures used to identify sample
Describe what occurs in step 3 of the data analysis process: Testing the Reliability of Measurement Methods
Cronbach’s alpha
* Test performed on any paper/pencil scales that were used
* Used to determine the Cronbach coefficient
* Shows homogeneity of instrument
* Value 0.70 – marginally acceptable
* Value 0.80-0.89 – measurement is sufficiently reliable to use in the study and for analysis of data
* Value < 0.70 – unacceptably low
Researcher must decide whether or not to analyze the data collected with the instrument
Describe what occurs in step 4 of the data analysis process: Conducting – Exploratory Analysis of Data
Examine data on each variable to determine nature of variation in the data and to identify outliers
Frequency/distribution
o Concerned with how many times a value appears in the data
o Frequency distributions are often represented by tables, bar or pie charts
Describe what occurs in step 5 of the data analysis process: Conducting – Confirmatory Analysis Guided by the Hypothesis, Questions, or Objective
To confirm expectations
Inferential statistics
* Based on laws of probability
* Provide a means for drawing conclusions about a population on the basis of data from a sample
* Uses theory
o Sample distribution of the mean
o Standard error of the mean
o Hypothesis testing
o Level of significance
Includes both parametric
Describe what occurs in step 6 of the data analysis process: Conducting – Post Hoc/Supplementary Analysis
- Statistical procedures can indicate differences between groups but do not specify which groups are different
- When significant differences are found, post hoc analyses are performed
Describe descriptive statistics
Describes participants by their demographics and pattern of response
Descriptive designs are those that use descriptive statistics
Measures of location and spread, used to find patterns
Used to synthesize, describe, or summarize data
* Measures of frequency of the occurrence of a concept
* Measures of central tendency of a concept
o Mean, mode, median
* Measures of the variability of a concept
Also called measure of spread or measure of dispersion
What is the range?
Descriptive statistic
the difference between the lowest and highest values
What is the semi-quartile range?
Descriptive statistic
Range of middle 50% of scores
Describe the mean
Descriptive statistic
- Sum of all scores and divided by the number of cases
- Average of all scores, most stable measure
- Larger sample size decreases the effect of extreme values
- Requires interval, ratio data
Describe the mode
Descriptive statistic
- The value that occurs most frequently
Describe the median
Descriptive statistic
- The exact middle value in a score distribution
Describe normal distribution
Descriptive statistic
- Mean, median, and mode coincide
- Normal curve is unimodal and symmetrical about the mean
- Bell shaped form
- Fixed % of scores fall within given distance from the mean
- Skewed distribution
o If skewed (non-normal) distribution, mean, mode, and median should all be reported
Describe standard deviation
Descriptive statistic
- Relation a set of scores has to the mean
o Always reported with the mean
o Average deviation of scores in a sample from the mean - Based on concept of a normal curve
- Expressed in terms of units and cannot be assumed to compare means that have different units
What determines the likelihood that results of the study occurred by chance or an actual reflection of the population?
Inferential statistics
What does inferential statistics use sample data to do?
o Draw conclusions about population characteristics
o Test hypotheses about relationships between variables in the population
What are the required conditions for using inferential statistics?
o Probability sample – representativeness of population
o Interval or ratio level of measurement of variable
What are the measurements that may be used with inferential statistics?
o Measures of the relations between concepts
o Measures of the effect of one concept on one or more other concepts
What is reliability and what is it affected by?
Consistency with which a measuring instrument yields a certain, consistent result when the entity being measured hasn’t changed
o The extent that the instrument yields the same result on repeated measures
o Analogous to variance
Low reliability = high variance
Affected by random error
o Observed score = true score ± error
o Instrument is reliable if it accurately reflects the true score and minimizes error
Describe random errors
- Variable, unpredictable chance errors
- Affects reliability
- An error that causes individuals’ observed scores to very haphazardly around their true score
- Sources
o Environmental factors/situational contaminants
o Researcher factors
o Subject factors
o Instrumentation factors
Describe reliability coefficient
- Proportion of true variability to the total observed (obtained) variability
- Denoted as r
- A coefficient of 0.85 means
o 85% of variability in observed scores is presumed to represent true individual differences
o 15% of variability is due to random error - Acceptable reliability is +0.80
What are threats to reliability in quantitative studies?
- Unstable/unrepeatable measurements
- Bias
- Small sample size
What tests can be done for reliability in a quantitative design?
Tests of Stability
Tests Of Internal Consistency/Homogeneity
Tests Of Equivalence
What is stability and how can we test for it?
Stability
o Instrument generates similar findings from the same group on different occasions
Test-retest
o Compare results of testing at two points in time with the same individuals
Parallel form
o Agreement of measuring instruments over time
i.e. collaboration, stable traits
What test ensures that the instrument generates similar findings from the same group on different occasions?
How can we test this?
Stability
We use tests of stability to test it including
- test-retest
- parallel form
What type of test compares the results of testing at two points in time with the same individuals?
Test-retest
It is a test of stability
What type of test shows the agreement of measuring instruments over time?
Parallel form
It is a test of stability
What is internal consistency and how can we test for it?
Internal consistency
o Degree to which items on a questionnaire measure a particular variable
o Strong internal consistency are homogeneous variables
o Extent to which tests assess the same characteristic or quality
o i.e. a questionnaire contains questions on anxiety, internal consistency tells us which ones focus on anxiety
Cronbach’s alpha
o For continuous data (range = 0-1)
o Measures proportion of variance that is shared among items
Kuder-Richardson 20 (KR20)
o Used for dichotomous data
What is the degree to which items on a questionnaire measure a particular variable and the extent to which tests assess the same characteristic or quality?
How can we test this?
Internal consistency
We use tests of internal consistency:
- Cronbach’s alpha
- Kuder-Richardson 20
What measures the proportion of variance that is shared among items and used for continuous data to test for internal consistency?
Cronbach’s alpha
Its a test of internal consistency
What test is used to test the internal consistency consistency of dichotomous data?
Kuder-Richardson 20 (KR20)
Its a test of internal consistency
What is another term for a test of internal consistency?
Test of Internal Homogeneity
What is equivalence and how can we test for it?
Equivalence
o Interrater reliability in observational studies
o Degree of similarity between two or more alternate forms of a measurement instrument
Intra-rater reliability
o Assesses how one person rates the same observation on 2 or more occasions
o Consistency
Inter-rater reliability
o Degree to which 2 or more independent observers agree
Measured by Cohen’s Kappa, Pearson’s r, or Spearman’s rank correlation coefficient
What describes the degree of similarity between two or more alternate forms of a measurement instrument or between 2 observers?
Can can we test this?
Equivalence
Tested with tests of equivalence
- intra-rater reliability
-interrater reliability
What assess how one person rates the same observation on 2 or more occasions?
Intra-rater reliability
It’s a test of equivalence
What measures the degree to which 2 or more independent observers agree on the measurement?
Inter-rater reliability
It’s a test of equivalence
What term in quantitative studies is used to determine the extent to which the findings are believable or true?
Validity
What term is used to ensure that the instrument measures what it is supposed to measure?
Validity
Describe a systematic error. What are some sources of this?
Constant
Affects validity
Error that is not random but occurs consistently
* i.e. a scale that inaccurately weight subjects 3 pounds heavy
Sources
* Researcher factors
o Observer bias
o i.e. may consistently rate subjects higher/lower
* Subject factors
o Response is biased
o i.e. participant always answers negatively or positively
* Instrumentation factors
o Inadequate sampling of items in domain of interest
i.e. forgetting to clear cache before timing a run or a weigh scale that always measures 2 kg lower
What is a consistent error that affects validity?
Systematic error
A researcher consistently rates a subject at a higher level because they like the person in the study. What is this error?
What type of bias is this?
Systematic error caused by a researcher factor
Observer bias
A participant is having a rough day and so they are consistently answers negatively throughout the test even though generally they do not feel this way. What is this error?
What type of bias is this?
Systematic error caused by subject factor
Response is biased
Someone did not calibrate the BGL machine this week and it is consistently measuring the BGL of the diabetic patients in your study 2.0 mmol/L higher than their actual number. What is this error?
Systematic error caused by instrumentation factor
What are the approaches to constructing validity in a quantitative study?
Factor analysis
o Statistical procedure to identify the underlying dimensions of an instrument
o One statistical technique that can be used to determine the constructors or domains with a developing measure and therefore contributes to establishing construct validity
Known-groups approach
o Also called contrasted groups approach
o Administer the test to 2 groups known to differ
o If the instrument is sensitive to individual differences in the trait being measured then results should show differences
Hypothesis testing
o Testing of the hypothesized relationships
o Test can discriminate between a group of individuals known to have a particular trait and a group who do not have the trait
What is a statistical procedure used to identify the underlying dimensions of an instrument?
Factor analysis; an approach to constructing validity
If administer a test to 2 groups known to differ in order to ensure that the instrument is sensitive to individual differences in the trait being measured, the the results should show the difference. What is this an example of?
Known-groups approach; an approach to construct validity
We establish a test that can discriminate between a group of individuals known to have a particular trait and a group that do not have the trait in order to test the hypothesized relationship. What is this an example of?
Hypothesis testing; an approach to construct validity
What is internal validity?
- Features of the research context that can compete with the independent variable (IV) to explain what you are observing about the dependent variable (DV) or outcome
- Relies heavily on the use of data collection tools or study instruments that promote accurate, precise and true collection of study data
- Threats are unreliable findings about the dependent variable
What are examples of internal validity?
Face validity - subjective evaluation of a measurement instrument
Content validity - data collection instrument is inclusive and applicable to the research question
Criterion related validity - indicates how well, or poorly, an instrument compares to either another instrument or another predictor (the criterion)
construct validity - most complex and difficult to test; how well did the test measure the construct it was supposed to measure
Describe face validity
- Concerned with reading the study instrument and judging whether the questions “appear” to measure what they are supposed to
- Subjective evaluation of a measurement instrument based on the way it appears
- Does the instrument look like it is measuring what it is intended to measure
Describe content validity
- Data collection instrument inclusive and applicable to the research question
- Does the content of the instrument adequately capture the construct?
- Questionnaires are subjective
- To measure:
o The instrument is distributed to a panel of experts under inquiry
o Compare the content of the measure with the theoretical definition of the construct
o Panel rates from 1 – not relevant to 4 – very relevant
o Score < 3 should be removed from the questionnaire
Describe criterion-related validity
- Indicates how well, or poorly, an instrument compares to either another instrument or another predictor (the criterion)
- Concurrent validity
o Data from 2 methods can be compared in tandem and have similar results
o Comparison of the measure in question and an outcome assessed at the same time - Predictive validity
o Instrument can accurately identify how an individual will respond or behave in a particular situation
Describe construct validity
- The most complex type of validity and can be a difficult concept to test
- Ensures congruity between the variables of the conceptual frameworks and how the researcher plans to measure these
- Attempts to validate a body of theory underlying the measurement and testing of the hypothesized relationships
- Indicates how well the scale measures the construct it was designed to measure
- i.e. a researcher inventing a new IQ test might spend a great deal of time attempting to “define” intelligence in order to reach an acceptable level of construct validity
A researcher is attempting to measure the amount of fluid removed from the system by looking at the measurement lines on the side of the beaker. What type of validity is this?
Face validity
A researcher is evaluating a questionnaire to ensure that it measures closely to the theoretical definition of the construct that is being used. What type of validity is this?
Content validity
What type of validity indicates how well, or poorly, an instrument compares to either another instrument or another predictor?
What are the two subtypes of this validity?
Criterion-Related Validity
concurrent validity - measure and outcome assessed at the same time with 2 different methods
predictive validity - instrument accurately identifies how well a person responds to a particular situation
What type of validity requires data from 2 methods compared in tandem and have similar results and is a comparison of the measure in question and an outcome assessed at the same time?
Concurrent validity; a subtype of criterion-related validity
What type of validity is exemplified by an instrument accurately identifying how an individual will respond or behave in a particular situation?
Predictive validity; a subtype of criterion-related validity
What type of validity is the most complex type of validity and can be difficult to test because it attempts to validate a body of theory underlying the measurement and testing of the hypothesized relationsip?
Construct validity
A researcher inventing a new IQ test might spend a great deal of time attempting to “define” intelligence in order to reach an acceptable level of ________ validity
construct
What is the history threat to internal validity?
Threats to Internal Validity
* External events – events that are unrelated and happen outside the study
* They occur at the same time as the IV that can affect the outcome
A study on anxiety was conducted in the US during the last presidential election. What kind of validity threat is this?
History threat, a form of threat to internal validity
Describe the maturation threat
Threats to Internal Validity
* Developmental change
* Process that occurs within the participants of a study as a result of time
* Most notable in children as their responses will improve and change as they develop
* May happen in new nurses but when you go back they will be improved nurses just because they have been working
Describe the testing threat
- Testing effects are also called order effects
- Occur in designs that have more than one stage i.e. pre-test and post-test
- Taking a test more than once influences the behaviours and scores on the post-test thus confounding the results
Describe the instrumentation threat
Threats to Internal Validity
* Reliability of measure
* Changes in data collection methods
o Different collectors with slight variations in how they measure
o Poorly trained collectors
o Changing to a BP cuff that hasn’t been calibrated
* Confounding by changes in what an instrument measures over time
Describe the mortality threat
Threats to Internal Validity
* Sometimes also called attrition
* Participants drop out or drop dead
* Usually recruit more than they need to account for attrition
Describe the selection bias threat
Threats to Internal Validity
* Poor selection of subjects
* The way people are chosen to participate in a study
* Arises from differences between groups that exist before the study takes place
* Avoid this with random selection and random assignment into groups
A nursing unit is assessing the efficiency of a new process. A nursing student’s efficiency is measured on the first day of the program role out, only her second day on the floor. She is measured again a month later.
What type of threat to internal validity may confound the results of the study for this participant?
Maturation threat
What type of internal validity threat is exemplified by taking an IQ test at the beginning of a 4 week program and then taking the same test again at the end of the program?
Testing threat
What type of internal validity threat occurs due to poorly trained collectors that do not measure the results in the same way?
Instrumentation threat
It is also a form of inter-rater reliability, which needs a test of equivalence
What type of internal validity threat is accounted for by recruiting more in the sample than they need in order to account for attrition?
Mortality threat
What type of internal threat arises from differences between groups that exist before the study takes place?
Selection bias threat
What is external validity concerned with?
The generalizability of the findings
What is an effect, similar to selection bias, that results in a threat to external validity caused by participant characteristics that affect the observed response to the independent variable in a way that the findings cannot be generalized to people with different characteristics?
Selection effects
What are selection effects?
- Similar to selection bias
- Participant characteristics may affect the observed response to the IV in a way that the findings cannot be generalized to people with different characteristics
- Increased threat – when ideal sample cannot be obtained
What type of external validity threat occurs when the ideal sample cannot be obtained?
Selection effect
What are reactive effects?
Subjects response to being studied
Subjects may be reacting to some feature of the research environment rather than the IV
We cannot assume that the findings will be the same when the IV is administered under different circumstances
Possible subject responses
* Experimenter effect
o Experimenters’ behaviour influences the result of an experience
o May be conscious or unconsciously influence
* Novelty effect
o Tendency for people to have a strong initial response to something new, which then fades over time
* Hawthorn effect
o Attention alone produces the results
Like watching an intervention on workers may cause the change just because the workers know they are being watched
What effect on external validity is caused by a subjects response to being studied rather than the independent variable?
Reactive effects
What type of effect occurs when the experimenter is very passionate about what he is studying and inadvertently influences the result of an experience?
Experimenter effect; a type of reactive effect which are for external validity
What type of effect is noted by a nurse participant having a strong initial response to a new piece of equipment in the hospital setting but this fades over time?
Novelty effect; a type of reactive effect which are for external validity
A nursing manager is ready with a clipboard to monitor the hand hygiene compliance on the unit. What effect may interfere with the results of this study?
Hawthorn effect
What is the experimenter effect?
Reactive effect on external validity
Experimenters’ behaviour influences the result of an experience
May be conscious or unconsciously influence
What is the novelty effect?
Reactive effect on external validity
Tendency for people to have a strong initial response to something new, which then fades over time
What is the Hawthorn effect?
Reactive effect on external validity
Attention alone produces the results
Like watching an intervention on workers may cause the change just because the workers know they are being watched
What is the measurement effect?
An effect on external validity
- Pretest can increase or decrease subject sensitivity to DV
- Affects generalizability if pretest not used in other settings, populations, etc.
- i.e. based on pre-test, participants modify their thinking and responses
- Increased threat – pretest is used
What are the basic steps to a qualitative data analysis?
- Thematic analysis
- Analyze text into codes
- Themes and concepts
What is the process of qualitative data analysis?
Iterative cycles of
o Comprehending
o Synthesizing
o Theorizing
o Recontextualizing
