Research and Program Evaluation Flashcards
Research
the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question
Evidence-based inquiry
the search for knowledge using empirical data which has been gathered systematically
Quantitative research
- assumes social facts have a single objective reality
- tends to study samples or populations
- researchers try not to influence collection of data (instruments)
- statistical methods comparing and contrasting groups occurs
- researchers examine for causes and relationships
Qualitative research
- assumes multiple realities socially constructed by individuals/groups
- tends to study individual units - person, family, community - in naturalistic setting
- researchers may be primary instrument for collecting data (through observation)
- researchers’ impressions, judgments, feelings may be used
- goal is to describe the nature of things
When to choose qual vs. quant
both kinds of research are valued
- one is chosen over the other because it better fits the assumptions of the researcher and the nature of the problem under investigation
- some professional journals prefer to publish one kind of research over the other
Inductive research
begins at the real world, practical level (small and builds to large theory)
- tends to be descriptive, correlational, historical
- leads to building of theory
- closer to qualitative research
Deductive research
springs from theory which is already established (starts broad and goes smaller to specifics)
- tries to determine what the relationships are between elements of the theory and may be experimental in nature
- closer to quantitative research
Quantitative; Non-experimental designs
Survey
may occur through questionnaires, interviews, etc. and is used to measure attitudes, perceptions, etc.
- ex. Public Opinion Poll
- often has low response rate, below 50%
- unless you know that characteristics of non-respondents are similar to respondents, must be cautious in generalizing
Quantitative; Non-experimental designs
Descriptive
describes an existing state of events
- numbers may be used to characterize groups/individuals
Quantitative; Non-experimental designs
Comparative
investigates whether there are differences between two+ groups
- no manipulation of conditions experienced by each group
Quantitative; Non-experimental designs
Correlational
this research method uses the correlation coefficient to determine the degree of relationship between two+ variables or phenomena
ex. income level and attitude toward counseling
bivariate: correlational data describing the nature of two variables
multivariate: more than two variables
Quantitative; Non-experimental designs
Ex post facto (Causal-comparative)
studies possible causal relationships among relationships ex post facto (after the fact) - no random assignment
- do not manipulate any variables; focus is on what has already happened
- may generate several reasons (causes) for the relationships you discover
- uses t-tests and ANOVAs
Quantitative; Experimental designs
True experiment
characterized by the use of experimental and control groups with random assignment to each
- used to determine cause-and-effect relationships
- ex. 60 college freshmen are enrolled in English class. 30 are randomly assigned to one-hour per week writing lab, and the others comprise a control group. End-of-semester essay exam results are analyzed to see if the lab was associated with better writing skills
Quantitative; Experimental designs
For experiments, there are design variations such as:
- treatment and control group with posttest only
- treatment and control group with pretest and posttest
- two different treatment groups with control groups and posttest
- etc.
Quantitative; Experimental designs
Quasi-experiment
similar to experimental research except that randomization of subjects to treatment and control groups is not possible
- may be that no control or comparison group is available
- result from such research will not be as unequivocal as results from a true experimental study
- ex. a school has two classrooms of 4th graders. Each classroom is taught arithmetic by a different method for the school year. In May, arithmetic achievement is compared for the two classrooms using scores on a national exam
Types of Research - Qualitative
Qualitative
emphasizes gathering data about naturally occurring phenomena (individual’s and groups’ living experiences) and events
- data collection may be in terms of words rather than numbers
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Case Study
the case may be a program, activity, set of individuals who are bounded in time and place
Types of Research - Qualitative
Two principal qualitative research designs:
Interactive - Ethnography
a description and interpretation of a cultural or social group/system. Data is typically collected through observation and interviewing
- Issue of observer bias is important
Types of Research - Qualitative
Two principal qualitative research designs:
Noninteractive - analytical research
conducted primarily through document analysis
- ex. historical analysis (collecting and analyzing docs describing former events)
- ex. biographical analysis (written or oral)
- ex. legal analysis (focuses on law and court decisions)
Mixed-Method Research Designs
Mixed-Methods
combine quantitative and qualitative methods in the same research effort
- researcher retains the flexibility to use both types of design
- typically designs are used sequentially (quant. may be gathered first and then qual. used to further explain or elaborate on findings using surveys, interviews, focus groups)
Other Specialized Research Designs and Types
Single-subject design
studies the effects of a program or treatment on an individual or group treated as an individual, usually after a baseline has been established
Other Specialized Research Designs and Types
Action research
conducted in an attempt to improve services or a program
- may be viewed as having an evaluative function
Other Specialized Research Designs and Types
Pilot study
small-scale research effort often used to determine the feasibility of a large scale effort with emphasis on refining procedures and instrumentation
Other Specialized Research Designs and Types
Longitudinal research
collecting data from the same group of individuals over a period of time (panel study)
- ex. studying career development of school children by reinterviewing them every two years until they were high school seniors
Other Specialized Research Designs and Types
Cross-sectional research
consists of collecting data from different groups at the same time and examining these differences
- ex. studying career development by interviewing each grade of students at the same time
also called Synchronic method
Research outcomes may be measured two ways:
Within-subjects
examining what changes occur within the members of a group
IPSATIVE
Research outcomes may be measured two ways:
Between subjects
examining what changes occur between two or more groups
Meta-analysis
research comparing findings across studies
- i.e., the results of many studies are examined simultaneously and one or more research questions answered
Internal Validity
Internal Validity
experiments are internally valid to the extent that extraneous variables have been controlled
- to the extent that the treatment variable is the only one producing the observed changes, the experiment is internally valid
Threats to Internal Validity
Selection of subjects
differences in the results between two groups may not be due to the treatment variable experiement by one group because the composition of the two groups are different to begin with (probably not randomly selected)
Threats to Internal Validity
Instrumentation
differences in results between two or more groups may be due to instruments which are unreliable or because the instruments are changed during the study
- or, perhaps the observers recording data become fatigued or bored and record behaviors differentially over time
Threats to Internal Validity
Maturation
results may be due to maturational or other changes in the subjects and not due to the treatment being applied. This is especially important if research data is gathered over a long period of time
Threats to Internal Validity
Mortality or attrition
losing subjects during the study could lead to different results than if everyone had stayed. Subjects with the most or least amount of important characteristics to the study may be the ones dropping out
Think of normal curve extremes
Threats to Internal Validity
Experimenter bias
the responses of the subjects may be influenced by the researcher. This may occur by treating some subjects differently, reinforcing different behaviors, as well as the presence of many other variables which deliberately or unintentionally influence subjects
Threats to Internal Validity
Statistical regression
sometimes subjects in a study are recruited because of extreme high or low scores (e.g., self-esteem, social skills) on the dependent variable being measured. Due to statistical regression, future measures would expect these individuals to score closer to the mean score even without any intervention
External Validity
External Validity
an experiment is externally valid to the extent that the results may be generalized to people and situations beyond the study
- there are several threats to external validity of experiments and some of these are also threats to internal validity
Threats to external validity
Selection of subjects
if subjects are not randomly selected, the results may only apply to the subjects in the study
- the results can only be generalized to people with similar characteristics
Threats to external validity
Ecological validity
the research has ecological validity if the results can be generalized from one setting or circumstance to another. Sometimes the circumstances, conditions, physical surroundings of the research are so unique that the results cannot be generalized beyond that study
Threats to external validity
Subject reactions (Reactivity): Hawthorne effect
1
the influence in performance which occurs when subjects receive attention or know they are participating in research
Threats to external validity
Subject reactions (Reactivity):
Demand characteristics
2
all the cues, info, knowledge, even rumors the subject has heard about the experiment which are likely to influence their performance
Threats to external validity
Subject reactions (Reactivity):
Experimenter bias (Rosenthal effect)
3
the changes in the subject’s behavior brought about by the researcher’s expectations, behaviors, attitudes
- Rosenthal conducted research into this phenomenon and called it Pygmalion effect, referring to the self-fulfilling expectation of doing well because it is expected
Threats to external validity
Subject reactions (Reactivity):
Placebo
4
any control treatment
should be identical to the experimental treatment except for the critical item being studied
even so, control subjects may be influenced by the placebo adn react in unintended ways
Threats to external validity
Novelty and disruption effects
the measured effect of the treatment on the subjects may be due to its novelty or the disruption it causes
- being selected for research may be exciting and energizing; as it continues it may begin to disrupt routine and one’s typical schedule
- when novelty and disruption wear off or stabilize, there may be no long-term effects of the treatment
Levels of measurement
Levels of measurement
determine the statistic you can use
Levels of measurement
Nominal
variable’s qualities or categories. use non-parametric test (ex. Chi-square)
ex. male/female
Categories YES
Rank order NO
Equal Spacing NO
True zero NO
Levels of measurement
Ordinal
differences in some magnitude of the variable
ex. scores on exam can be ranked from highest to lowest
Categories YES
Rank order YES
Equal Spacing NO
True zero NO
Levels of measurement
Interval
intervals between the numbers on a scale contain the same amount of the variable throughout the scale. provide a constant unit of measurement
NO REAL ZERO
Ex. Fahrenheit temperature (the distance/interval between 11 and 12 degrees is the same as the distance between 100 and 101 degrees)
Categories YES
Rank order YES
Equal Spacing YES
True zero NO
Levels of measurement
Ratio
numbers are on a scale which has a true zero
Numbers can be compared by ratios
Ex. weight, distance, time
Ex. someone who weights 200 lbs is twice as heavy as someone who weighs 100 lbs. BUT we cannot say someone is twice as introverted as someone else (so this ex is not ratio)
Categories YES
Rank order YES
Equal Spacing YES
True zero YES
Sampling
Sampling
how well samping is conducted will determine how validly we can generalize from a sample to a population
Involves the selection of a part of the population
Probability Sampling
Random sampling
all the individuals in the population have an equal and independent chance of being selected
Probability Sampling
Stratified sampling
sampling in such a way that major subgroups in the population will be sampled (ex. gender, age, ethnicity, etc.)
- can be proportional or disproportionate
Probability Sampling
Proportional stratified sampling
randomly selecting the same proportion of individuals for the sample as they represent proportionally in the major subgroups in the population
Ex. if 1/2 of a population is Hispanic and 1/2 is white, you would randomly select your sample to be 1/2 Hispanic and 1/2 white
Probability Sampling
Systematic sampling
researchers select members of the population at a fixed interval determined in advance
ex. selecting every 3rd study on a roster
Probability Sampling
Cluster sampling
the unit is not an individual but a naturally occurring group of individuals (ex. classrooms, city blocks)
Clusters are randomly selected for the study
Nonprobability Sampling
Non-random or nonprobability samples
samples of convenience or volunteer samples
- cannot be counted on to yield a normal distribution of scores but can yield useful/important data
Nonprobability Sampling
Purposeful sampling
in some studies, there may be no interest in generalizing findings so this may be used.
- selecting participants based on specific criteria such as characteristics or lived experiences
- can be helpful to gather in-depth info on a topic
Possibilities:
- comprehensive sampling where every case/event is selected
- there is extreme-case or typical-case selection
Ex. researchers studying individual’s lived experience with depression post flyers in counseling center
Nonprobability Sampling
Convenience sampling
participants are recruited who are easily accessible and who are usually close in proximity
ex. you select your classmates for your research on graduate counseling experiences
Nonprobability Sampling
Snowball sampling
participants are asked to assist researchers in identifying other potential subjects
Can be helpful when conducting research about people with specific traits
ex. you recruit 5 counselors in supervision and have each give you names of other counselors in supervision who you then recruit
Study tip: Michael Scott stuck in a pyramid scheme coded
Nonprobability Sampling
Quota sampling
the researcher identifies participants meeting different criteria that are needed for the study and then sets a target number for each category in the sample. Then, participants are nonrandomly recruited to fill the quotas for each criteria
ex. you are studying EMDR for trauma recovery, so you recruit 4 veterans, 4 first responders, etc.
Difference between Quota sampling and Stratified sampling
- both methods specify subcategories and attempt to fill targeted numbers for each subcategory
Quota sampling: participants selected nonrandomly so have more control over who you select (e.g., convenience or purposive sampling)
Stratified random sampling: participants selected using random selection technique (once in a subcategory). Since this is a probability method, you can generalize
Sample Size
influences statistical hypothesis testing
- tables for determining appropriate sample sizes are available
- 5-10% of the population is generally used
Suggested minimum sample sizes for different research:
- correlational (30)
- ex post facto and experimental (15)
- survey (100)
Statistical analysis may be
Descriptive
- sometimes called summary
- used to describe the data collected
- ex. means, SDs, frequency counts, percentages
Statistical analysis may be
Inferential
used to make inferences from the sample to the population
- goal is to determine probability of some event occurring
- ex. t-test and ANOVA
Statistical analysis may be
Parametric
used when a sample is randomly drawn from a population and the data is normally distributed
- para (two-sided) data that yields a bell curve
- assume the variance of the sample is homogeneous (similar) to the variance of the population from which your sample is drawn
- scores would give you normal bell curve
- ex. t-tests and ANOVA
Statistical analysis may be
Nonparametric
used when you cannot make any assumption about the shape of the curve or variance of the population scores (they may not be normally distributed and variances may not be homogeneous)
ex. Chi-square, Mann-Whitney U Test, Wilcoxon Signed-Ranks Test
Variables
Independent variable
the variable you manipulate or vary to see what change occurs in the DV
- precedes or is antecedent to DV
- sometimes you group/categorize IV (ex. you categorize a group of individuals by gender F/M; high school students into grades)
- sometimes called stimulus variable, predictor variable, experimental variable
Variables
Dependent variable
variable you are measuring or trying to change. Value of this variable depends upon the value of the independent variable you selected
ex. the effects of three kinds of therapy (IV) on anxiety (DV)
- also called respone variable, outcome variable, criterion variable
Research questions and hypotheses
Some ask research questions to be answered
ex. Is there a relationship between disciplinary practices and leadership styles for men in the military?
ex. is there is a significant difference in the mean number of client contacts between public and private counseling agencies?
Research questions and hypotheses
Null hypothesis
states there is no difference between the variables or groups measured
ex. There are no significant differences in final academic grades (GPA) between boys and girls finishing 10th grade)
Research questions and hypotheses
Alternative hypothesis (directional)
states that one group’s scores will be significantly different from another group’s score (one-tailed test)
Ex. the GPA of girls will be higher than the GPA of boys finishing 10th grade.
Research questions and hypotheses
Alternative hypothesis (non-directional)
there will be differences between the groups but which group has higher/lower scores is not indicated
Ex. the GPA of girls and boys finishing the tenth grade will be different.
Significance level
Significance level
will determine likelihood of making Type I or Type II errors
Significance level
How to select level for significance
generally set at 0.05 or 0.01 (or up to 0.001)
level selected is willingness ot make an error (rejecting null hypothesis when there is not a significant difference between the groups)
- at 0.05, you are willing to accept the possibility of rejecting the null hypothesis in error 5 times out of 100
Type I error (Alpha)
rejecting the null hypothesis (which states that there is no difference) when it is correct
! You can change the probability of Type I error by changing significance level
- Saying there is a difference when there isn’t
- Saying there is a wolf when there is not a wolf
Type II error (Beta)
failure to reject the null hypothesis when there is a difference
! Small sample sizes can result in Type II errors
- Saying there is no difference when there is a difference
- Villagers believing there is no wolf when there actually was one
Negative relationship between Type I and Type II errors
as significance level goes down (e.g., from 0.05 to 0.01), Type I error decreases but Type II error increases
As one goes up, other goes down
T-test
used to determine whether mean scores of two groups are significantly different from each other
- can only be used when there are two groups (two mean scores)
- You would compare your obtained value of t with the value of t presented in a Table of T values to make the determination
Difference between T-test and ANOVA
t-test looks at differences between two groups
ANOVA is 3+ groups
One-Way Analysis of Variance (ANOVA)
examines the differences between two or more (usually 3+) groups based on one IV (that has at least 2 levels)
- this yields an F-ratio which can be compared to other values listed in F Distribution Table to determine whether significant differences are present
- ex. types of therapy (levels: CBT, DBT, ACT)
Use One-Way ANOVA when there is only 1 IV with multiple levels)
Factorial ANOVA
used to determine if mean scores on 2+ IVs (factors) differ significantly from each other and whether the factors interact significantly with each other
- Types of counseling (3 levels: CBT, DBT, ACT)
- Duration of counseling (2 levels: 4 weeks, 8 weeks)
- One DV: symptoms of anxiety
- Factorial design: 3 x 2
Multivariate ANOVA (MANOVA)
** USE when have 2+ DVs**
- integrates all DVs into a single composite variable
- ! DVs must be related!!
- used to gain insight into how different factors/treatment influence multiple outcomes simultaneously
ex. effect of MSC-T program on internalizing disorders (depression, anxiety, somatic symptoms)?
- IV: two groups: treatment/no treatment
- DV: single composite variable of internalizing disorders (depression, anxiety, somatic symptoms)
Study tip: Man, what a lot of DVs
Analysis of Covariance (ANCOVA)
used when influence of one or more IVs on the DV is controlled (i.e., initial group differences are adjusted statistically on one or more variables that are related to DV)
Covariate: variable that influence the DV but are not of primary interest
- ex. Therapy (2 levels: DBT and Gestalt)
- DV: Depression level
- Covariate: peer support
Identifiers for each type of ANOVA
One-Way ANOVA: only one IV with multiple levels (ex. multiple types of therapy)
Factorial ANOVA: 2+ IVs, 1 DV (ex. medication and types of counseling)
MANOVA: 2+ IVs; 2+ DVs integrated into single composite variable
ANCOVA: used when influence of one or more IVs on the DV is controlled (covariate)
Post hoc or multiple comparisons tests
Post hoc tests
if your ANOVA yields a significant F value, you still will not know which particular pair of mean scores is significantly different from each other
- must apply post hoc tests to determine between which groups the significance lies
Ex. Tukey’s HSD, Scheffe
Nonparametric tests
use when you cannot assume that your distribution of scores is normally distributed (resembles a normal curve) or that the variance of your sample is similar to the variance of the population (homogeneity)
Nonparametric tests
Mann-Whitney U Test
when you collect data from two samples that are independent (uncorrelated/unmatched) from each other and the scores are not normally distributed
Study tip: Mann-Whitney U = uncorrelated or unmatched
Nonparametric tests
Wilcoxen signed-rank test
when you have scores for two samples and these scores are correlated (you matched them or got two scores for each individual - repeated measures)
However, the scores do not approximate a normal distribution
study tip: Wilcoxen = correlated
Nonparametric tests
Kruskal-Wallis test
when you have more than two mean scores on a single variable. A nonparametric one-way ANOVA
Parametric with Non-parametric Equivalent
Independent t-test
Mann-Whitney U test
Parametric with Non-parametric Equivalent
Dependent t-test
Wilcoxon Signed-Rank Test
Parametric with Non-parametric Equivalent
One-way between-groups ANOVA
Kruskal-Wallis test
Parametric with Non-parametric Equivalent
One-way within-groups ANOVA
Friedman test
Dumb Mnemonic Device for Parametric with Non-Parametric Equivalent
Man
What
Krispy
Fries
Just remember the order of Parametrics (T-test indep. dep. ; ANOVA between within)
Independent t-test | Mann-Whitney
Dependent t-test | Wilcoxon
One-Way Between Groups ANOVA | Kruskal-Wallis
One-Way Within Groups ANOVA | Friedman
Chi-square test
examines the relationship between two or more categorical (nominal) variables
- utilizes a contingency table: displays frequencies for categorical variables
Study tip: chi = contingency
Chi-square test example
Is there a significant association between gender (F/M) and preference for learning mode (online/in-person) among students?
IV: Gender (F/M); Learning mode preference (online/in-person)
Then do a contingency table displaying the data of males who prefer online/in-person; females who prefer online/in-person; totals
Between Groups Design Experiments
Posttest-Only Control Group Design
participants are randomly assigned to either a treatment or control group and are tested on DV once
- compares how groups differ from each other after treatment to determine the effect
Between Groups Design Experiments
Pretest-Posttest Control Group Design
participants are randomly assigned to either treatment or control group and are tested on DV twice
- measures the effect of a treatment (IV) on an outcome variable (DV) by comparing the group scores before/after exposure
- should use same scales for pretest and posttest
Between Groups Design Experiments
Solomon Four-Group Design
takes into account the influence of pretesting on susequent posttest results
- allows you to determine whether the pretest by itself made a difference, whether the treatment by itself made a difference, whether a combination of pretest and treatment made a difference, or whether nothing mades a difference
Multiple regression
the use of the correlation coefficient to determine the strength of the relationship of predictor (independent) variables on a criterion (dependent) variable
- adds together the predictive power of several IVs (predictors)
ex. predictor variables like high school GPA, class rank, ACT scores may be used to predict the criterion (outcome) variable, which could be freshman year GPA
Scatterplot (scattergram)
a graphic representation of the relationship between two variables for a group of individuals
each point on graph is an individual score (stress score on horizontal X axis and depression score on vertical Y axis)
- can draw a line of best fit to display the direction, form, and strength of the relationship
Correlation coefficient
measures and describes the relationship between two variables
** Pearson correlation coefficient (r)** measures a linear correlation
- sign (+/-) indicates direction of the relationship
- numerical value (0.0-1.0) indicates strength of relationship
Study tip: pearson r is used for interval or ratio data
Factor analysis
a statistical method using the correlation coefficient to determine whether a set of variables can be reduced to a smaller number of factors
ex. breaking SES into income, education, occupation factors etc.
Factor analysis example
ex. a factor analysis of all the items on a long inventory with 15 scales may uncover only 4 or 5 factors independent of each other underlying the scales. Thus many of the constructs of the 15 scales overlap
Likert scale
measures attitudes/opinions
allows for several response choices
0 - not at all
1 - occassionally
2 - often
3 - always
Biserial correlation
an appropriate correlation coefficient to use when one variable yields continuous data and the other yields data that is dichotomous
Cross-sectional
studying or measuring characteristics of several groups at the same time
Longitudinal
studying or measuring characteristics of a group over a period of time
also called diachronic method
Degrees of freedom
the number of observations that are free to vary
Single-blind technique
the subject does not know whether they are in control or experimental group
this helps eliminate demand characteristics which are cues/features of a study which suggest a desired outcome (the subject can manipulate/confound an experiment by purposely trying to confirm or disprove hypothesis
Double-blind technique
occurs when neither the researcher nor the subject knows who is getting the active substance or the placebo
reduces experimenter effects - which flaw an experiment because the experimenter might unconsciously communicate their intent or expectations to the client
Halo effect
the tendency for the observer (researcher or data collector) to form an early impression of the person being observed and then letting this impression influence observations or ratings of that individual
- can be positive or negative
Can happen with things that are not being evaluated having an impact on the ratings (like finding someone attractive makes you rate them higher on their counseling skills)
Heteroscedasticity
one end of a distribution of scores has more variability than the other end resulting in a fan-like appearance
Homoscedasticity
there is an equal distribution of scores throughout the range of scores (i.e., around the line of best fit)
Inter-rater reliability
In qualitative research, the reliability calculated by correlating the responses of several raters
Observer bias
the tendency of researchers to see, hear, remember what they want to
Pilot study
a preliminary trial/test of research techniques and measures
Placebo
control treatment that gives subjects the same amount and kind of attention as experimental group subjects get (reduces hawthorne and Rosenthal effects)
- could be sugar pill etc.
Rank-order correlation (Spearman rho)
used when the values of the variables are reported in rank form rather than continuous
used for ordinal data (rho ends in O!)
Counseling program evaluation
Counseling program evaluation began as
emphasis on accountability in human services field accelerated in 1970s and continues today coming primarily from funders like Health Maintenance Organizations, insurance companies, gov. funding sources, etc.
- determines a ‘bottom line’
Counseling program evaluation
Need for counseling program evaluation
acute need to demonstrate the efficacy of counseling in general and effectiveness of specific theories, techniques, approaches in particular
- emphasis on short-term therapy (6-12 sessions) argues for research and evaluation to determine what works well for what kinds of problems with what clients under what circumstances
Counseling program evaluation
Goals and measurable objectives must be
specified in advance
- without those, evaluation data has little relevance
Counseling program evaluation
On what level (community, individual, etc.) does evaluation occur
the effectiveness of counseling techniques/processes often occurs on individual client basis
Counseling program evaluation
Evaluation
the systematic collection of evidence of the worth of a program, process, technique
Counseling program evaluation
Types of evaluation:
Formative evaluation
ongoing, process evaluation to measure the effectiveness of a technique or part of a program
- tries to determine how well a new technique, process, treatment works
- process evaluation
Counseling program evaluation
Types of evaluation:
Summative evaluation
summary or product evaluation designed to measure the effectiveness of a program, usually conducted at the end of a cycle such as a school year, fiscal year, etc.
- conducted to see how well agency or program goals have been met
- usually a product (document) is generated so this is ‘product’ evaluation vs ‘process’ evaluation (formative)
Ethical Issues in Research
Confidentiality
no one should have access to the data except for the researcher and research assistants
- release of research data to others is only ethical with consent of the subject
Ethical Issues in Research
Deception
deception may be justifiable if no risk to subjects is involved. Such research should be followed by debriefing of the subjects
Ethical Issues in Research
Informed consent
subjects should be informed of the research they will be participating in and give their consent
Ethical Issues in Research
Ethical issues revolve around benefits of research
significance of research results should outweigh the potential benefits denied to the control group
Ethical Issues in Research
Institutional Review Board (IRB)
approves human subject research projects when conducted within institutions or agencies when federal funding is involved
Writing research
- most writing should be done in APA format
- sexist language is to be avoided
- never submit a manuscript to more than one journal for publication consideration at a time
APA’s Journal of Counseling Psychology
publishes more counseling research articles than any other periodical in our field
Ethics in research
- subject is informed of risks
- negative after-effects are removed
- subjects can withdraw at any time
- confidentiality of subjects is protected
- results will be presented in an accurate format that is not misleading
- use only techniques that you are trained in
Hypothesis testing pioneered by
R.A. Fisher
Biserial correlation
one variable is continuous (i.e., measured using an interval scale) while other is dichotomous
ex. trying to correlate state licensing exam scores to NCC status
Phi-coefficient correlation
both variables are dichotomous (two valued)
ex. correlating gender with certification status (does or does not have a certification)
Withdrawal designs
ABA or ABAB
ABA: baseline is taken (A), intervention is implemented (B), new baseline taken (A)
ABAB: if the pattern for the second AB administration mimics that of the first, then the chances increase that B (intervention) caused the changes rather than extraneous variable
- ABAB is good to rule out extraneous variables
Empirical rule
68-95-99.7
68% of scores fall within +/-1 SD, 95% within +/-2SD, 99.7% within +/-3 SD
Mode
the highest point on the curve!
the point of maximum concentration
The benefit of standard scores like percentiles, t-scores, z-scores, stanines, SDs over raw scores is that
a standard score allows you to analyze the data in relation to the properties of the normal bell curve
horizontal x axis is also known as
abscissa
vertical y axis is also known as
ordinate
Stanine
divide the distribution into 9 equal intervals with stanine 1 as lowest 9th and 9 as the highest 9th.
5 would be the mean
Survey problems
- need at least 50-75% completion rate to be accurate
- poor construction of the instrument
- low return rate
- subjects are often not picked at random
Nocebo effect
a placebo effect with a negative effect
ex. when a doctor says that individuals with this condition live for 6 weeks
Standard error of measurement (SEM)
tells the counselor what would most likely occur if the same individual took the same test again
