Evaluating Impact - Creating Data Collection Tools Flashcards
What are the key decisions that need to be made when creating data collection tools?
- the purpose of the evaluation instrument
- what will and will not be included in it
- the type of evaluation to use
- how to isolate the data to make valid comparisons
- determine control groups
- get management support and overcome barriers
What are some tools that test knowledge, and what is important to remember about them?
Exams, assessments, and tests
They may not be the best way to test knowledge.
Validity
the evaluation instrument measures what it was intended to measure
Important because it ensures all learners interpret the meaning of a test question the way it was meant
How do you verify validity?
Solicit feedback from a subject matter expert
5 ways to determine whether an instrument is valid
- content validity: the extent to which the instrument represents the program’s content
- construct validity: the degree to which the instrument represents the construct it’s supposed to measure (the construct is the abstract variable that should be measured, such as knowledge or skill)
- concurrent validity: the extent to which an instrument agrees with the results of other instruments administered at approximately the same time to measure the same characteristics
- criterion validity: the extent to which the assessment can predict or agree with external constructs; determined by looking at the correlation between the instrument and the criterion measure
- predictive validity: the extent to which an instrument can predict future behaviors or results
Test validity boils down to 2 things
- the test should be reasonably reliable and free from measurement errors
- the test should include all of the content that is needed to perform the job safely and competently
Reliability
the ability of the same measurement to produce consistent results over time
How do you determine reliability?
- the instrument must be administered to a sample of participants and undergo statistical analysis; should have reliability coefficients at or above 75%
- non-scientifically, it can be determined and improved by wording questions and evaluating responses over time
What are the possible consequences if a test is too easy?
poor job performance, on the job accidents, damage to expensive equipment
What factor maximizes test validity and test reliability?
the degree of difficulty
What could a test that seems to have a low degree of difficulty indicate?
Testers knew the information already, the test was too easy, or the answers were cued in some way
What could a test that seems to have a high degree of difficulty indicate?
information wasn’t presented adequately in person or in reading materials or the item was so difficult only the most knowledgeable could answer it
What are two ways to test reliabilty?
Split-half reliability, test-retest check of reliability
Split half reliability
Two groups of testers take half of the test; the correlation of results is calculated. Then the group takes the other half as their re-test.
Test-retest check of reliability
The same test is administered twice to the same group of people
Shortcoming of test-retest check of reliability
Memory bias; people just remembered the answers. Too short of time between tests.
Too long of time between tests and other factors enter the equation, like learning new information
When would a TD professional use an instrument other than an exam? What would they use?
When evaluation is tied to a needs assessments
They may use surveys, questionnaires, observations, and and interviews
What 4 things should you remember when creating surveys, questionnaires, and interview guide instruments?
- Be certain that the questions are directly connected to the measurement plan.
- Determine whether any definitions or other standards exist that need to be clarified.
- Decide whether reading ability or a second language is a concern.
- Explore whether to use a pilot test on the instrument.
Advantages and disadvantages of assessment and tests
Advantages:
- they’re objective
- identify the specific gap between the current and desired performance, knowledge, and skills
Disadvantages:
- they can be time-consuming
- don’t always examine the thought processes that support why an individual performed in a certain way
Advantages and disadvantages of observation
Advantage:
- they are great for measuring skill ability (Evaluating Impact, not Consulting)
- it may be used to create a step-by-step procedure (algorithm) that can be standardized for all learners as a flowchart, diagram, graphic, list of steps, or job aid
- can capture job environment conditions that make a difference and can be incorporated into solution (Consulting, not Evaluating Impact)
- Provides realistic view of situation
Disadvantage:
- some performers may not act as they normally would because they know they are being watched (known as the Hawthorne effect)
- only indicates behavior, not reason for the behavior
- can be difficult to identify when one portion ends and the next begins (Consulting, not Evaluating Impact)
Advantages and disadvantages of interview
Advantages:
- clarify ambiguous or confusing information obtained otherwise
- gives participants ownership in the process
- they provide rich detail through a two-way conversation to clarify statements or ensure understanding of questions
- in-depth information
- use the same wording and interview protocol for consistency
- can be used to design quant tools
Disadvantages:
- can be time consuming and labor intensive
- the interviewer must be careful to record exact responses and not interpret them
- interviewer must represent the target population
Advantages and disadvantages of focus groups
Advantage:
- it can create richer concepts as individuals build on one another’s ideas
- able to observe nonverbal behavior (Consulting, not Evaluating Impact)
- able to interview more people in a shorter amount of time
Disadvantage:
- the group can be influenced by particularly verbal individuals, which gives the impression of unanimity when it is not the case
- time and resource intensive
Advantages and disadvantages of surveys
Advantages:
- inexpensive
- the results are easy to tally
- they provide quick results
- they can reach a lot of people at a distance (participation is easy)
- can be qualitative and quantitative
Disadvantages:
- return rates can often be low, thus not giving a good sample
- they also need to be worded in a way so that everyone understands them
- questions don’t allow for free expression
- constructing questions and selecting scale must be done carefully
What is the best survey type?
The one that provides the needed data, not the one that’s the fastest, cheapest, or eaisest
