Week 2: Guidelines for measurement and evaluation process; Validity, reliability and objectivity Flashcards
Describe a needs assessment (14 points)
- Initial assessment to determine needs of the program - What do we want to improve and why is that important?
- Exercise science:
- Initial fitness levels
- Skill levels
- Prevention of injuries
- Knowledge
- Sports management:
- Attendance
- Decrease cost associated with injury
- Merchandise sales
- Public perception of your club
- Participation levels
- Serve as an initial pre-test to obtain baseline data
- Enables progress to be monitored
Describe program development process (7 points)
- An active plan of program development should be in place in any activity based setting.
- Ongoing process is:
1. Establish program philosophy
2. Develop program goals
3. Plan program activities
4. Deliver the program
5. Measure and evaluate the program
Describe establishing a program philosophy (8 points)
- Perhaps the most important task- Ask yourself why.
- Several philosophical aims would likely be included in any quality program:
- Provide optimal participation for clientele
- Educate the participants about benefits of regular activity
- Systematic evaluation and improvement of the program
- Philosophy of a program should reflect characteristics and needs of the participants.
- Need to consider how much emphasis on each of the learning domains.
- A logical end to this stage = preparation of a mission statement.
Describe developing program goals (4 points)
- Program goals evolve from the philosophical aims
- Primary link between philosophy and activities of program
- Program goals should be ‘SMART’
- Start with one main goal that can then be broken down into smaller goals
Describe planning program activities (9 points)
- When choosing activities:
- Sequence activities in logical order
- Determine desirable outcomes for each activity
- Must consider :
- Participant needs, characteristics and interests
- Availability of facilities and equipment
- Length of program
- Instructor expertise
- Climate
Describe delivering the program (3 points)
- Constantly monitor the program throughout the delivery phase.
- Make modifications as necessary based on assessments and professional observations.
- For example, Coaches constantly assess their players in practice and game situations. During the season, the coach and athletes make constant adjustments
Describe Evaluating & Improving the Program (8 points)
- Complete evaluation after program has been delivered. Need to be done consistently and regularly
- When is the best time to evaluate in:
- School/ academic settings = end of year/semester
- Management = end of financial year
- Adult fitness settings = six weeks
- Athletic training programs = at the end of a training block (mesocycle)
- Evaluating the effectiveness of a program is an ongoing process. Continually needs fine tuning for improvement and to remain current
- With the wide range of measurement tools available, need to ensure you use assessment procedures that are appropriate.
Describe the Guidelines for the Measurement & Evaluation Process (9 points)
- Relevant – to program outcome statements
- Learning experience for client
- Enjoyable – motivational
- Discriminative – continuum – not S/U
- Economically feasible - cost efficient
- Independent – items in battery measure different variables
- Gender appropriate – account for differences in a nondiscriminatory way
- Performance not reliant on another’s performance
- Safety – physical e.g. exhaustion, psychological
Describe quantitative and qualitative tests (4 points)
- Quantitative = number
- Examples - survey, multiple choice tests, performance statistics
- Qualitative = words
- Examples - interview, focus group, open-ended questionnaire
Describe Criterion-Referenced Measurement and Norm-Referenced Measurement (5 points)
Criterion-Referenced Measurement
- Compare against a standard or criterion.
- Not compared to the performance of other individuals
- Example: assessment rubric for an assignment
Norm-Referenced Measurement
- Compares individuals’ performances.
- Examples of norms are percentiles, z-scores and T-scores.
What are the advantages and disadvantages of criterion referenced testing? (5 points)
Advantages:
- Performance linked to specific outcomes
- If standards not met – evaluations made to improve
- Competition is based on reaching standard, not bettering someone else’s performance
Disadvantages:
- Scores always involve some subjective judgement
- Motivation based on reaching/not reaching cut-offs
What are the advantages and disadvantages of norm referenced testing? (5 points)
Advantages:
- Administration procedures standardised
- Quality of test items generally high
- Assume statistical rigor
Disadvantages:
- The time the norms were established – need to be updated periodically
- Individuals may not see link between what they have learnt and assessment
List the three criteria that must be met when developing and/or finding a measure (3 points)
Ensure that it is:
- VALID – measures what you intend.
- RELIABLE – gives consistent/trustworthy results.
- OBJECTIVE – reliable administration of test.
List the types of validity (8 points)
- Content validity
- Construct validity
- Criterion validity
- Concurrent validity
- Predictive validity
- Ecological validity
- External validity
- Trustworthiness
Describe Content validity (5 points)
- A.K.A - Logical or face validity
- Relies on logic and comparison
- The content of the test must measure all the aspects of what you want to know
- Weakest test of validity
- Example: when choosing a test to measure swimming ability, the test would involve swimming and not running
