AOR 4 Evaluation & Research

Question 1

Both program evaluations and research require designing plans that include ____________________-

Answer 1

selecting adapting or creating valid & reliable data collection instruments, developing sampling plans, collecting & managing data, analyzing collected data, interpreting results, applying findings & communicating results

Question 2

What is the primary focus of evaluations? Research?

Answer 2

Evaluations - determining if program objectives have been met/success of program

Research - identifying new knowledge or practices that answer questions/test hypotheses about health-related theory, behavior, or phenomenon

Question 3

Delimitations & give examples

Answer 3

decisions made by evaluator or researcher that are used to help identify parameters & boundaries set for a study (helps to manage scope of study)

• why some literature is not reviewed
• why populations are not studied
• why certain methods are not used

Question 4

Evaluation

Answer 4

series of steps that evaluators use to assess a process or program to provide evidence and feedback about the program

Question 5

Limitations

Answer 5

phenomena evaluator/researcher cannot control that place restrictions on methods & conclusions

• e.g. time, nature of data collection, instruments, sample, analysis

Question 6

Logic Model

Answer 6

depict aspects of a program to include inputs, outputs, & outcomes

• provide scaled down, somewhat linear, visual depiction of programs

Question 7

Research

Answer 7

organized process in which a researcher uses the scientific method to generate new knowledge

Question 8

Reliability

Answer 8

Consistency, dependability, & stability of measurement process

Question 9

Validity

Answer 9

degree to which a test/assessment measure what it is intended to measure

Question 10

Unit of Analysis

Answer 10

what or who is being studied or evaluated

Question 11

Formative Evaluation

Answer 11

Process of assessing quality of a program during planning and implementation

• used by evaluators & researchers

Question 12

Process Evaluation

Answer 12

measures quality of performance and delivery throughout program implementation

Question 13

Summative Evaluation

Answer 13

evaluation occurs after program has ended

• Designed to produce data and information on the program’s efficacy or effectiveness during its implementation phase
• Provides data on the extent of the achievement of goals (impact & outcome)

Question 14

Impact Evaluation

Answer 14

Immediate & observable effects of a program leading to desired outcomes

Question 15

Outcome Evaluation

Answer 15

assesses whether there is also a demonstrable change in the targeted health outcome

• What changed about the public health problem
• Focuses on ultimate goal, product, or policy
• measured in health status, morbidity, mortality, etc.

Question 16

Choosing what instrument to use to collect data is based on ______________

Answer 16

goal of data collection, population under investigation, & resources of those trying to collect data

• data collection instruments have both advantages & disadvantages

Question 17

HES must follow _________________ when planning & conducting evaluation

Answer 17

federal & state laws and regulations, organizational & institutional policies, & professional standards

Question 18

Data instruments (whether new, existing, adapted) should be tested for ________________

Answer 18

literacy reading level

Question 19

Why should data collecting instruments be tested?

Answer 19

To ensure validity of responses

Question 20

Examples of instrument readability tools

Answer 20

SMOG and FleschKincaid

Question 21

Advantages and Disadvantages of using existing data collection instruments

Answer 21

Advantages: previously tested for reliability, validity, direct comparison measures, reduced cost (compared to creating new instrument), & user familiarity

Disadvantage: potential for unreliable measures given different population demographics & situations

Question 22

What should HES consider before using an existing data collection instrument?

Answer 22

• if item is appropriate for intended purpose
• if language appropriate for population
• whether test has been performed using sample from intended population
• to whom you should give credit for using instrument

Question 23

Difference between research and evaluation

Answer 23

Research: conducted with intent to generalize findings

Evaluation: determine if a specific program was effective

Question 24

The most rigorous evaluation or research design available should be used. However, what are things that could require less rigorous approach?

Answer 24

Ethics, cost, politics, & resources

Question 25

Considerations of implementation

Answer 25

* find reliable, trustworthy, skilled people to COLLECT, ENTER, ANALYZE, & MANAGE data to ensure quality results * define roles, responsibilities, & skills needed to collect QUALITATIVE data(focus groups, interviews, etc) which may differ when collecting QUANTITATIVE data (surveys) * monitor data collection to ensure implementation of process will assist in maintaining established time frames & objectives * maintain integrity of data collected & ensure protocols address quality control measures (during both collection and entering of data)

Question 26

Field procedures for data collection

Answer 26

* protocols for scheduling initial contacts with respondents * Introducing instrument to respondent * keeping track of individuals contacted * following up with non-respondents when appropriate

Question 27

All ata must be carefully ____________ into a useable format

Answer 27

coded & organized

Question 28

Research and Evaluation can help record ______________ & ____________

Answer 28

what changes have occurred & identify what led to those changes

Question 29

Advantages & Disadvantages of online surveys

Answer 29

Advantages: cost-effective, versatility, larger potential reach, convenience, anonymity of respondents, risk of data entry errors significantly reduced during transcription, Disadvantages: lower response rate, language barriers, not feasible for all populations (lack of internet access), risk for multiple response rates

Question 30

What should data management/analysis plan include?

Answer 30

* procedures for transferring data from instruments to data analysis software * detail how data will be scored & coded * how missing data will be managed * how outliers will be handled * scoring guide * Data Screening - assessing accuracy of data entry

Question 31

What can data screening tell us?

Answer 31

If statistical assumptions are met

Question 32

Problematic Outliers

Answer 32

not representative of population

Question 33

Beneficial Outliers

Answer 33

those representative of population

Question 34

Multivariate Outliers

Answer 34

unusual combinations of scores on different variables * hard to detect without statistical tests

Question 35

Missing Data

Answer 35

observations that were intended to be made but were not

Question 36

What should guide data analysis?

Answer 36

research/evaluation questions & level of measurement of data

Question 37

Correlates (relationship/connection where something affects or is dependent on another) can be derived through ______________________

Answer 37

interpretation of data, reach & effectiveness or size of effect

Question 38

HES need to incorporate ___________________ to their evaluation & research findings & __________________ into decision making, policy development, & implementation of programs

Answer 38

evidence based practice approach; scientific evidence

Question 39

What does evidence-based practice approach look like?

Answer 39

* best available scientific evidence (from literature) & data are combined * program planning frameworks are employed * engage the community * programmatic evaluation is used * results are disseminated

Question 40

How doe HES plan for future programs/interventions?

Answer 40

interpretation of evidence to determine significance & draw relevant inferences * data interpretation is stronger when including key stakeholders and getting community perspective during evaluation process

Question 41

Findings from a study must be analyzed based on specific ________________ of the study

Answer 41

Delimitations e.g . narrowing study by geographic location, time, & population traits

Question 42

Findings from evaluation & research are subject to systematic error known as ____________

Answer 42

BIAS - from sampling, design, implementation, or analysis

Question 43

Confounding Variables

Answer 43

extraneous variables outside scope of intervention that can impact the results * variables that are not accounted for in the study design

Question 44

What should be done before applying recommendations to program or policies?

Answer 44

stakeholders have opportunity to review & discuss research or evaluation findings