Chapter 10 & Readings

Question 1

What are some critical issues related to data collection?
A. Language of participants
B. Literacy level of participants
C. Use of a dominant or colonizing language
D. All of the above

Answer 1

D

Question 2

Mertens and Wilson recommend that when you use mixed methods data collection that you be aware of the implications of your ontological and epistemological beliefs.
A. True
B. False

Answer 2

True

Question 3

Reliability can be influenced by how the instrument is administered.
A. True
B. False

Answer 3

True

Question 4

Reliability and validity are commonly used terms to describe the quality of quantitative data collection. What does validity mean in this situation?
A. Does the instrument measure cultural competence?
B. Does the instrument (as used with the participants) really measure what it is supposed to measure?
C. Does the instrument measure what is it is supposed to measure consistently?
D. Does the instrument reliably measure what it is supposed to measure over time?

Answer 4

B

Question 5

An evaluator created a test. In order to test reliability, he had the participants take the test and he analyzed the results to examine the consistency of their responses. What is this an example of?
 A. Repeated measures reliability
 B. Intraparticipant reliability
   C. Internal-consistency reliability
 D. Multi-dimensional reliability

Answer 5

C

Question 6

In multiple regression, when we say that we control for the effects of some variable(s) we are:
A. statistically adjusting or subtracting the effects of a variable to see what a relationship would have been without it
B. actually removing a variable from a model so that it does not interact with the effects of other variables
C. changing the mediating capabilities of an endogenous variable
D. changing the mediating capabilities of an exogenous variable

Answer 6

A

Question 7

What are some important things to notice when you are conducting observation based on Michael Patton (2002b) as discussed in your textbook?
A. Observing what does not happen, program setting, and native language used
B. Social setting, program activities and behaviors, and nonverbal communications
C. Informal interactions and unplanned activities
D. All of the above

Answer 7

All of the Above

Question 8

What is the validity/credibility evidence strategy used when evaluators share data with participants to obtain feedback on perceived accuracy and quality?
   A. Multiple data sources
 B. Member checks
 C. Persistent observations
 D. Progressive subjectivity

Answer 8

B

Question 9

What is INTRArater reliability?
A. It is used to determine whether a single rater or observer is consistent over time.
B. It compares the data of two raters or observers to see whether they are rating the same behavior consistently.
C. It is used to compare two kinds of data collection to see whether they are describing the same event.
D. It is used to compare when different raters are administering similar instruments

Answer 9

A

Question 10

What are some forms of evidence used to support validity/credibility in quantitative data collection?
A. Construct validity and criterion-related validity
B. Peer debriefing
C. Member checks
D. Persistent observations

Answer 10

A

Question 11

What does the depiction of an evaluand include:

Answer 11

Specification of outputs
outcomes
knowledge
impacts

Question 12

What are the levels that outcomes and impacts need to be measured?

Answer 12

individual client level
program or system level
broader community level
organizational level

Question 13

___________ is a critical issue that permeates decisions about data collection.

Answer 13

Language

language differences can be translated and then back translated into the original language

Question 14

_________ means does the instrument really measure what it is supposed to measure?

Answer 14

Validity

Question 15

______________ is the consistency in measurement. Does the instrument measure what it is supposed to.

Answer 15

Reliability

Question 16

Evaluators in the Values Branch developed parallel critiera for the quality of qualitative evaluations; ________ instead of reliability and __________ instead of validity.

Answer 16

dependability: credibility

Question 17

Reliability Coefficient can be interpreted in two ways_______ and ____________.

Answer 17

Coefficient of stability and alternate-form coefficient

Question 18

__________ is when the evaluator administers the same instrument twice, separated by a short period of time. Results are compared using a statistic such as correlation coefficient, also called test-retest reliability.

Answer 18

Coefficient of stability

Question 19

___________ is when the evaluator administers two equivalent versions of the same instrument (parallel forms) to the same group of people. Results are compared using a coefficient of stability.

Answer 19

Alternate form coefficeint

Question 20

____________ is when participants take one instrument, their socres are subjected to an analysis to reveal their consistency of responses within the instrument.

Answer 20

Reliability/precision

Question 21

_______________ is when 2 observer’s data are compared to see whether they are consistently recording the same behaviors when they view the same events.

Answer 21

Interrator reliability

Question 22

______________is used to determine whether a single observer is consistently recording data over a period of time.

Answer 22

intrarater reliability

Question 23

___________ can be influenced by how the instrument is administered.

Answer 23

Reliability

Question 24

Psychologists recommend the use of different types of evidence to support ________ claims.

Answer 24

validity

Question 25

_________ is considered the unitary concept of validity. It is to what degree does all accumulated evidence suppor the intended interpretation of scores for the proposed purpose.

Answer 25

Construct Validity

Question 26

_________ refers to items on the test that represent content covered in the program.

Answer 26

Content-related evidence

Question 27

__________ indicates that the measure actually reflects current or future behaviors or dispositions.

Answer 27

Criterion-related evidence

Question 28

Evaluators need to be aware of the consequences of using data, especially with regard to the potential to worse inequities

Answer 28

Consequential evidence

Question 29

Evaluators need to stay on site for sufficient time

Answer 29

Prolonged and substantial engagment | Strategy to enhance credibility

Question 30

Observations need to be conducated a variety of times of the day, week, and year

Answer 30

Persistant observations | strategy to enhance credibility

Question 31

An evaluator should find a peer with whom to discuss the study at different stages

Answer 31

Peer debriefing | strategy to enhance credibility

Question 32

Evaluators need to be aware of their assumptions, hypotheses, and understandings, and of how these change over the period of the study.

Answer 32

Progressive subjectivity | strategy to enhance credibility

Question 33

Evaluators can share their data with participants to obtain feedback on the perceived accuracy and quality of their work.

Answer 33

Member Checks | strategy to enhance credibility

Question 34

Qualitative evaluators recommend the use of _______.

Answer 34

Multiple Data Sources | strategy to enhance credibility

Question 35

Triangulation

Answer 35

The use of multiple data of multiple data sources and different data collection strategies to strengthen the credibility of the findings of an evaluation

Question 36

___________ evaluators want to assume their stakeholders that they have measured what they say they are measuring.

Answer 36

Methods Branch

Question 37

____________ evaluators want to provide evidence of their believability of their findings.

Answer 37

Values Branch

Question 38

___________ evaluators begin data collection by acknowledging power differences between themselves and study participants as well as the need to establish a trusting relationship with community members.

Answer 38

Social Justice Branch

Question 39

What are examples of Quantitative Data collection methods?

Answer 39

Tests, performance and portfolio assessments, surveys, goal attatinment scaling, and analysis of secondary data sources

Question 40

What are the types of tests?

Answer 40

standardized, locally developed, objective, nonobjective, norm-referenced, criterion referenced *Pilot testing is important

Question 41

Strong survey research utilizes what kind of samples

Answer 41

random

Question 42

survey data collected from the research participants during a single, relatively brief time period

Answer 42

cross-sectional study

Question 43

Longitundinal, panel and Trend studies; describe them

Answer 43

longitudinal collected at one than one point over time panel study is a synonym for longitudinal trend-samples are taken over time with the same questions asked, diff than Long because different people are selected.

Question 44

A technique used to measure the meaning participants attach to various attitudinal objects or concepts:

Answer 44

semantic differential technique

Question 45

A term whose meaning is debated by philosophers, but in everyday language implies that manipulation of one event produces another event

Answer 45

Causation

Question 46

Effect The difference between what would have happened and what did happen when a treatment is administered

Answer 46

Effect