Lecture 2

Question 1

Name the four possible measurement modes for typical performance tests

Answer 1

• self report
• other report
• somatic indicators
• physical traces

-> these measurement modes can be reactive (people can influence their score on the construct) or nonreactive

Question 2

The literature reports different strategies to construct typical performance tests. Three broad classes of strategies are distinguished

Answer 2

• intuitive class: the relation between the construct and the items is of an intuitive nature
• inductive class: the tests are derived from empirical data
• deductive class: the tests start from theoretical or conceptual notions of the construct

Question 3

The intuitive class consists of …

Answer 3

• the rational method: uses a loose description of the construct that is based on the knowledge of experts or members of the target population
• the prototypical method: asks members of the target population to think of persons having the construct, and write down their behavior that is typical for the construct

Question 4

The inductive class consists of …

Answer 4

• the internal method: starts from a broad pool of personality or attitude items. The items are administered to a sample of the target population, and the associations are computed between item scores. The test developer searches for clusters of items.
• the external method: starts from a broad pool of items and a criterion that has to be predicted. The items and the criterion are measured in a sample of the target population

Question 5

The deductive class consists of ….

Answer 5

• the construct method; starts from a explicit theory, and items are derived from this theory.
• the facet design method; starts from a conceptual analysis of the construct.

Question 6

What are the distinctive item response mode in a typical performance test?

Answer 6

Open ended versus closed ended
Closed ended is divided in: frequency scale and endorsement scale
Endorsement scale is divided in: all or none and intensity
Intensity endorsement scale is divided in: discrete and continuous

Question 7

When is kappa best to used, and when weighted kappa?

Answer 7

Use kappa when you have dichotomous items

Use weighted kappa when you have ordinal-polytomous items

Question 8

When do we use the coefficient of identity?

Answer 8

In case of a bounded continuous scale

Question 9

What are the two versions of the identity coefficient?

Answer 9

• coefficient of identity for interrater agreement

- coefficient of identity for intrarater agreement

Question 10

How is the identity coefficient for interrater agreement computed?

Answer 10

1. Compute Ra * Rb for every item (Ra = score rater A)
2. Add the outcomes of step 1
3. Multiply the outcome of step 2 by 2
4. Square all the individual ratings
5. Add all the squared individual ratings
6. Divide outcome 3 by outcome 5

Question 11

What is a response tendency?

Answer 11

The differential application of the response scale

Question 12

What is a response style?

Answer 12

The differential use of the item response scale by different persons ; this is a person-specific property

Question 13

What is acquiescence ?

Answer 13

The tendency to agree with an endorsement statement, independently of the content of the statement

Question 14

What is dissentience?

Answer 14

The opposite of acquiescence

Question 15

What is a response set?

Answer 15

The differential use of the item response scale by different persons and with different constructs -> a person/construct - specific property

Question 16

What is the aim of testing?

Answer 16

To yield scores of test takers maximum or typical performance

Question 17

What are the two types of test scores?

Answer 17

• the observed test score = derived from the item scores by taking the weighted or unweighted sum of the itemscores
• latent variable score = derived from the item responses under the assumption of a latent variable item response model

Question 18

What is measurement by fiat?

Answer 18

The item scores are assigned to a test takers responses without any theoretical justification

Question 19

Name two naive strategies for handling item non response

Answer 19

• delete test takers who have not answered one or more items or delete items that are not answered by many test takers
• to substitute the test takers personal mean -> the variance of the test score is affected in this way

Question 20

An item score consists of two parts, namely …

Answer 20

A fixed component and a random residual

Question 21

How can you calculate the fixed component of the test taker’s missing item score?

= TWjk

Answer 21

TWjk = PMj + IMk - OM

PMj = the personal mean item score
IMk = the mean of item k 
OM = the overall mean score

Question 22

What is TWEjk and how is it computed?

Answer 22

TWEjk is the value which is imputed in case of item non response. It is calculated as:
TWEjk = TWjk + Ejk
TWjk = PMj + IMk - OM
Ejk -> a value is randomly selected from a normal distribution

Question 23

How is the variance computed?

Answer 23

N - 1

1. Calculate the mean score
2. For each person: calculate item score - mean score
3. Square the outcomes of 2 and add them all together
4. Divide by n - 1

Question 24

How can you detect acquiescence and dissentience?

Answer 24

Compute the personal mean scores for the contra-indicative items and the indicative items
A high PMIj & low PMCj = acquiescence
A low PMIj & high PMCj = dissentience

Question 25

When you would like to choose a construct of interest, you can use three methods

Answer 25

• focus group method
• key informant method
• observation method