Development of maximum performance tests

Question 1

What seven steps are involved in test developments

Answer 1

1. The construct of interest
2. The measurement mode
3. The objectives of the test
4. The population
5. The conceptual framework- The theoretical framework on which the test was based.
6. The item response mode
7. The administration mode

Question 2

In what three ways can latent variables vary?

Answer 2

Scope
Content
Between educational and psychological variables

Question 3

What three main objectives can tests have?

Answer 3

Description, diagnosis and decision making

Question 4

What is the conceptual framework used for?

Answer 4

Conceptual distinctions and organise ideas

Question 5

Name as many item writing guidelines ass you can think of (19 mentioned)

Answer 5

1. Focus on one relevant aspect
2. Use independent item content (except reading passage)
3. Avoid overly specific and overly general content (leads to ambiguity)
4. Avoid items that deliberately deceive
5. keep vocal simple for population
6. Put items vertically
7. Minimize reading time and avoid unnecessary information
8. Use correct language
9. Use non-sensitive language
10. Use a clear stem and include central idea in the stem
11. Word items positively
12. Write three options unless there are easy to write and plausible distractors
13. One option must be unambiguously the right answer
14. Place the item in alphabetical, logical or numerical order
15. Vary the location of the correct options across the test
16. Keep the options homogenous in length, content and grammar
17. Avoid all of the above as last option
18. Make distractors plausible
19. Avoid giving clues to the right answer

Question 6

What responses have to be rated by raters?

Answer 6

To free- or constructed response items

Question 7

What item-rating guidelines are there? (9)

Answer 7

1. Rate responses anonymously
2. Rate the responses to one item at a Time ( all the responses to one item should be rated before moving to the next item)
3. Provide the rater with a frame of reference- raters should be given instructions, schemes or ideal responses
4. Separate irrelevant aspects from the relative performance
5. Use more than one rater
6. Re-rate the free responses
7. Rate all the responses to an item on the same occasion
8. Rearrange the order of the responses
9. Read a sample of the responses - a sample of the responses should be read before the rating

Question 8

What is meant by pilot studies?

Answer 8

Studies to test the quality of the concept items

Question 9

What is the coefficient used for?

Answer 9

The consistency between different occasions for a rater

Question 10

What formula does the coefficient kappa use?

Answer 10

𝐾𝑎𝑝𝑝𝑎= (𝑂−𝐸/1−𝐸) (𝐸<1)

E refers to the expected proportion of identical ratings and O refers to the observed proportion of identical ratings. E can be calculated by multiplying the marginal proportions.

Question 11

How is O calculated?

Answer 11

Adding the diagonal cells

Question 12

In what way are the pilot raters studied?

Answer 12

In consistency and the agreement between raters

Question 13

What can pilot studies reveal?

Answer 13

1. Many raters are inconsistent with their ratings of many items (procedure is inadequate)
2. Some of the raters are inconsistent (do not understand instructions- remove raters)
3. Some of items are inconsistent- remove items
4. Disagree on many items- procedure inadequate

Question 14

What kind of questions should be placed at the start of the test and why?

Answer 14

Easier items because people are often nervous and should build confidence