Lecture 3 Test Development (Catherine) Flashcards
To Provide the main content covered in Lecture 3 on Test Development
What are the 5 stages of Test Development?
- Test Conceptualisation
- Test Construction
- Test Try Out
- Item Analysis
- Test Revision
What is the aim of Test Conceptualisation?
To establish the reasons for designing or revising a psychological test
What is the aim of Test Construction?
To ascertain how numbers should be assigned to psychological attributes and which scales of measurement should be used
What is the purpose of the Test Try Out phase?
To consider to whom the test should be administered and to generate data about its reliability and validity
What is the purpose of the Item Analysis phase?
- To identify the properties of the test’s items & scales.
- to ascertain which types of analyses need to be performed to understand item validity, reliability, difficulty and discrimination.
- in short, to identify which are ‘good’ items and which are ‘bad’ items
What is the purpose of Test Revision?
Determining the items of the test that may need to be revised or discarded.
*establishing whether the test is measuring what it was designed to measure
What are the key stages involved in test conceptualisation?
- Conceiving the need for a test
- Considering the assumptions for test design
- Running a Pilot Test
- Conceptualising a Measure
What are the key considerations when Conceiving the need for a test?
- There is always a question in the mind of the test developer e.g.
- There ought to be an instrument designed to measure [some psychological construct] in [such & such] a way
- even if a measure does exist, it may have poor psychometric properties (e.g. low reliability, poor construct validity and suspect content validity)
It what way do the key questions a test developer poses when designing a test correspond with assumption 1 of test design?
Assumption 1 Psychological states & traits exist:
- What is the test designed to measure?
- Is there a need for this test?
- What content will this test cover?
It what way do the key questions a test developer poses when designing a test correspond with assumption 2 of test design?
Assumption 2 Psychological states & traits can be Quantified & Measured
- What is the ideal format for this test?
- What type of responses will be required of test takers?
It what way do the key questions a test developer poses when designing a test correspond with assumption 3 of test design?
Assumption 3 Test related behaviour predicts non-test related behaviour
- Should more than one form of the test be developed?
- How will meaning be attributed to scores on this test?
It what way do the key questions a test developer poses when designing a test correspond with assumption 4 of test design?
Assumption 4 Psychological tests have strengths & weaknesses
- Who benefits from administration of this test?
- Is there any potential for harm as a result of the administration of the test?
What are the key considerations in the stage of pilot work?
- Pilot work refers to the preliminary work surrounding the creation of a prototype of a test
- During this phase a test developer attempts to determine how best to measure a targeted construct using a number of strategies: e.g. literature review, experimentation
- Test items may be pilot studies to evaluate whether they should be included in the final form of the instrument. (creation, revision, deletion of test items)
What are the 4 key considerations when conceptualising a measure to be used in a test?
- What is the test designed to measure?
- Is there a need for this test?
- What content will the test cover?
- What is the ideal format of the test?
What are the key elements required for test construction?
*Identifying appropriate measurement scales (N.O.I.R.)
*Developing a scale appropriate for the test: e.g.
-a summative scale,
-binary rating scale
-faces scale
-Method of paired comparisons
-comparative scaling
Categorical scaling
Describe the key aspects of a summative scale and give examples
- a summative scale is a scale on which a test-taker’s answers to test/scale answers may be summed to create an aggregate score
- This aggregate score is thought to indicate the strength of the test takers traits attitude or other characteristic
- A Likert scale is a popular summative scale
- Test developers often treat summative scales as interval even though they are actually ordinal
Describe the key aspects of a Likert (summative) scale
A Likert Scale typically provides the test taker with 5 or 7 possible responses along an ‘agree’ to ‘disagree’ continuum
*Easy to construct and are used extensively in psychology because it yields ordinal level data, which approximates interval-level data well enough for data analytic purposes
What are the key points of the Method of paired comparisons
*Test-taker has to choose one of two options (e.g. a statement, object, picture) on the basis of some rule
*The value (e.g. 1 or 0) of each option in each paired comparison is determined by judges prior to test administration
*test takers must select the behaviour they feel is most justified
(NB Ordinal Data)
What are the key points of Comparative Scaling?
*a test taker must sort or rank stimuli according to some rule e.g. best to worst, most justifiable to least justifiable
(NB Ordinal Data)
What are the key points of Categorical Scaling?
Test takers must categorise stimuli according to some rule
e.g. sort the behaviours into good sleep practice and bad sleep practice
(NB can give nominal or ordinal data)
What are the key points to consider when Writing Test Items?
- The first step in writing items is to create an item pool or item bank
- as a general rule it is advisable to have at least twice as many items as you will include in your final draft test/measure
- the process of test try out, item analysis, and test revision, will knock many of these items out as the test developer strives to maximise reliability & validity
What types of item format are there that can be used in writing test items?
- Constructed response format
* completion answer
* short answer
* essay - Selected response format
* matching choice
* true-false
* multiple-choice
What are the key points from Completion items from the constructed response format of test items?
Completion items:
- Test-taker provides a word or phrase that completes a sentence
- There must be a specific correct answer
- Poorly designed completion items can lead to scoring problems
What are the key points from Short Answer & Essay items from the constructed response format of test items?
- Short Answer items:
- a word, a sentence, & paragraph all qualify as a short answer
- If more than 2 paragraphs, then the item is more likely to be an essay item
- Essay items:
- Useful when the examiner wants the test-taker to demonstrate in-depth knowledge
- Allows for creative integration & expression of what has been learned
- Only a limited range of material can be examined this way
