Testing - AMT - Chapter 9 Flashcards
According to the authors, the best way to test people’s writing ability is….
to get them to write (test writing ability directly)
Writing task problems
- Representative Tasks (population + expected performance)
- Valid samples of writing (represent ability)
- Scored Validly (Can/Will be)
How to Create Representative Writing Task
- Specify all possible content
- Include Representative Sample of Specified Content
Specify all possible content of a Writing Task
Operations (Describe, explain, compare, etc.)
Types of text
Addressees
Length of texts
Topics
Dialect and style
Text Operations
(e.g.)
Describe, explain, compare and contrast, argue for and against a position, etc.
ideal writing test for Construct Validity
One which requires candidates to perform all the relevant potential writing tasks.
Why writing task with
- Wide ranging
- Representative sample of specifications
++Validity
++Backwash Effect
Really good coverage of the range of potential writing tasks is…
Not possible in a single version of the test.
To know something ACCURATE and MEANINGFUL about a person’s WRITING ability is ALWAYS
- Expensive
Elicit a valid sample of writing ability
- Separate tasks (as many as is feasible)
- Test ONLY writing ability
- Restrict candidates
Ensure valid and reliable scoring
- Reliably scored tasks
- As many as possible tasks
- Restrict candidates (directly comparable)
- Long enough Samples
Scales of Scoring (approaches)
Holistic & Analytic
Scales of Scoring
(where in specifications)
Criterial Levels of Performance
”"”impressionistic”” refers to”
Holistic scoring
Holistic Scoring
- Single Score
- Overall Impression
Holistic Scoring
- Advantages
Very Rapid (2min)
Holistic Scoring
- disadvantages
- Not reliable
- if scored 4 times = reliability improved
Analytic Scoring
Seperate Score for each Aspects of a Task
Analytic Scoring
- Advantages
- Subskills development in individuals considered
- All Aspects of performance considered by Scorer
- Number of Scores = ++Reliable
- Weight of grade adapted
Analytic Scoring
- Disadvantages
- Time consumming to score
- Attention to different aspects distract from overall effect
- Frequency of errors vs. Communication
If diagnostic information is required directly from the ratings given
(Analytic or Holistic??)
Analytic scoring is essential
Calibrate the scale to be used
1 - Collecting samples of performance
2 - Expert assing sample to point(s) on relevant scale
3 - Result = reference points
Training Stages
1 - Background and Overview (study handbook)
2 - Rate + Discuss (handbook answered)
3 - Agreement on how to rate
