norm v criterion referenced tests Flashcards
advantages of NRTs
- Test score variability allows a differentiation to be made (between results) when scores of criterion-referenced tests are closely similar.
- Due to its roots in statistical theory, norm-referenced testing can be subject to complex analysis methods
- Popular with employers who seek the best applicants if they consider their requirements are not linked to specific aspects of school learning
disadvantages of NRTs
- Test results only show ranking compared to norm; results say nothing about what and how well it is learned.
- Because of its complex analysis methods it is more present as a tool of psychometric experts rather than educators
- Tests can get outdated since the same norm is used for aprox. 5 years before it is changed and ‘normalised’ – outdated tests can result in an apparent increase in scores, but due to familiarity not to better understanding.
- Teachers in schools under pressure to produce good results can end up teaching to the test (also known as, exam preparation). This in itself can be a good instruction as long as results are generalizable. (There is a difference between learning the answer to a set of questions and understanding the underlying concepts behind the questions). This can lead to test score pollution effects.
what are NRTs and their purpose
Norm-referenced assessment – compares the learner’s performance with that of the other individuals in the normative sample (the sample we are comparing it with). Norm tests use developmental scores (e.g. ‘Reading Ages’ or ‘Spelling Ages’) or derived scores (e.g. standard scores, or percentiles). Technically, cannot fail a norm-referenced test.
A test result which is expressed in relation to the distribution of results amongst a group is said to be norm referenced. An example of such a test would be the IQ result given by an intelligence test
Norm-referenced tests are designed and constructed to provide a basis for making comparisons among examinees in the content areas measured by the test. Test score norms are used as a standard for comparison. Knowing the ranking of a pupil within a class depends on the priorities employed in the business of learning. If the model is one of competition, then the information is useful. The result becomes of little use when one wants to know how capable the pupil is in the content area or how much they mastered from the material they have been taught. However, one might argue that as long as the result indicates the pupil is well above average in ability, the other aspects are not as important (local and transient details).
what are CRTs
Criterion-referenced test consists of previously determined criteria that should be achieved by a student independent of the performance of others. In other words, it tests the performance of the testee against a clearly defined domain of content or behaviours. It relies heavily on the mastery or non-mastery of content areas. On a criterion-referenced test, every student taking the test could possibly fail if they do not meet the expected standard and on the other end of the spectrum every student could earn the highest possible score and this is very desirable.
purposes and examples of CRTs
example - GCSEs
Criterion referenced tests are specifically constructed to provide a deeper interpretation of examinee performance in relation to the test content. It describes the level of performance, allows for classification of performance and evaluates the effectiveness of educational programmes.
advantages of CRTs
- They provide a vast wealth of knowledge into students’ strengths and weaknesses.
- They allow for independent assessment. Students pass or fail depending on their work not based on how well others in the population do as it does in norm-referenced assessments.
- The tests apply the same criteria to all students. Therefore, it may raise the standard expected of low income and EAL students by casting light on the specific difficulties and allows educators to make sure they achieve the specific expectations.
disadvantages of CRTs
- Aggregation- The detailed performance profile that is created for and individual that has been assessed using criterion-referenced tests is difficult to condense into a simple grade without losing the tests meaning. Most criterion-referenced tests have many domains or areas of content focus. If a single score is used it does not specify the capabilities in each domain. For example, a student has a maths test consisting of trigonometry, statistics and algebra questions. If he gets high marks in the trigonometry and algebra components but fails the statistics component an aggregate mark will just show he passed the maths test. Therefore, one may infer that he is capable doing all types of maths to a high degree meanwhile this is not the case. Also in the bid to correct this situation, some tests require an individual to attain a specific number in each domain for them to be classified as being part of a specific level. If a child fails to reach the target grade in one and passes all the others, he will still be classified as the lower level that ascribes to that one grade area. When someone else like a parent looks at the level as a whole they may believe that the child is attaining the lower level criteria overall.
- Problems arise when comparisons need to be made between the test takers. The scores are usually tailored to the individuals and therefore it is hard to compare using the scores given.
- The scores or inferences made from criterion-referenced tests may be skewed if the content specifications are not clearly defined or if they are poorly written.
how do educational professionals use CRTs
- Special educational needs co-ordinators and teachers use Criterion-referenced tests to monitor students’ progress and achievements.
- Teachers use the criterion-referenced tests to manage students’ daily progress. They also use it to evaluate teaching and instructional methods’ effectiveness.
- Criterion-referenced tests can be used to make decisions about grade promotion or diploma eligibility.
- Criterion referenced test are also used to test whether trainees have successfully completed training programmes.
- The test can help educational psychologists and policy makers to determine whether educational policies and educational techniques are working as intended.
what is CBA
Curriculum-based assessments compare an individual’s test scores against the curriculum content in order to alter the instructional methods if need be. Therefore it is domain specific
The purpose of this type of assessment is to focus more closely on the progress of individual students and the improvement of teaching methods
advantages of CBAs
- Curriculum based assessments provide a clear representation of an academic performance goal for the school year and an individual’s progress each time they are administered. This is usually done using curriculum based measurement graphs.
- The information gotten from the assessments and their graphs are easier to understand as they are already in a descriptive form unlike most standardized tests.
- The results are easily communicated between educational professionals.
- It is easy to pick up where another teacher left off if there is a change in staffing.
- Students or individuals can track their own progress and are more motivated to try and achieve their academic goals.
- Curriculum based learning automatically evaluates the success of the instruction students receive
disadvantages of CBAs
- Curriculum based assessments have problems with validity that arise from the difficulty of test materials. A good score could be due to easy test materials rather than the student actually improving.
- Due to test questions being drawn from the curriculum there can be a familiarity effect where students have recently done very similar questions and because of this score highly instead of it being based on their understanding of what is learnt.
- It is not easy to generalize scores from these types of assessments as curricula vary from institution to institution.
- Instead of measuring instructional value as is claims research has found that it is more about measuring how the curriculum material is used. This can be due to the increasing difficulty of the test materials. For progress and instructional value to be measured accurately the tests need to be comparable over time. This is difficult to maintain because as the curriculum gets harder due to mastery so does the assessment.
- Teachers may be teaching just to fulfil the curriculum and not to increase the knowledge of the subject as a whole. Therefore children are not learning to adapt this information to new or applied situations.
how do educational professionals use CBAs
Curriculum based assessments can be used as a screening test by educational professionals to identify students at risk of failing, to determine whether or not the child needs to receive special education.
Teachers use curriculum based assessments to find out how well a child is progressing in learning the content for the academic year.
Teachers also use based assessments to monitor the success of their instructional methods. So if a child is not meeting the learning outcomes expected the teacher then changes the way of teaching the child to try to find the best type of instruction the child needs to make sufficient progress toward meeting the academic goals. This could be done by increasing instructional time, changing a teaching technique or way of presenting the material, or changing a grouping arrangement. Due to the assessment being used frequently the teacher can see immediate effects of a change in instructional style and can keep changing it until there is progress.
Educational Psychologists can use the assessment to develop better goals and objectives for children with special educational needs.
how do educational professionals use NRTs
In the case of GCSE examinations, the overall scores are used to allocate candidates to grades and attention is paid to the proportions of the entry allocated.
When the purpose of testing is to ensure school accountability, norm-referenced tests represent an attractive feature.
Educational Psychologists and SENCos make use of norm-referenced tests results such as IQ scores or spelling ages (e.g. Vernon Spelling Test) to offer an explanation for difficulties, on the premise of ‘low potential’ (Freeman & Miller, 2001). However, norm-referenced tests are considered the least useful as an aid for understanding abilities and needs, as well as planning teaching responses to them (compared to Criterion-referenced assessment which is considered the most useful).
how is the norm established
The norm is established by administering the test to a large sample of the population, and the distribution of the results will follow the normal bell-shaped curve. There are two concerns when establishing the norm.
Firstly, the sample needs to be large enough, and it has to be representative for the population we are referring to. Being in the top 10 percent of your class does not mean that you will also be in the top 10 percent for the whole country, unless a far wider sample is used to ‘normalise’ the test.
The second concern is regarding to item selection. The purpose of a norm-referenced test is to rank, discriminate and spread out the population. Although items where everyone succeeds may provide useful information for other purposes, these items will be removed during test development
