Lecture 5: Deveopment of a study plan Flashcards
One-stage designs
decisions are made after all measures have been administered and thus, all information is available and considered
Single screen
a singular test result justifies all ensuing decisions
(generally not advisable)
Non-sequential test battery
Every applicant takes the entire test battery
- Decision is made after all info has been integrated according to the a priori formulated decision rules
What are three strategies of a multi-stage desing?
- Pre-reject strategy
- Pre-select strategy
- Completely sequential strategy
Pre-reject strategy
A selection of measures is used, but not everyone takes all measures.
1) 1. Test - applicants are dropped who fail to reach a certain score (the rest of applicants take further test)
2) After 2. testing - final rejection or positive diagnosis
Pre-select strategy
- applicants are already terminally accepted after the 1st test
- everybody else is subjected to futher testing
Completely sequential strategy
combination of pre-rejection and pre-selection strategy
1) After 1. test applicants are catgeorized in three groups (pre-selected, pre-rejected and further tests)
2) For poeple taking further test: Further accepted (positive diagnosis) or further rejected.
What should be considered in one-stage and multi-stage study plans?
- Selection rate
- Base rate
- Criterion validity
- Interactions of these variables.
Selection rate
- the rate of applicants who are hired.
(b+d)/(a+b+c+d)
Hit rate
From the selected people, which are suited
b/(b+d)
If a high criterion validity is given …
… the scatter plot is more narrow -> the error areas for a and d get smaller (good for the hit rate)
What happens if you shift the selection rate? (case for higher and lower cutoff)
- It makes only one of the errors smaller
- Higher cutoff:Less suitable people get accepted, more unsuitable poeple get rejected.
- Lower cutoff: More suitable people get accepted, less unsitable get rejected.
Base rate
the proportion of suitable candidates
(a+b)/(a+b+c+d)
- hits that can be expected upon random selection.
Interaction between base rate and hit rate
high base rate (and good criterion validity) are helpful for achieving a high hit rate (suitable people are selected)
Taylor-Russel Tables - what are they good for?
For estimating hit rates
What can we take from the taylor-russel table?
- the higher the base rate, the higher the hit rate
- the less poeple selected, the better the hit rate
- VALIDITY is also very influential.
General advise for planning the assessment
- Order of tests
- Formalia
- Preparation for test instructors
Order of tests (What is important)
- exhaustion of testees
- practice and transfer effects
Formalia (What is important when testing?)
- ensure all organisational aspects are in order
- create friendly and open atmosphere
- give feedback during testing?
Preparation for test instructors (what is important)
- often more than one person involved
- intensive preparation vital
- check for prejudice against testees
- control and regulate sympathy
- ensure to be open for all results at all times.
Why do we often see in practice one-stage designs?
- Due to time
- Due to financial constraints
What are tests of the one-stage design?
- single screen
- non-sequential test battery
What are examples for a singular screen (1) and a non-sequential test battery (2) ?
- Interview
- An Assessment center.
What would be a pre-rejection strategy - applied to the real world?
- You send in your application documents (first poeple are dropped based on this)
- Further selection or rejection based on a following interview.
What are advances of a multi-stage design (1) vs. of a one-stage desing (2)
- different measures are used
- Less time, less money :)
What does criterion validity have to do … with selection, base rate etc. ?
- when criterion validity is given, the plot becomes more narrow
- then the a and d error becomes smaller
- the hit rate becomes larger.
What is one of the onliest reasons how both of the errors can become smaller?
-increasing criterion validity
- hence, errors become smaller and the hit rate become larger.
What can reduce one of the errors?
- Shifting the selection rate
What does the multi-stage plan tell us about selection rate (1) and base rate (2)
(in regard to validity)
1) Use a lower cutoff it test (scores) has a low reported validity - hence, less poeple selected.
2) If test (scores) valid, increase base rate.
What is desirable for achieving a high hit rate?
- high base rate
- good criterion valdity
