Lecture 8- Assessment

Question 1

when do we so assessments

Answer 1

When we attempt to select someone for a job, we are trying to predict if that person will be able to perform the requisite tasks to the requisite standard. This will also be the case if we are considering a person for promotion to a more responsible job.

Question 2

why is it important to assess people

Answer 2

Assessing people correctly is important, for example, in determining whether the person will behave safely in a safety-critical job – but also to prevent stress at work by helping to ensure that people are suited to the jobs they do.

Question 3

what should we do before assessing someone

Answer 3

before we can decide upon an assessment method, we must establish our requirements using job and competency analyses techniques. When we know what the person will be expected to do, and what type of person (in terms of behaviours) we are looking for, we can then assess candidates.

Question 4

what must the assessment methods be

Answer 4

The assessment method(s) chosen must be valid and reliable, i.e. it should measure what it is supposed to measure - and in a consistent manner.
To ensure this, it is important that potential assessment methods are validated against requirements – usually by criterion-related validity studies (CRV) - after performance criteria have been established.

Question 5

how do we ensure that assemment methods are valid and reliable

Answer 5

To ensure this, it is important that potential assessment methods are validated against requirements – usually by criterion-related validity studies (CRV) - after performance criteria have been established.

Question 6

why are ob Analysis and Competency Analysis completed

Answer 6

To determine the type of person we are looking for

Question 7

what is job analysis

Answer 7

Job-oriented analysis looking for the nature/content of the job itself.

Question 8

name some methods of job analysis to deter

Answer 8

Observation:- by an analyst. Could be watching and noting, or even participating. Possibility of Hawthorne Effect. Also looking at any available documentation. Problems with mental work.

b) Interviews:- with job holders, supervisors or peers. Such interviews can be rather specific, such as the Critical Incidents technique of Flanagan (1954), which asks for examples of job behaviour that characterise good or bad performance. Brief job analysis by asking for listing and rating of tasks by subject matter experts (SME).
c) Diaries/Logbooks:- kept by the job incumbent. Headings.
d) Structured Questionnaires:

Question 9

what type of structured questionnairres can be used for job analysis

Answer 9

Functional Job Analysis (Fine and Wiley, 1974) uses analyses of constituent tasks in terms of action sequences and comes up with the percentage of the job directed towards ‘data’, ‘people’ and other aspects. Aids observation (direct, video, etc.).
• Position Analysis Questionnaire (PAQ) (McCormick et al, 1972) has 187 job elements in 6 categories:- information input - where and how is the information gained that is needed to perform the job? mediation processes - what reasoning, decision-making, planning and information-processing activities are needed? work output - physical activities, tools and devices
relationships with other persons, job context - the physical and social context, other factors, e.g. job structure, hours of work, etc.
• Work Profiling System (WPS) and Job Components Inventory (JCI), etc.

Question 10

describe a functional job analysis

Answer 10

(Fine and Wiley, 1974) uses analyses of constituent tasks in terms of action sequences and comes up with the percentage of the job directed towards ‘data’, ‘people’ and other aspects. Aids observation (direct, video, etc.).

Question 11

who created the functional job analysis

Answer 11

(Fine and Wiley, 1974)

Question 12

who created the position analysis questionnairre

Answer 12

McCormick et al, 1972)

Question 13

describe the position analysis questionnairre

Answer 13

(PAQ) (McCormick et al, 1972) has 187 job elements in 6 categories:- information input - where and how is the information gained that is needed to perform the job? mediation processes - what reasoning, decision-making, planning and information-processing activities are needed? work output - physical activities, tools and devices
relationships with other persons, job context - the physical and social context, other factors, e.g. job structure, hours of work, etc.

Question 14

what is a competancy analysis

Answer 14

Person-oriented analysis looking for competencies

Question 15

what does a competancy analysis involve

Answer 15

the behavioural indicators (including knowledge, skills and attitudes) needed to perform job tasks with competence………’
e.g. for the ‘Ideal’ Air Traffic Controller (Pearn & Kandola, 1983)
Foresight, judgement, application of professional/technical knowledge, and reliability under pressure, for ‘outstanding’ controllers.
Should be able to absorb information simultaneously from multiple sources
Should be able to project forward and to constantly adjust the whole picture.
….within a context of time pressure, distractions and noise.
Also……convergent, concrete thinking, with self-control, teamwork, decisiveness and conscientiousness.
Key seems to be emotional stability with its emotional detachment, self-assurance and the absence of tension, but latter two must not relate to complacency.

Question 16

what type of skills/ competancies should graduates have

Answer 16

Analytical Thinking/Judgement/Decision Making
Drive and Decisiveness
Teamworking
Planning
Communicating
Creative Thinking
Motivating Others
Change Tolerance

Question 17

what is a person specification and when shoudl it occur

Answer 17

After analysis, we need to specify the type of person required for successful job performance.
This is straightforward when the analysis is at a sufficient level of detail to define competencies (CA) as ATC examples above.
Similarly, when required skills and abilities are defined, such as manual dexterity or visual acuity, the person specification is clear.

Question 18

when does person specification become problematic

Answer 18

person specification is more difficult if such detail is not available from the analysis. Inferences are required when factors such as those relating to general intelligence and personality characteristics are identified, e.g. ‘should be motivated’, ‘should get on with other people’, etc.
The better the level of detail acquired in both job and especially competency analyses, the more we can be sure that our choice of assessment method will be appropriate.

Question 19

what improves person speciification

Answer 19

The better the level of detail acquired in both job and especially competency analyses, the more we can be sure that our choice of assessment method will be appropriate.

Question 20

why do we establish performance criteria

Answer 20

It is not sufficient simply to determine what the job entails. In order to carry out ‘criterion-related validity’ studies on our assessment methods, it is necessary to establish levels of acceptable performance for the tasks involved.
This may come from supervisor ratings of performance, simple output measures, and so on, after recruitment has taken place.

Question 21

give some examples of assessment methods

Answer 21

Interviews, Psychometric Tests, and Work Sample Tests.

Question 22

what is validity

Answer 22

validity in general refers to ‘the extent to which a test or other measuring technique measures what it sets out to measure’

Question 23

what is criterion rated validity

Answer 23

Refers to the strength of the relationship between a predictor (assessment method) and some criterion (or criteria) of work performance. Predictive or Concurrent.

If a significant correlation coefficient (validity coefficient) was found, we could reasonably use that assessment method for future applicants as a basis for acceptance or rejection.
Latter via deciding on ‘cut-off’ scores on the predictor.

Question 24

what should be done if there seems to be validity

Answer 24

If there is a large difference in rejection rates between different groups, need to look more closely at the relationship between test scores and job performance. If low predictor scores are not reflected in low job performance, this shows that, for that particular group, the test has no validity.

Question 25

what is validity generalisation

Answer 25

This is the ‘transportability’ of validity coefficients to similar groups, e.g. similar jobs, elsewhere, thus precluding the need for further CRV (predictive or concurrent validity) exercises. This is done where samples are not large enough to carry out a validity study. Its worth, though, is a subject of much current debate, e.g. what about situational variables such as stress which differ from one work environment to another.

Question 26

what are the 2 types of validity which can measure assessment methods

Answer 26

content validity

face validity

Question 27

what is face validity

Answer 27

Face validity refers to the subjects’ perceived relevance of the measure,

Question 28

what is content validity

Answer 28

the representativeness of a measure, e.g. topics covered in test in relation to the job

Question 29

what is reliability

Answer 29

the consistency with which a measure provides results, i.e. its stability and freedom from random variation

Question 30

what is interrater reliabillity

Answer 30

the degree of agreement between 2 or more raters

Question 31

what is intra-rater reliability

Answer 31

how far a single rater is consistent across time.

Question 32

what is a psychometric test

Answer 32

a standardised sample of behaviour which can be described by a numerical scale or category system’ - Cronbach (1984).

objective rather than subjective via standardisation (of test conditions, instructions, time, content, scoring, and interpretation)

Question 33

what needs to be included in psychometric tests

Answer 33

Administer, score -
and (more controversially)
interpret tests (computer based test interpretation (CBTI). Expert systems with decision rules and interpretative strategies. Report produced from linking of preset phrases and comments.

Question 34

for ethical reasons, what do you need to carry out psychometric tests

Answer 34

BPS Level A (Cognitive) and Level B Personality)

Question 35

name some examples of psychometric tests

Answer 35

Wechsler Adult Intelligence Scales (WAIS)

• Myers-Briggs Type Indicator (MBTI)
• Occupational Personality Questionnaire (OPQ)

NEO PI-R five factor model (FFM) (Costa and McCrae, 1990)- OCEAN

Question 36

Are psychometric tests and inventories reliable

Answer 36

All these tests have been subjected to CRV studies for a range of occupations. Results vary for particular traits/tests/ occupations, but this is only to be expected. A test can only be valid if its use is based on a thorough job analysis.
Once reliable, always reliable.

Question 37

are psyhcometric tests and inventories valid?

Answer 37

Validity depends on appropriate use, e.g. little validity if a communication skills test were used to predict perceptual-motor performance – even if the test were valid when used appropriately (i.e. to assess communication skills !)

Question 38

name some problematic issues with psychometric tests and inventories

Answer 38

i) culture fairness of items;
ii) definition of intelligence and personality;
iii) social desirability/faking/impression management/self deception.

Question 39

what is the aim of an interviewer

Answer 39

im is to gather data, evaluate it, and select or reject - an ‘intuitive regression equation’ who must combine and weigh data to come to a decision. How well this is done determines the validity of the interview. Importance of level of training and experience, and biases. Latter may be due to a multitude of personal factors such as age, sex, race, intelligence and personality factors. Also preferences, stereotypes held and ‘set’.

Question 40

what should interviewees be influenced by

Answer 40

behaviour, again, influenced by whole host of personal factors.

Question 41

what is interpersonal communication

Answer 41

‘a conversation with a purpose’. Behaviour of interviewer can affect the responses of the interviewee, e.g. agreeing, paraphrasing, silence or disagreement. Behaviour of both is influenced by the self-perception of each party, this may be affected by the other’s presence, and so on. Not possible, though, to have a ‘standard’ interviewer

Question 42

what questions arise when deciding to do group or individual interviews

Answer 42

together or separately. Is validity improved if a team decision is made rather than two separate individual decisions?

Question 43

Are interviews valid

Answer 43

Very little predictive power for interviews (Mayfield, 1964; Arvey and Campion, 1982)

but thousands of studies showing a whole range of validities.

Question 44

are interviews reliable

Answer 44

Studies have shown poor inter-rater reliability, i.e. the degree of agreement between two or more interviewers of the same interviewee.

Even poor intra-rater reliability. Usual range of biases in social perception when ‘traditional interview’ used.

Question 45

how can interviews be improved

Answer 45

Training of interviewers

situational interviews

interview structure eg 5 or 7 point plan

Question 46

what factors are on the 5 point plan for improving interview structure

Answer 46

Impact on Others

2) Qualifications
3) Experience
4) Motivation
5) Adjustment.

Question 47

what factors are on the 7 point plan for improving interview structure

Answer 47

1) Physical makeup
2) Attainments
3) Intelligence
4) Aptitudes
5) Interests
6) Disposition
7) Circumstances.

Question 48

what are situational interviews and how can they improve interviews

Answer 48

Use job analyses to provide job-related incidents for job-related questions. ‘What would you do if….?’
For example, the type of questions asked to graduates based on the requirements/ competencies discussed in our seminar

Question 49

how can training of interviewers improve interviews

Answer 49

To use job-related questions in a structured manner. Overcoming possible biases in social perception.

Question 50

what is a work sample test

Answer 50

Any assessment which involves the assessee being asked to perform a behaviour/task/activity which is the same as/similar to/representative of those performed in the real work role.

Question 51

name some types of work sample tests

Answer 51

• Psychomotor
• Job knowledge tests
• Individual/situational decision making, e.g. in-tray
• Group discussions/decision making - for assessing managerial potential/teamwork/influencing others/clarity of thinking under stress, etc. Indoor/outdoor exercises/leaderless/with leader, etc.
Observed and Rated.

Question 52

what is the 2 ways of completing work sample tests

Answer 52

observed and rated

Question 53

are work sample tests reliable or valid

Answer 53

If designed appropriately, based on job and person analyses, reliability and validity good.

Question 54

what is the aim of assessment methods

Answer 54

Seek to account for ‘all the variance’ by using multiple methods of assessment with multiple assessors over several days with groups of candidates.

Question 55

what is synthetic validity (Assessment centres)

Answer 55

‘…the inferring of validity in a specific situation from a logical analysis of jobs into their elements, a determination of test validity for these elements, and a combination of elemental validities into a whole’

Question 56

are assessment centres reliable or vallid

Answer 56

Reliability and validity dependent on those of assessment methods used and ‘combination’ thereafter.