Lecture 3: Psychological Assessments

Question 1

True or false: a predictor needs to be both reliable and valid for it to be useful

Answer 1

True

Question 2

Define Reliability

Answer 2

Reliability refers to the consistency, stability, and equivalency of test scores/measure of predictability. The measure should yield the same estimate on repeated uses if the measured trait has not changed, even if inaccurate.

Question 3

NOT NEEDED SKIP What are the types of reliability often used in IO Psychology?

Answer 3

1. Test-retest reliability
2. Equivalence-form reliability
3. Internal-Consistency reliability:
   
   • split-half reliability
   • Computing two coefficients: Cronbach’s
     alpha coefficient/KR 20 (Kuder-Richardson
     20)
4. Inter-rater reliability

Question 4

SKIP Explain the 4 types of reliability used in I-O Psychology

Answer 4

1. Test-retest reliability: this refers to when a test is given at a certain time, and then after a certain period of time, it is re-given and the test scores’ similarity are measured using the coefficient of stability. the higher the coefficient of stability the better. An acceptable amount is 0.70 but generally 0.80 and above are better. Also, the longer the time interval between the administrations of the test, the higher the coefficient of stability, and thus, the more reliable the test is. Considered the simplest reliability test.
2. Parallel/Equivalence-form reliability: this is when two forms of a tests are given to the same group of people to measure the same attribute. The correlation resulted is called the coefficient of equivalence and it reflects the extent to which the two forms are ‘sufficiently comparable measures of the same concept/attribute.’ Least used because it is difficult to create one good test let alone two. And many tests do not have a “parallel form.” In intelligence and achievement testing, equivalent forms of the same test are sometimes available. If the resulting coefficient of equivalence is high, the tests are sufficiently comparable and are viewed as reliable measures. Similar in terms of 0.70 being good, but 0.80 and above being better.
3. Internal-Consistency Reliability: measures the extent to which something has homogenous content. Two types typically computed:

a) split-half reliability: the tests are just divided into two and given to two different groips of people. Their scores are then compared. But people argue that this approach is not fair as people tend to face fatigue and then do worse on the second half. this is why researchers often divide it using odd/even schemes. If the test has internal-consistency reliability, then there will be a high degree of similarity between the responses to the items from the two halves. The longer a test, the greater its reliability.

b) Compute one of two coefficients: Cronbach alpha test or Kuder-Richardon 20 (KR20). KR20 used for tests with dichotomous answers. each test is a mini test in essence. he response to each item is correlated with the response to every other item. The average of those inter-item correlations is related to the homogeneity of the test. homogenous test will have higher internal consistency in comparison to a heterogeneous test. s homogeneity of content in I-O psychology with an acceptable coefficient lying in the .70-.80 range, with higher coefficients being better (more reliable).

4. Inter-Rater reliability/conspect reliability: this is when two different raters are used to observe the same behavior. the coefficient reflects the degree of agreement and high inter rater reliability establishes a base that is reliable and accurate observation.

Question 5

Define validity

Answer 5

whether a test is correctly measuring what we intend to measure. The accuracy or appropriateness for predicting/drawing inferences from test scores. Validity depends on the use of a test as opposed to reliability which focuses on the measuring device. Scores from a given test may be highly valid for predicting employee productivity but completely invalid for predicting employee absenteeism.

rather than having types of reliability, we have sources of evidence that are used to support interpretations or inferences abput whatever it is we are measuring.

Question 6

What is construct vs operationalization.

validity

Answer 6

construct refers to the theoretical concepts we use to explain behavior (intelligence, motivation, work ethic etc.)

operationalization refers to the ways of measuring a construct. (eg: test is a way to measure the construct of intelligence)

Question 7

Reiterate what construct vs operationalization is and accordingly explain construct validity

Answer 7

Construct validity refers to the degree to which an actual measure is an accurate and faithful representation of its underlying construct.

To establish the construct validity of a test, you want to compare the scores on your test with other known measures of the construct. if faithful then scores should converge with other measures. If high correlation between this measure and existing measures: convergent validity coefficient is higher (reflect the degree to which scores converge when measuring a construct.

there should be very low correlation between scores from new test of a construct and other non-related constructs. the correlation for such are referred to as the divergeny validity coefficients reflect the degree to which these scores diverge. Also known as discriminant validity coefficients because the concepts are distinguishable.

Question 8

SKIP Inferential linkages in construct validation

Answer 8

the process of construct validation involves examining the linkages among multiple concepts of interest to us.

We would want to know that the empirical measures of X and Y are faithful and accurate assessments of the constructs (1 and 2) they purport to measure. Because our empirical measures are never perfect indicators of the constructs we seek to understand, it has been suggested that researchers should devote more attention to assessing linkages.

pg 124 TB

Question 9

SKIP Criterion-related validity

what are the variations in criterion-related validity

Answer 9

the degree to which a test.measure forecasts or is statistically related to the criterion its supposed to measure.

two major variations of criterion-related validity are:
**concurrent ** used to diagnose the existing status of some criterion
predictive used to predict the future status of the criterion.

When predictor scores are correlated with criterion data, the resulting correlation is called a validity coefficient. Whereas an acceptable reliabil-ity coefficient is in the .70–.80 range, the desired range for a validity coefficient is .30–.40. Validity coefficients less than .30 are not uncommon, but those greater than .50 are rare. Just as a predictor cannot be too reliable, it also cannot be too valid. The greater the correlation between the predictor and the criterion, the more we know about the criterion based on the predictor.

Question 10

SKIP explain validity coefficient

Answer 10

When predictor scores are correlated with criterion data, the resulting correlation is called a validity coefficient. Whereas an acceptable reliabil-ity coefficient is in the .70–.80 range, the desired range for a validity coefficient is .30–.40. Validity coefficients less than .30 are not uncommon, but those greater than .50 are rare. Just as a predictor cannot be too reliable, it also cannot be too valid. The greater the correlation between the predictor and the criterion, the more we know about the criterion based on the predictor.

By squaring the correlation coefficient (r), we can calculate how much variance
in the criterion we can account for by using the predictor. For example, if a predictor correlates .40 with a criterion, we can explain 16% (r2) of the variance in the criterion by knowing the predictor. This particular level of predictability (16%) would be con-sidered satisfactory by most psychologists, given all the possible causes of performance variation. A correlation of 1.0 indicates perfect prediction (and complete knowl-edge). However, tests with moderate validity coefficients are not necessarily flawed or inadequate. The results attest to the complexity of human behavior. Our behavior is influenced by factors not measured by tests, such as motivation and luck. We should thus have realistic expectations regarding the validity of our tests.

Question 11

SKIP Content Validity

Answer 11

The degree to which a predictor covers a representative sample of the behavior being assessed.

to assess content validity we use subject matter experts in the field of test covers. Civil War historians would first define the domain of the Civil War and then write test questions about it. These experts would then decide how content valid the test is. Their judgments could range from “not at all” to “highly valid.” Presumably, the test would be revised until it showed a high degree of content validity

Question 12

SKIP Face Validity

Answer 12

This is concerned with the appearance of the test items: do they look appropriate for such a test? their face value

in contrast to content validity, estimates of its validity are made by test developers whereas face validity is estimated by test-takers*

It is possible for a test to be content valid but not face valid, and vice versa; can be used for employee testing

Question 13

SKIP **Predictor development:
**1. construct vs behavioral sampling
2. past vs present characteristics

Answer 13

Predictor development: Predictor development refers to the creation of tools or assessments that help psychologists understand certain characteristics or qualities of an individual. these predictors can be classified along two dimensions

1. construct vs behavioral sampling: this is to see whether a predictor is seeking to measure the underlying psychological construct or a sample of the behavior that is going to be exhibited on a job.
   2.** past vs present characteristics**: whether a predictor is seeking to measure something about the individual right now or in the past.

Question 14

Ability Tests:

OK DO THIS **cognitive
**physical
psychomotor
sensual/perceptual

Cognitive ability/general mental ability (g)

pros and cons

explanation

Answer 14

many researchers believe cognitive ability is the single best predictor of future job performance.

CONS OF COGNITIVE ABILITY:
* Some researchers believe that conceptualizing intelligence merely as g encourages oversimplification of the inherent complexity of intelligence. The idea is that intelligence is not a unitary phenomenon, and other dimensions of intelligence are also worthy of our consideration.
* biases that exist for certain demo-graphic groups: A great deal of research has revealed that members of some minority groups have lower average scores on general mental ability tests compared to majority group members. thus can result in adverse impact

how to address the issue of adverse impact?
* supplement the use of cognitive ability rests with other non-cognitive predictors (such as personality inventories) which helps increase diversity and reduce adverse impact.
* you could also adjust scores based on group membership (e.g., giving bonus points to members of certain groups or setting different cutoff scores); however, this is not recommendable from a legal perspective. However, IO psychologists tend to disagree. Namely, within-group norming, a practice in which individual scores are converted to standard scores or percentile scores within one’s group, has been advo-cated as the most scientific solution to the issue of adverse impact for cognitive ability tests, yet it has been deemed unlawful for non-scientific reasons.

PROS:
* but even though it may be oversimplifying the complexity of intelligence, data has found that g accounts for approx. 80% of the variablity in criterion-related validity estimates while the remaining 20% of the variability is accounted for by other mental abilities.

HIGH VALIDTY
HIGH GENERALIZABILITY
LOW COST

Question 15

Ability tests

SKIP cognitive ability
**physical ability
**psychomotor
sensu

Physical Ability

pros and cons

explanation

Answer 15

there are 4 critical physical abilities that relate to work/job performance:
1. static strength: ability to use muscle to lift, push, pull, carry objects.
2. explosive strength: ability to use short bursts of muscle force to propel oneself or an object
3. gross body coordination: ability to coordinate movement of arms, legs, and tose in activities where whole body is in motion
4. stamina: ability of lungs and circulatory systems of the body to perform efficiently over time

men exhibit greater static and explosive strength than women.

pros:
* work analysis should be conducted to determine how important a phys ability test is in the overall assessment of a candidate. can be used to reduce adverse impact as if a job permits employees to perform a wide variety of tasks, it may be possible to assign women to those tasks that involve less static and explosive strength.

cons:
* difficult from an assessment perspective as the test or assessment method may not necessarily be predictive of the actual sustained job description/behavior required. For example manual labor needs endurance to work 24 hours but the test will just see if he/she can lift the heavy material, not necessarily endurance.
* can cause adverse impact if used wrong.

Question 16

ability tests

SKIP cogntive ability
physical ability
**psychomotor ability

psychomotor ability

pros and cons

explanation

Answer 16

involves motor skills related to flexibility, cooridnation, and balance.

distinguished between gross motor skills (large groups of muscles sich as arms and legs and akin to gross body coordination) and fine motor skills (smaller groups of muscles like fingers)

common psychomotor tests assess: finger dexterity, multi-limb coordination, rate control, and control precision. Used for variety of jobs such as aviation, automachine mechanics, sewing machine operators, craft workers, firefighters.

pros:
* In an examination of psychomotor tracking abilities for predicting U.S. Air Force pilot training performance, researchers found that general psychomotor abil-ity predicted how well pilots did on their flying work sample (criterion-related validity coefficient of .28)
* favorable candidate reactions so including such tests may be a matter of gain-ing applicant approval rather than adding to our ability to predict training or job performance.

cons:
* Much research suggests that the predictive validity of psychomo-tor tests is due to its association with g and general psychomotor ability, with little predictive power of specific psychomotor abilities.
* In addition, the incremental valid-ity of psychomotor ability over and above g is small. As such, the value of assessing psychomotor abilities is limited, as such assessments may not provide much predictive power beyond what we get by measuring cognitive ability.

Question 17

ability tests

SKIP cognitive
physical
psychomotor
**sensory/perceptual **

sensory/perceptual ability

Answer 17

sensory/perceptual ability refers to the ability to detect and recogonize stimuli within their environment.

depth perception, peripheral vision, auditory attention, visual color discrimination, and sound localization, to name a few

pros:
* such abilities have been found to be relevant for a number of jobs including nuclear power plant control room operators, pilots, air traffic controllers, and casino dealers.
* for tasks for warehouse workers: perceptual speed and accuracy have been shown to be better predictors than general intelligence in predicting task performance for warehouse workers.
* For jobs in which sensory/perceptual ability is critical for safety reasons, such as for air traffic controllers or pilots, such tests can be a matter of life and death.

also: although Age Discrimination in Employment Act generally prohibits mandatory retirement ages in the U.S. (as discussed in Chapter 3), there are some exceptions, most notably for pilots, federal law enforcement officers, and air traffic controllers. because as we age, our sensory/perceputal abilities diminish.

Question 18

SKIP PERSONALITY INVENTORIES

Answer 18

The basic ratio-nale is that successful employees possess a particular personality structure, and scales reflective of that structure become the basis for selecting new employees.

5 FACTOR MODEL OF PERSONALITY/BIG 5 PERSONALITY THEORY:
1. openness to experience: has been found to be positively related to training performance and speed of adjustment in a new job
2. conscientousness: of all big 5 factors, constantly shows highest correlations with job performance criteria for all occupations. also it is positively related to one’s salary, motivation, and safety performance at work and is negatively related to turnover and absenteeism.
3. extraversion: positively related to sales performance, salary, and job satisfaction
4. agreeableness: positively related to engagement in citizenship behaviors and negatively related to retailiatory behaviors following mistreatment by others.; negatively related to salary.
5. emotional stability: positively related to salary as well as to job satisfaction

cons:
* some authors have cautioned the use of self-report personality tests in selection settings due to low predictive validity for overall job performance. better to use observer ratings of personality (e.g., from peers, friends, roommates, fam-ily members), as there is evidence they result in superior validity coefficients.
* correlations between personality and cognitive ability are generally low

Dark triad: consists of three dysfunctional personality types that have been linked to increased counterproductive work behavior. These characteristics are Machiavellianism, narcissism, and psychopathy
1. machiavellianism: having a dark and cynical interpretation of human nature. The goal in life is to get your way by manipulating other people. No actions or tactics are deemed inappro-priate if they lead to obtaining what you want. Thus, the phrase, “the end justifies the means” is often associated with Machiavellianism.
2. narcissism: high degree of self-importance. They simulta-neously desire to be in control of other people and be admired by them. self-promoting and unaffected by criticism, they exude subtle arrogance. They tend to be dismissive of advice, primarily because they don’t view others as competent com-pared to themselves.
3. psychopathy: lacking any concern for others. They may come across as glib, but they are emotionally shallow. Frequently exhibiting a sense of charisma, others may be drawn to them without being aware of the basis of attraction.

Scores on assessments of the dark triad have been found predictive of counterproductive work behavior.
there is link between narcissism and hindight bias.

Paradoxically, some organizations may initially reward employees who exhibit dark triad behaviors because of their relentless pursuit of self-advancement. In such cases, organizations place more emphasis on the employees having reached cer-tain performance criteria (e.g., sales volume), while being less concerned about the harm inflicted on others in the process. What may at first be seen as positive behav-iors — such as assertiveness, confidence, and persuasion — may be as-yet-unidentified problematic behaviors such as domination, coercion, and manipulation. Indeed, psy-chopaths have been described as “snakes in suits,” because they can work their way into leadership roles and are positioned to cause immeasurable harm within the orga-nization due to their destructive personalities.

FAKING IN PERSONALITY ASSESMENT:
applicants may fake their answers to give what they believe are socially desirable responses.

this is quite common in personality tests because there is less opportunity to explain one self; however, organizations may benefit because when individuals choose to portray themselves as more conscientious than they are, they are demonstrating that they understand the importance of being conscientious within the workplace. we are able to behave counter to our personalities. but there is still desire to halt faking.

how can faking be addressed:
1. Identify faked responses: utilize specific qns designed to reveal response distortion; eliminate candidates showing signs of faking during assessments
2. Use statistical corrections on the test scores, essentially re-scoring the test: practically concerning because question of how much their test scores should be lowered because of detected faking.
3. Compare response times to questions with one another for each candidate sepa-rately: The rationale for this suggestion is based on findings that candidates take longer to respond dishonestly than to respond honestly to items.66 Of practical con-cern with this option is that not all test items are easily timed, and there are many reasons beyond faking that could lead to longer response times on some items (e.g., item reading difficulty, distractions)
4. Use a different source of personality assessment, such as ratings from colleagues, former supervisors, family members, or even from information gleaned from how people present themselves online; problem with this is logistical issues associated with obtaining them for making personnel selection decisions in an employment context
5. warn candidates against giving fake responses; however candidates may perceive few negative consequences for faking versus the potential benefit of being hired

INTEGRITY TESTS
A type of test that purports to assess a candidate’s honesty or character. Designed to identify if job applicants will not engage in deviant behavior on the job.
Fall into two types:
1. overt integrity test: the job applicant clearly understands that the intent of the test is to assess integrity. The test typically has two sections: one deals with attitudes toward theft and other forms of dishonesty (namely, beliefs about the frequency and extent of employee theft, punitiveness toward theft, perceived ease of theft, and endorsement of common rationalizations about theft). Second section deals with admissions of theft and other illegal activities.
2. personality based integrity tests: makes no reference to theft. These tests contain conventional personality assessment items that have been found to be predictive of theft.

concerns:
1. faking
2. biased
3. negative candidate reactions: considered more invasive than other selection procedures; minimized for personality based measures.

polygraph is another method used to establish truthfulness. measures responses of the autonomic nervous system — physiological reactions of the body such as heart rate and perspiration. Controversial use though as people can appear innocent but be guilty or vice versa. but continue to be used in the hirjng process of government agencies involved in national security as well as in law enforcement.

Question 19

SITUATIONAL JUDGEMENT TESTS

Answer 19

A type of test that describes a problem to the test taker and requires the test taker to rate various possible solutions in terms of their feasibility or applicability.

Question 20

SKIP: BIODATA INVENTORIES

Answer 20

A method of assessing individuals in which biographical information pertaining to past activities, interests, and behaviors in their lives is considered.

consist of questions that ask applicants about their life history (i.e., past events) as opposed to questions about behavioral intentions or presumed behavior in a hypothetical situation.85 These questions are meant to assess a wide variety of constructs that are assumed to be predictive of future job perfor-mance.

SCORING OF BIODATA:
1. Rational keying which is when they give more points to response options that most reflect consutrycts that they are intended to reflect.
2. empirical keying: which is when they give more points to response options that are most predictivr of the criterion of interest - such as job performance or turnover.
3. hybrid keying: — a combination of rational and empirical keying such that points are given to options that are most predictive of the desirable criteria, but only if they also make conceptual sense

authors recommend using hybrid scoring to capture the predictive advantages of empirical keying while capitalizing on the interpretability of rational keying. In addition, they found that biodata inventories appear to have moderate-to-high overlap with the Big 5 personality dimensions, but only modest overlap with cognitive ability, suggesting they may add incremental validity when used in combination with other selection assessments.

questions should not be invasive: two types of biodata questions regarded as intrusive:a question that refers to an event that could have been explained away if the applicant had the chance to do so, and a ques-tion with a response that does not reflect the type of person the respondent has since become.89 Questions that are perceived to invade privacy invite litigation against the hiring organization by job applicants.

Question 21

SKIP: DRUG TESTING

Answer 21

method of assessment typically based on an analysis of urine that is used to detect illicit drug use by the candidate

Question 22

SKIP: COMPUTERIZED ADAPTIVE TESTING

Answer 22

computerized adaptive testing (AKA CAT)
A form of assessment using a computer in which the questions have been precalibrated in terms of difficulty, and the examinee’s response (right or wrong) to one question determines the selection of the next question.

military uses it a lot. research as found that CAT achieves greater test security than traditional paper-and-pencil tests

Question 23

SKO[:ONLINE TESTING

Answer 23

refer to tb pg 149 and read it super quick I CBA LOL

Question 24

DO THIS: INTERVIEWS

important!!

Answer 24

most commonly used method personnel selection. not needed for ksao assessment necessarily, but because there is a basic human need to want personal contact with others before placing them in a pisition of importance even if they have a proven track record, a tendency from which personnel managers and others involved in organizational selection do not appear to be exempt.

more dynamic means of testing.

candidates can be succesfully coached to perform well in an employment interview: Impression management is the process of creating a desired impression or effect on individuals we are trying to influence. may backfire if it is too forced or if the candidate’s self promotion is viewed as arrogance.

first two-three mins of small tall, first impression, dress code, firmness of handshake etc. have been found to be very predictive of the internship offers and performance on the formal questions in the interview.

DEGREE OF STRUCTURE: determined by the degree of standardization in questions as well as the degree of standardization in the scoring of responses.

HIGHLY UNSTRUCTURED INTERVIEW: interviewer may ask each candidate different questions. also, the scoring of responses are largely unspecified ahead of time.

HIGHLY STRUCTURED INTERVIEW: interviewer asks standard questions of all job candidates. also the interviewer would use a standardized rating scale to assess the answers given by each candidate.

in reality: interviewer ratings of a candidate’s responses to questions always involve some degree of subjectivity. Thus, no matter how structured the interview, the interviewer does exert some influence on the judgment of the candidate. Most employment interviews fall somewhere along the continuum between highly unstructured and highly structured.

REMEMBER THIS DETAIL: In particular, highly unstructured interviews often focus more on constructs such as general intelligence, education, work experience, and interests, whereas highly structured interviews often focus more on constructs such as job knowledge, interpersonal and social skills, and problem solving.

INTERVIEW FORMATS:
1. situational interviews: presents a situation and asks for a description of the actions the hypithetical, future-oriented contexts in which the applicants are asked how they would respond if they were confronted with these problems. intentions are predictive of actual behavior
2. behavior description interview: asks applicants to provide specific examples of instances from their past that reflect important attributes for the position. based on the idea that past behavior is the best predictor of future behavior.

Question 25

DO : WORK SAMPLES AND SITUATIONAL EXERCISES

Answer 25

WORK SAMPLES:

Work samples are high-fidelity situations where fidelity refers to the level of realism in the assessment. this is when the candidate demonstrates proficiency on a task representative of the work performed in the job. In general, work samples are among the most valid means of personnel selection.

LIMITATIONS:
* they are effective primarily in jobs that involve either mechanical trades or the manipulation of objects; but not effective when the job involves working with people rather than things
* assess what a person can currently do, not potential. seem best suited to evaluating experienced workers rather than trainees.
* time consuming and costly to administer.

SITUATIONAL EXERCISES:
method assessment in which examinees are presented with a problem and asked how they would respond to it.

used for managerial and professional jobs.

situational exercises are different from work samples in that they only mirror part of a job. are usually a family of tests that assess problem-solving ability.

• inbox assessment/in-basket or mailbox test: applicants sort through a full mailbox of things to do. contents are craftedcorrespondence requiring immediete attention and response. The applicant takes the appropriate action to solve the problems presented, such as scheduling a meeting, delegating a task to a sub-ordinate, or asking questions to better understand an issue. Oftentimes, there is not enough time for applicants to fully address all issues in the assessment. This is inten-tional, as it forces the applicant to prioritize information and determine which matters are urgent and which ones can wait. Raters score the applicant on such factors as pro-ductivity (how much work got done) and problem-solving effectiveness (versatility in resolving problems). The inbox test is predictive of the job performance of managers and executives, a traditionally difficult group of employees to select.

problem with this: it is an individual test. Thus, if there are many applicants, too much time may be needed to score the test.

• leaderless group discussion (LGD): a group of applicants (normally, two to
  eight) engage in a job-related discussion in which no spokesperson or group leader has been named. Raters observe and assess each applicant on such factors as individual prominence, group goal facilitation, and sociability. Scores on these factors are then used as the basis for hiring. The reliability of the LGD increases with the number of people in the group

Question 26

DO: ASSESSMENT CENTERS

Answer 26

A technique for assessing job candidates using a series of structured, group-oriented exercises that are evaluated by raters.

NOT PLACES but rather are compilations of group-oriented, standardized activities (that may be conducted in one setting but need not be)

Because assessment centers are expensive, they have been used mainly by large organizations; however, consulting firms also offer assessment centers for smaller companies. Here are five characteristics of the assessment center approach:
1. Individuals selected to complete assessments as part of an assessment center (the assessees) are usually management-level personnel that a company wants to evaluate for possible selection, promotion, or training. can be used for both current and future employees; development
2. can be evaluated individually or in groups up to 20
3. several raters do the evaluation
4. Assessors evaluate the assessees on a number of performance dimensions judged rel-evant for managerial jobs. These dimensions typically include leadership, decision making, practical judgment, and interpersonal relations skills.
5. A wide variety of assessment methods or exercises are used. Many involve group exercises — for example, leaderless group discussions. Other methods include oral presentations, written case studies, role plays, inbox exercises, and interviews. Typi-cally, each exercise is used to provide an evaluation of multiple performance dimen-sions, but it is rare to use each exercise to evaluate every performance dimension. The assessment typically takes one to several days.

LIMITATIONS:
* variance in assessment center ratings is also attribut-able to assessors, the exercises (e.g., leaderless group discussion, interview) used in the evaluations, and the dimensions of behavior (e.g., motivation, problem-solving ability) being rated.
* a matter of theoretical debate whether there should be high agree-ment across different exercises designed to measure the same dimension of behavior. The lack of agreement can be reflective of the psychological complexity of the behavior being assessed
* HIGH COST
* LOW GENERALIZABILITY
* MAXIMAL VS TYPICAL PERFORMANCE