PSY598 Flashcards
Testing and Assessment
Psychological Assessment
An objective procedure for sampling and quantifying human behaviour describes
how a person thinks, acts and feels
Organisational Testing- Purpose
Diagnosis -What to do next
What to focus on – (to use resources time money personnel -wisely)
As a benchmark for a longterm project research
To help Individuals develop and change
To recruit the best possible person for the role
Organisational Testing Methods
Benchmark: Quantitative standard
Best Practice: Qualitative standard
HR Audit: cultural functional and effectiveness factors
Organisational Types of Assessment
Observation Questionnaires and Surveys Interviews Focus groups Tests and inventories Existing data – turnover absenteeism data safety stats etc
Hollands Theory
People search for environments that will let them use their skills and abilities express their attitudes and values behaviour is determined by an interaction between personality and the characteristics of environment. TYPES: Realistic Investigative Social Conventional Enterprising Artistic
Criticism of Educational Testing
Serves to perpetuate SES differences no evaluation of soft skills- teamwork, leadership, emotional IQ- are important (if not more) in determining academic success
UMAT: measures reasoning, and intrapersonal skills but does not better than school grades in predicting outcomes in medical training
Organisational: Selection Tools
Application (resume/ cv) Biodata Psych Test Work Sample Interview Reference Checking
Steps in a Clinical Interview
1 Referral question
2 Case History Data
3 Mental Health Examination
4 Psychological Testing (if needed)
Mental Status Examination
an assessment of a patient’s level of cognitive (knowledge-related) ability appearance emotional mood speech and thought patterns at the time of evaluation used in psychiatry neurology gerontology
Components of MSE
Appearance Behaviour Speech Mood Affect Thought form Thought content Perceptions Cognition Insight and Judgement
Assessments Used in Clinical Settings
Intelligence: WAIS Personality: MMPI
Psychopathology: Personality Assessment Inventory/ Millon Adolescent Clinical Inventory
Depression and Anxiety: Becks Depression/ Anxiety Index State-Trait Anxiety Index
Hallucination vs. Illusion
Hallucination: False sensory perception
Illusion: misinterpretation of actual stimuli
Derealisation vs. Depersonalisation
Derealisation: The feeling that ones surroundings are not real
Depersonalisation: a state in which one’s thoughts and feelings seem unreal or not to belong to oneself estranged from the body
Summative Assessment
purely evaluative function- (end of year exam)
Formative Assessment
an assessment aimed at facilitating and evaluating learning
Educational Aptitude Test & Why?
WISC WHY? Giftedness Special needs Admissions
Clinical Interview
technique for collecting info about a client psychoanalytic- detailed exploration of the personal family history pyschosocial development conflict defence self interpersonal processes
Behavioural Assessment- Methods of Analysis
Frequency: # of times behaviour occurs
Duration: time behaviour occurs
Interval: observation time partitioned into small intervals
Behavioural Assessment: The Observed
Antecedent
Behaviour (duration frequency intensity)
Consequences (reinforcement: punishment/reward)
Behavioural Assessment: The Observer?
Individual
Significant other (parent/teacher/ partner)
Objective observer
Electronic device (video recorder)
Behavioural Assessment
Method of data collection (Skinner) observing covert behaviour (direct observation self-report checklists questionnaires checklists and interviews significant-other ratings)
MMPI
assesses major patterns of personality and emotional disorders using empirical-keying approach 567 items
Social Cognitive Approach to Personality
examines the relationship between peoples behaviour the situations in which these occur and their cognitions about them
Multivariate (trait) Approach to Personality
Personality oldest approach- number of dimensions of individual differences that people have in common and that serve to specify the individual’s personality
Personological Approach to Personaity
Murray- personality in terms of the psychogenic needs (un/conscious) of the individual and extent to which the environment promoted/inhibited those needs
Interpersonal Approach to Personality
proposes that personality exists only in the interaction between people and that the study of interpersonal processes is therefore central to personality assessment
Psychoanalytic Approach to Personality
Freud: role on unconscious motivational processes in normal and abnormal personality functioning
Bases for Personality Assessment
Theoretical
Empirical
Diagnostic
Paradigms in Personality Assessment
Psychoanalytic Interpersonal Personological Multivariate (trait) Empirical Social-cognitive Positive psychology
Subscales of WAIS
FSIQ: Full Scale IQ- Verbal Comprehension Perceptual Reasoning Working Memory Processing Speed
Weschler Definition of Intelligence
the aggregate of global capacity of the individual to act purposefully to think rationally and to deal effectively with his environment.
Wechsler Adult Intelligence Scale (WAIS)
most widely used individually administered intellectual assessment batteries- (verbal comprehension working memory processing speed visual-spatial abilities fluid reasoning
Triarchic Theory of Intelligence
Sternberg
- Componential (analytical)
- Experiental (creative)
- Contextual (practical)
Crystallised Intelligence
Accumulated knowledge and skills resulting from educational/life experience (achievement test)
Fluid Intelligence
More pure inherited aspects of intelligence used to solve novel problems and situations (aptitude test)
Explicit Theories of Intelligence
devised by psychologists and other scientists validated using scientific methods sometimes informed by implicit theories
Implicit Theories of Intelligence
generated by individuals and based largely on their observations and opinions of how the world works
Norm Referenced Test
Test that uses performance of a representative sample on the test for evaluating performance of the test taker
(percentile ranks)
Criterion Referenced Test
Test that uses a predetermined empirical/ absolute standard as an objective reference point
e.g. >50% pass <50% fail
Expectancy Tables
method of interpretation that is used when scores on one measure (the predictor; ATAR) are used to predict performance on another measure (the criterion; GPA)
Also used to establish cutoff scores (percentiles z-scores stanine)
Z-score
linear transformation M= 0 SD= 1 number of standard deviations one data point is from the mean
Sten Score
non-linear standard score with a M= 5.5 SD= 2
Percentiles
non-linear transformation expression of the position of a score within a distribution (divide the distribution into 100 parts)
Can be used to ‘normalise’ non-normal distributions of test scores
Non-Linear Transformation
A transformation that preserves the order but not the equivalence of distance between the raw scores (percentile) used for norm-referenced test
WHY? raw score distribution are bunched in the middle- little discrimination draws out the differences in the middle but leaves the value at the tails unchanged
Linear Transformation
a transformation involving + - ÷ × of a constant to all raw scores- preserves the order and equivalence of distance of the raw scores(z-score) used for norm-referenced test
Standard Score
where a score lies in with reference to its mean can be read as a proportion of a standard deviation
Question Formats
Dichotomous (yes/no; true/false)
Polychotomous (more than two options- MCQ)
Likert (uni: 1 to 5 /bipolar -5 to 5 )
Q-Sorts (cards to sort into piles- not like me to alot like me) Open-ended
Constructed response
Levels of Measurement
Nominal/Categorical: categorise people-(Nationality/ DSM IV)
Ordinal: rank order people- (Pay bands: A B C D)
Interval: distance is meaningful (Likert Scale)
Ratio: Meaningful 0 point (Height)
ICC, Fairness and IRT
Construct validity test-retest reliability internal consistency factor loading of items, item correlations, and total score and ICC should be the same across all groups- p value may differ
ANOVA should now show any group x item interactions
Differential Test/ Item Functioning
Test/Items may behave differently for different group of respondents
Item Characteristic Curve
graph of the probability of answering an item correctly with the level of ability on the construct being measured demonstrating
items difficulty
items discriminatory power
probability of correct answer by guessing
x-axis: total
score y-axis: probability of success on a specific item
IRT: assumes people with high scores have a greater ability than those with low scores
Item Response Theory
Functional relationship between a response to a single item and the underlying trait,
detects item/test bias- DIF/DTF
Classical Test Theory
Believes that individuals observed score (on the test)= true score (actual measurement) + error (things that cause things difference)
doesnt not test for biases only reliability
Predicted and True Values- Contingency Table
Predicted Positive Value: a+b
Predicted Negative Value: c+d
True Positive Value: a+c
True Negative Value: b+d
Specificity (Sp)
Extent to which test correctly identifies people who do not have the condition of interest (valid negatives: d/b+d)
Sensitivity
Extent to which test correctly identifies people who have the condition of interest (valid positive: a/a+c)
Utility
Usefullness of test
As validity of selection tool increases- the mean performance of sample increases
Low selection ratios= largest increase in performance
Concurrent Validity
Form of predictive validity in which the test and the criterion are administered jointly.
Criterion-Related Validity
Form of predictive validity in which scores predict/ relate to future performance or test score.
Predictive Validity
The extent to which a score on a test allows estimates to be made about scores on a criterion external to the test itself (criterion/concurrent validity)
Diagnostic Validity
Extent to which test correctly classifies people who do and do not have specific disorders (specificity/ sensitivity)
Convergent/Divergent Validity
Types of construct validity- specifies where correlations should be small and large in terms of psychological constructs (used multitrait multi method matrix)
Construct Validity
Extent to which scores on a test are related to the underlying construct it is measuring and correlates with appropriate variables (convergent/discriminative)
Incremental Validity
Extent to which knowledge of a score on a test adds to that obtained by another, pre-existing test score or psychological characteristic
Content Validity
degree to which the content of a test is representative of the domain it’s supposed to cover, through inspection of the items on the test.
Types of Internal Validity
Construct Validity (Convergent/Divergent)
Diagnostic Validity (Specificity/Sensitivity)
Predictive- (Concurrent/Criterion)
Instrumental Validity
External Validity
the extent to which the results of a study can be generalized to other settings (ecological validity), other people (population validity) and over time (historical/temporal validity).
Validity
Extent to which a test measures what it purports to and to which evidence supports meaning and use of test
5 Steps to Validity
- Careful theory specification
- Development of informative hypothesis tests
- Use of sound research design
- Examination of the degree to which observation confirm hypotheses
- Ongoing revision of both theory and measures
Test Specification
A written statement of the attribute or construct that the test constrcter is seeking to measure and test conditions under which it will be used
Construct/Attribute
Specific idea or concept about a psychological process/underlying trait that is hypothesised on the basis of a psychological theory and target of a psychological test
Empirical Approach
Constructing a psychological test that relies on how the items behave when administered (discriminate between no/high scorers) and tool for personality prediction
Rational-Empirical Approach
constructing a pyschological test that relies on
1) . reasoning from what is known about the construct
2) . collecting and evaluating data about how the test and the items actually behave when administered to a sample
Decision Theory
approach to predictive validity the likelihood of someone displaying certain characteristic or not (above/below cutting point incremental validity criterion-related validity)
Statistics of Regression
Predicting one variable from a score on another- incremental validity
Standard Error of Estimate
Standard deviation of the distribution of the differences between actual and predicted scores
Multitrait-Multimethod Matrix
the pattern of correlations resulting from testing all possible relationships among two or more methods of assessing two or more constructs
Method Variance
variability among scores arising because of the method rather than the constructs of interest
Measurement Error
Standard Deviation: (√ of variance)
Variance: (arithmetic mean of the 2 differences between each value and mean value)
Standard Error of Measurement: (standard deviation of error- tells us how much error is in the score)
Factor Analysis
Identifies clusters of related items on a test- way to increase reliability
Discriminability Item Analysis
Examines the relationship between item response (right/wrong) and total score on all other items- way to increase reliability
Item Analysis
Analyses of an item following its administration and how well item relates to construct measured- way to increase reliability
Increase Reliability
Increase number of items
Factor and Item analysis
Discriminability Item Analysis
Parallel Forms Reliability
Extent to which a person obtains a similar score on an equivalent form of the same test
Inter-rater Reliability
Extent to which different raters agree in their assessment of same sample of test takers
Internal Consistency (Cronbach’s Alpha)
Examines correlation between pairs of items in every possible combination
Types of Reliability
Internal Consistency (Cronbachs Alpha)
Inter-rater Reliability
Test-retest Reliability
Parallel Forms Reliability
Reliability
The consistency stability and accuracy of a test to measure what it purports to in any given circumstance
Psychological Testing vs Assessment
Testing: the process of administering psychological test and obtaining and interpreting scores (IQ of a child) Assessment: more complex answer to a referral question which may include (but not limited to) psychological testing (why does my child have learning difficulties)
Achievement Test
A test designed to predict past learning- crystallised intelligence (end of year exam)
Aptitude Test
A test designed to predict a persons future learning potential-fluid intelligence (GAMSAT)
Objective Tests
Stimuli is less ambiguous require minimal clinical interference in scoring. Have reliability validity & normative database
Projective Tests
Use ambiguous stimuli to indirectly uncover unconscious needs (ink blots)
Generalisability Theory
ideas and procedures based on the idea that consistency of psychological tests depends on specifying the desired range of conditions over which this is to hold
Domain-Sampling Method
sees the test as a representative sample of the larger domain of possible items that could be included
Stratified Sampling
non-probability sampling method where the population is divided into strata and participants randomly selected from each stratum
Local Norms
Derived and developed from and for specific population groups or geographical location- local (as opposed to national norms)
Norms
Average or expected values unique to specified groups allow interpretation of an individual score by comparison to the score of the relevant group
Culture-Fair Test
Test designed to minimise distortion of scores resulting from differences in cultural backgrounds of the test-takers
Types of Psychological Tests
Projective Objective Aptitude Achievement Personality
Defining Characteristics of Psychological Tests
- To make inferences about an individual in a specific context (sample of behaviour/ sign of disposition)
- Objective procedures of administration and scoring (standardised)
- Provides an objective reference point (criterion/ norm reference)
- Must meet ‘psychometric properties’ criteria (reliable and valid)
Test Obsolence
A result of continual development of test result: loses its utility because theory or content of items is wrong/no longer appropriate
Developmental Norms
Norms derived on the basis of any trait, ability, skill, or other characteristic presumed to develop, deteriorate or otherwise be affected by stage of life (chronological age)
Convenience Sample
is one which is convenient or available for use - not necessarily the most representative of the population being studied
Wellbeing at Work
Work conditions (rosters)
Impacts (diet/stress)
Employee Assist Programmes
Clinical Incident Management
CAPA
Computer Assistant Psychological Testing
Accommodation
Adaptation of a test, procedure, or situation, or the substitution of one test for another to make the assessment more suitable for an assessee with exceptional needs
KSAO
Knowledge
Skills
Abilities
Other characteristics
Base Rate
AKA prevalence: Proportion of individuals in the population who show the behaviour of interest in a given psychological testing or assessment
(sum of valid positives and false negatives)
Bias
Factors inherent in a test which systematically prevents accurate impartial results
Fallacy of Bias
Egalitarian Fallacy: any mean difference between groups is evidence of test bias
Culture-Bound Fallacy: group differences are due to the culture-bound nature of items
Standardisation Fallacy: test standardised in one population is biased if used on another.
Sources of Error
Test construction
Test Administration
Scoring & Interpretation
Variance
squared differences between mean and each score
Total Variance is made up of:
True Variance: from differences assumed to be stable (consistent in retest and parallel forms)
Error Variance: irrelevant random sources (can affect reliability)
Variance Goals
To maximise the proportion of total variance attributed to True variance, and minimise proportion of error variance
Central Tendency
Rating error: rater exhibits reluctance to rate at either the positive or negative extremes- ratings are in the middle
Confirmatory Factor Analysis
Employed when an explicitly hypothesised factor structure is tested for its fit with an observed relationship between variables
Exploratory Factor Analysis
Employed to estimate and extract factors, and decide how many to retain
Factor Loading
An outcome of factor analysis, explains correlations between observed variables using a smaller number of factors
Selection Ratio
Proportion of those ested who can be allocated to the category of showing behaviour of interest (sum of valid positives and false negatives)
Instrumental Validity
Extent to which the instrument used is correct for that measurement
Content Validity Ratio
Formular to measure agreement among raters regarding how essential individual item is for inclusion
Criterion Contamination
When a criterion measure is influenced by something other than performance related to the construct caused by: low reliability rater bias cheating
Criterion Deficiency
Criterion fails to measure important aspects related to the construct
Test-retest Reliability
The degree to which test results are consistent over time
greater than 6 months
Confidence Interval
Range/band of test scores that is likley to contain trues scores
