Final Course Review

Question 1

Definition of psychological testing

Answer 1

: A test is a standardized process or device that yields information about a sample of behavior or cognitive process in a quantified manor; same testing conditions for everyone writing the test, we want the results to be quantifiable, going to be of behavior or cognition

Question 2

Four critical assumptions

Answer 2

1. People differ in important traits 2. Ability to quantify these traits 3. Traits are relatively stable within the individual 4. Measures of the traits related to an actual behavior (whether you would actually do the behavior in the real world)

Question 3

Constructs

Answer 3

concepts for understanding, describing, and predicting human behavior; cannot be directly observed; examples: perfectionism, neuroticism, etc.; essentials: 1. Abstract properties (that occur in some regularity in nature) 2. Connected to concrete observable behavior *A construct is directly or indirectly related to some observable behavior

Question 4

What results in issues in Psychometrics

Answer 4

we make measures and decisions are being made with these measures that impact real people—resulting in challenges)

Question 5

Challenges in measurement

Answer 5

psychological phenomena are complex; participant reactivity (e.g., social desirability, malingering, demand characteristics); observer bias/expectations; use of composite scores (take a bunch of constructs and mash them together and hope it’s one construct e.g., intelligence is comprised of memory and special awareness and more); score sensitivity (don’t know how well the test does as predicting outcomes); lack of awareness of any psychometric information (people in legal and business settings that are using the tests don’t know about psychometric concepts like norms and validity)

Question 6

Testing concerns

Answer 6

issues of privacy; fair use of tests (are they being used in the appropriate setting and individuals; what is the information being used for?); impact on society

Question 7

Ethical themes

Answer 7

the need for high standards of ethics when administering and using tests *reread ethics code; the need for good practice in choosing, administering, interpreting, and communicating the results of tests

Question 8

Competence

Answer 8

a part of the ethics code; to utilize tests responsibly, the psychologist should develop competence in assessment concepts and methodology: 1. An understanding of norms, reliability, validity, and test construction 2. Knowledge of specific procedures applicable to a particular test (administration, scoring, etc.) 3. The psychologist is responsible for continually updating their knowledge and assessment skills 4. Recognizing the boundaries of competency

Question 9

Scales of measurement

Answer 9

Nominal scales: numbers take on the meaning of a verbal label, but don’t signify any particular amount of a trait (e.g., gender); Ordinal scales: numbers denote order or ranking, but not amount of a trait, and there is no consistent differences between numbers (e.g., class rank, 3rd year students compared to 1st year students); Interval scales: numerical differences in scores represent equal differences in trait being measured (e.g., temperature); Ratio scales: have a true zero point, with zero=total absence of the trait being measured AND can make proportional statements, with twice the score=twice the attribute (e.g., weight)

Question 10

Arbitrariness of metrics

Answer 10

unknown when a given score locates an individual on the underlying psychological dimension; unknown when a one-unit change on the observed score reflects the magnitude of change on the underlying dimension

Question 11

Important concepts in measurement

Answer 11

central tendency, variability, normal curve, z-scores, t-scores, correlations

Question 12

Test Construction steps

Answer 12

1. Defining the Test’s Purpose 2. Preliminary Design Issues 3. Item Preparation 4. Item Analysis 5. Standardization and Ancillary Research 6. Preparation of Final Materials and Publication

Question 13

Preliminary Design Issues

Answer 13

1. Background research (*most important thing you can do!) 2. Mode of administration 3. Length 4. Number of scores 5. Question and response format 6. Administrator training

Question 14

Question format

Answer 14

dichotomously scored; Likert scored; forced choice, multiple choice; graded response options; ranking; visual analog scale; open format; performance assessment

Question 15

Guiding principles in writing items

Answer 15

Deal with one ONE central thought in each item (otherwise it’s double barrel); Be precise; Be brief; Avoid awkward wording or dangling constructs; Avoid irrelevant information; Present items in positive language; Avoid double negatives; Avoid terms like all or none; Avoid indeterminate terms like frequently or sometimes

Question 16

Item analysis

Answer 16

item tryout; statistical analysis (*item difficulty [how well people did], *item discrimination [how well it discriminates between high and low], distractor analysis); factor analysis; item selection *know formulas

Question 17

Dimensionality: Importance

Answer 17

FA reveals; implications for appropriate scoring, evaluation, and interpretation of test scores; for example, the number of dimensions (Each dimension should be scored separately (i.e., a person might receive more than one score from a test); Each such score requires its own psychometric evaluation; Each score would be interpretable in terms of the psychological dimension underlying the score)

Question 18

Standardization and Ancillary Research

Answer 18

norming; reliability & validity studies; equating programs (alternate forms, multiple levels [scaling]; equating to a previous edition)

Question 19

Final Materials and Publication

Answer 19

technical manual; score reports; supplementary materials

Question 20

Important Considerations in Test Construction

Answer 20

The original conceptualization is more important than the technical / statistical work; You need to spend substantial time studying the area before starting to write items; In the original design stage, you need to think about the final score reports; When preparing test items, aim for simplicity; be sure to tryout enough items (2-3x); do a simple, informal tryout before the major tryout; from a statistical viewpoint, the standardization group does not need to be large if properly selected

Question 21

Types of Bias:

Answer 21

Sample related issues 1. Selection bias (1. Under-coverage 2. Non-response bias 3. Voluntary response bias) 2. Sampling error (variability among statistics from different samples); Response bias (*know all of these); Test bias (construct bias and predictive bias)

Question 22

Reliability

Answer 22

goal of psychological measurementdetect psychological differences; test scores are used to indicate levels of psychological attributes; differences among people’s test scores are used to indicate true psychological differences among people; to what degree are differences in observed (test) scores consistent with differences in (true) levels of psychological attributes; reliability refers to the degree to which test scores are free from measurement error

Question 23

Assumptions of classical test theory

Answer 23

1. Observed scores on a psychological measure are determined by a respondent’s true score and by measurement error 2. Error is random (consequences: 1. Error tends to cancel itself out across respondents 2. Error scores are uncorrelated with true scores)

Question 24

Random error

Answer 24

blue distribution is the true score; random error increases the variability, does not impact average; we use reliability to understand variability

Question 25

Systematic error

Answer 25

: from things like timing and administration; it does shift the average

Question 26

Estimates of reliability

Answer 26

: test-retest reliability; parallel-forms reliability; internal consistency reliability; inter-rater reliability

Question 27

Reliability and behavioral research

Answer 27

1. There’s a precise link between true correlations, reliability, and observed correlations 2. Measurement error attenuates observed correlations 3. These points apply to all effect sizes, not just correlations 4. Next—reliability affects the likelihood of obtaining results that are statistically significant

Question 28

Reliability in test construction and refinement: item variability

Answer 28

recall—purpose of measurement is to detect psychological differences; items with no/limited variability (in terms of responses by test-takers) may be poor at detecting difference)/ additionally, correlations among items depends upon variability; items without variability cannot correlate with other items (co-variability depends on variability)

Question 29

Validity vs Reliability:

Answer 29

Reliability: degree to which differences in test scores reflect differences in the psychological attribute that affects test scores, whatever that attribute is; does the test’s score reflect something with precision? Validity: what exactly is being reflected in the test scores? Meaning and interpretation of the scores; reliability places an upper bound limit on validity; tests that are reliable are not necessarily valid