Zumbo 2013

Question 1

Define a construct:

Answer 1

A construct may be conceived of as a concept or a mental representation of shared attributes or characteristics, and it is assumed to exist because it gives rise to observable or measureable phenomena.

Question 2

Name the 6 Measurement Theories

Answer 2

Observed Score approaches include: 1. classical test theory and 2. generalizability theory

Latent variable approaches include 3. factor analytic theory, 4. item response theory (IRT), 5. Rasch Theory and 6. Mixture Models

Question 3

1. CTT - True Score Theory

Answer 3

is based on the decomposition of observed scores (x) into true Scores (T) and Error Scores (E). X = T + E. Only produces a single estimate of reliability and standard error of measurement.

Question 4

2. Generalizability Theory

Answer 4

is an outreach of CTT because it is often used to decompose the E into different facets or sources ie) error results from items selected, raters used, gender of test administrator. Unpacking the error E one redefines the true score T

Question 5

3. Factor Analytic Theory

Answer 5

It is based on computational tools and statistical modeling strategy , model estimation methods and fit statistics.

Question 6

4. IRT - Item Response Theory

Answer 6

Focuses on the range of latent ability )theta) and the characteristics of items at various points along the continuum of this ability. IRT produces estimates across the range of the latent variable measured by a test.

Question 7

5. Rasch Theory

Answer 7

Can be characteristed as IRT with only one item parameter, item difficulty. It has a guessing parameter of zero and an item discrimination parameter value of 1 for all items.

Question 8

There are 4 primary approaches to the development of scales and measures

Answer 8

1. Rational-Theoretical
2. Factor Analytic
3. Emperical Criterion keyed
4. Projective

Question 9

Rational Theoretical

Answer 9

1. the researcher uses either theory or an intuitive common sense approach to developing items for a test and is the most commonly used approach. Expert opinion forms the basis for the development and selection of items.
2. The most common
3. Expert opinion forms the basis for the development and selection of items.

Question 10

Factor Analytic Approach

Answer 10

Items are selected on the basis of whether they load on a factor and a statistical rule forms the basis for the development and selection of items. Many large personality inventories have been developed using this approach.

• I want to derive the items from a large pool to determine which items for which to factor.
• Second most commonly used
• Most tests today use some combination of the rational-theoretical and factor analytic approaches.

Question 11

Emperical Criterion Approach

Answer 11

• items are selected if they can discriminate the group of interest from a control group, is not frequently used today in the development of measures. MMPI

Do these items discriminate between items? Very effective in prediction.

Question 12

Projective Approach

Answer 12

not many new tests use this. The basic idea behind it is to use ambiguous stimuli, inkblots, pictures and have individuals create their own drawings, and they will project their own concerns, fears, attitudes and beliefs onto the drawing.

Not popular in NA, used a lot in Europe

Question 13

2 Types of Tests

1. By field of study ie) personality, intelligence, achievement, aptitude
2. by general administration procedures: individual tests administered one on one or group tests.
3. by general type of info gathered: a) self report personality test, b) performance or task - IQ, class exam or c) observational
4. By purpose and scoring / interpretations Norm Referenced and Criterion Tests

Answer 13

1. Maximum Performance Tests measure howe well an individual performs under standard conditions when exerting maximal errort and are presumed to include measures such as intelligence test and achievement tests. ie) performance or task related tests.
2. Typical Response tests measures an individual’s response in a situation and are presumed to include measures such as personality tests and attitude scales. ie) self report and observational tests

Question 14

Another two types of tests

Answer 14

1. Norm Referenced Tests: compares an individuals performance on a test with a predefined population or normative group. Items selected to have average difficulty levels and high discrimination between low and high scorers on the test. Interpretation of test scores are based on percentiles, standard scores and grade equivalent scores. Scores are based on how an individual scored relative to the normative group but gives little information on the persons knowledge of, performance on, or level of the construct.
2. Criterion Referenced Tests: evaluates performance in terms of mastery of a set of well defined objectives, skills or competencies. Items are selected on the basis of how well they match the learning outcomes that are deemed most important. Scores based on percentages or non-master-mastery categories. Outcomes give detailed info about how well a person has performed on each of the objectives, skills or competencies included in the test.

Question 15

A third test

Answer 15

Ipsative can be contrasted with norm referenced tests. An individual’s performance is compared with his or her performance either in the same domain or construct over time or relative to his or her performance on other domains or constructs. This is referred to as profiling.

Question 16

Types of Scaling

Answer 16

1. Thurstone’s Equal Appearing Interval Scaling
2. Likert’s Summarative Scaling
3. Guttman’s Scalogram Analysis

Question 17

1. Thurstone’s Equal Appearing Interval Scaling
2. Selection of 100 - 200 Statements
3. No of judges asked to sort them into single pile from highest to lowest
4. Median rank of each statement computed and its the scale value of that statement
5. Select a limited no of statements about 25 having equal intervals btw successive items and spanning the entire range of values.
6. Applying scale to respondent - they were asked to indicate the statement which applies to him/her
7. Respondent score will be avg score of item selected.

Answer 17

One selects items that not only reflect a range of attitudes but also cover that range at roughly equal intervals. To do this, judges rate each item according to the severity of response (or level of attitude) it represented on an 11 point scale and the mean and SD are used to select items at these intervals. Respondents are asked to agree or disagree with each item, the score for each item is equal to the mean rating assigned to it and the overall score is obtained by averaging the ratings over all of the items with which the respondent agrees. It is expensive and time consuming and is meant to produce scores on ain interval scale.

Question 18

2. Likert’s Summarative Scaling

Answer 18

Respondents used symbols to indicate the degree to which they agreed or disagreed with statements and these symbols were converted to a scale ranging from 1 - 5. The total score was obtained by summing the points assigned for each statement. The goal is to combine item responses for people in such a way that the obtained numbers represent reliable and valid individual differences among people.

Question 19

3. Guttman’s Scalogram Analysis

Answer 19

Consists of a unidimensional set of items that are ranked in order of difficulty from the least extreme position to the most extreme. If someone scoresa 3 on a 5 item Guttman scale then they would agree with items 1 to 3 but and disagree with items 4 and 5. This perfect relationship is rarely achived and thus some degree of deviation from this is expected. Many achievement tests use a form of Guttman scaling to order questions on the basis of difficulty. If you can answer a 5 difficulty question than you would assume that you could answer 1 - 4

Question 20

Response Formats

Answer 20

1. Continuous -
2. Orginal
   They can both be broken down into direct estimation and comparative methods
3. Dichotomous - yes or no, true or false and agree/disagree

Question 21

Direct Estimation Methods

Answer 21

The respondent provides a direct a estimation of the magnitude of an attribute or characteristic. Likert scale in format. A semantic differential format presents bipolar adjectives ie) strong vs weak as anchors with a # of point between them.

Question 22

Comparative Methods

Answer 22

Compare the magnitude of an attribute or characteristic with something else. ie rate your quality of life relative to a) other people, b0 their quality of life in the past or c) their ideal way of life. Likert scale ranging from much worse to better than others

Question 23

Scoring

Answer 23

1. Continuous: Infinite or large # of points.
2. Ordinal: involve 3 to 10 possible values
3. Dichotomous: only 2 values, 0 or 1 ie) True/False
4. Reverse Scoring: use of positively and negatively worded items on a scale to identify when a respondent is displaying acquiescence.
5. Item Weighting: usually equal weight is assigned to each item.

Question 24

Item Analysis Procedures in CTT

Answer 24

1. Item-Total correlations consist of the correlation between the score on a single item and the composite or total score on the scale.
2. Alpha if Item Deleted
3. Corrected Item - total correlations consist of the correlation between the score on a single item.
4. Inter-item correlations
5. Item Difficulty Index
6. Item Discrimination Index

Question 25

Item Difficulty and Item Discrimination

Answer 25

1. The item difficulty index = the proportion of respondents who answer an item correctly.
2. The item discrimination index: a measure of the effectiveness of an item in discriminating between high and low scorers on a test.
   Rule: The item discrimination index is maximized when the item difficulty index is close to .5.

Question 26

Item Analysis Procedures in IRT

Answer 26

1. Item Difficulty (the b parameter)
2. Item Discrimination (the a parameter)
3. Guessing (the c parameter); the probability of selecting the correct answer at the lowest level of ability)
4. Conditional reliability
5. Conditional Standard error of measurement

Question 27

Unlike CTT, which consists of item difficulty and discrimination in relation to the sample of respondents, IRT examines difficulty and discrimination across the range of the latent variable.

Answer 27

The item characteristic curve = the regression of the probability of endorsing the item (or, in achievement tests, the probability of getting the item correct) onto the latent variable score (commonly denoted as theta)

Question 28

Test Validation and Explanation

Answer 28

Validity is an integrated and evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores or other modes of assessment.

It involves presenting evidence and providing a compelling argument to support the intended inferences and show that alternative or competing inferences are not more viable.

Question 29

Messick’s Progressive Matrix

Answer 29

is the consequential basis which adds both value implications and social consequences. Value implications challenge researchers to reflect on the values that led to their interest in and labeling of the construct, the theory underlying the construct and its measurement and the broader social ideologies that affected the development of the identified theory.

Question 30

Social Consequences:

Answer 30

Refers to the unanticipated or unintended consequences of legitimate test interpretation and use. Concerned with the about the relationship between score meaning and social consequences.

Question 31

Hubley and Zumbo put together a framework and links social and personal consequence and unintended social and personal side effects.

Answer 31

Psychometrics and other theories as well as values influence the construct, the measure, and validity and validation.

Question 32

1. Content Related Evidence

Answer 32

Examines the degree to which elements of assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose. Not just item content but also instructions, response formats, and scoring instructions.

Question 33

2. Score Structure

Answer 33

Factor Analysis is used to a) discover how many factors are being tapped by the items in the test or b) confirm whether the test items measure the factors intended. Knowing the factor structure of tests is important because it greatly affects how one scores a test and how one assesses the reliability of score and the validity of inferences made from a test.

Question 34

3. Reliability

Answer 34

Reliability refers to the degree to which test scores are repeatable or consistent. Free from measurement of error.

1. Alternate forms: two different forms of the same test were administered to the same group of people in the same session or on different occasions.
2. Test-Retest the same test administered on two different occasions.

Question 35

There are 3 different estimates of reliability:

Answer 35

1. Split Half Reliability: one correlates the scores from two halves of a test that has been administered only once and applies the Spearman-Brown formula to correct for an underestimation of reliability that results from treating the test as being only half its length.
2. Coefficient Alpha: is the most commonly reported estimate of reliability. It can be thought of as the mean of all possible split-half coefficients corrected by the Spearman-Brown formula.
3. Kuder Richardson formula 20: refers to the case in which coefficient alpha is used with dichotomous (yes/no, true/false) data. In each case a reliability coefficient tells you the variability in the sample scores that is the result of individual differences rather than random (unsystematic) measurement error.

Question 36

Interrater (or interscorer) reliability

Answer 36

One is interested in how repeatable the scores are when two or more different people are scoring or observing the same behaviour. If interrater reliability is low, then one needs to consider whether the scoring criteria are clear or complete enough, whether training was sufficient or whether some raters are not doing a good job.

Question 37

Reliability and Validity Coefficients

Answer 37

provide related but separate info. High reliability coefficients indicate a high proportion of true score variance in the observed scores and thus one is confident that one is measuring real individual differences.

Question 38

4. Criterion Related Evidence

Answer 38

demonstrates the degree to which scores obtained on a measure are related to a criterion. A criterion is an outcome indicator that represents the construct, diagnosis or behaviour that one is attempting to predict using a measure.

a criterion can be thought of as being what one would really like to have (a diagnosis) but cannot obtain because of cost or time and so the test acts as a substitute or a short cut.

Can be predictive or concurrent

Question 39

Predictive evidence

Answer 39

examines how well a score on a measure is related to or predicts a future criterion (behaviour, test performance, or diagnosis obtained at a later date)

Question 40

Concurrent evidence

Answer 40

examines how well a score on a measure is related to or predicts a current criterion.

Question 41

5. Convergent and Discriminative Evidence

Answer 41

Convergent measures may consists of measures of highly related constructs (eg depression and anxiety ) or the same constructs (depression) in the latter case, correlations of such scores are sometimes misidentified as criterion-related validity evidence.

Question 42

Correlations between convergent measures should be relatively high, whereas correlations between discriminant measures should be relatively low.

Answer 42

Discriminant validity coefficients should be significantly lower than convergent validity coefficients.

Question 43

The most common types of norms used are:

Answer 43

1. Percentiles

2. Standard Scores

Question 44

Percentile:

Answer 44

A percentile is a ranking that provides info about the relative position of a score within a distribution of scores. It indicates the percentage of people in the specific normative group whose scores fall below a given raw score.

Question 45

Standard Scores:

Answer 45

indicate where a raw score sits in the distribution relative to the mean. They indicate how far away the raw score is above or below the mean.

Common standard scores are z scores, with a mean of 0 and a standard deviation of 1.

T Scores with a mean of 50 and a SD of 10.

They are linear transformations of raw scores. That is the mean and the SD of the scores may change but the shape of the distribution remains the same.

Question 46

Norms consist of percentiles and standard scores and are often used to:

Answer 46

interpret an individuals performance relative to that of other people of the same group.