Week 3 Reliability and Validity

Question 1

Define reliability

Answer 1

The degree to which a test tool produces consistent results (when measuring the same thing).

e.g. the scale which measures a consistent weight each time is considered reliable.

Question 2

Define validity

Answer 2

The extent to which a test measures the construct it is intended to measure. e.g. a scale measures weight nothing else, an IQ test measures intelligence

Question 3

Why are reliability and validity important?

Answer 3

• diagnosis
• assessment of ability
• treatment decisions and monitoring outcomes
• research

Question 4

True or false, tests can be reliable without being valid.

Answer 4

TRUE

tests can consistently produce the same results but not accurately measure what you want them to.

Question 5

True or false, tests can not be valid but still be reliable

Answer 5

FALSE

Tests cannot be valid without being reliable.

Question 6

Describe classical test theory (Charles Spearman)

Answer 6

States that test scores are the result of:
- factors which contribute to consistency - stable under examination (“True Scores”)

• factors which contribute to inconsistency - characteristics of test taker, or situation that are not related to the characteristic being tested (errors of measurement/ confounders)

Question 7

What is the formula for test theory?

Answer 7

X = T + e

X= obtained score
T= true score
e= errors of measurement 

e.g. Anxiety score on test =(true) anxiety + error

Question 8

List the different sources of error

Answer 8

Item selection

Test administration

Test scoring

Systematic measurement error

Question 9

Describe the following source of error: item selection

Answer 9

sample of items chosen may not be reflective of every individual’s true score

Question 10

Describe the following source of error: test administration

Answer 10

general environmental conditions e.g. temperature, lighting, noise, states/mood of the test taker

Question 11

Describe the following source of error: test scoring

Answer 11

Subjectively scored tests e.g. projective tests and essay exams

Question 12

Describe the following source of error: systematic measurement error

Answer 12

test may consistently tap into something other than the attribute being tests

e.g. test of introversion may actually test aspects of social anxiety without knowing

Question 13

Explain domain sampling theory

Answer 13

Central concept in classical test theory;

With Domain Sampling, tests are constructed by randomly selecting a specified number of measures from a homogeneous, infinitely large pool.

A sample of items is reliable to the extent that the score it produces correlates highly with these true scores

Question 14

Are longer tests more reliable?

Answer 14

Technically, yes because according to domain sampling theory, these tests will include more items from the “universe” of possible domains thus testing more aspects of an item.

Question 15

What are two elements of reliability that are observed/tested?

Answer 15

Stability over time - extent to which test remains stable when it is administered on more than one occasion

Internal consistency - extent to which a psychological test is homogenous or heterogenous

Question 16

Describe the test-retest (stability) measure of evaluating reliability.

Answer 16

same test administered to same group twice at two different time points.

Question 17

What are considerations/ limitations for the test-retest measure?

Answer 17

• consider strong correlations between 2 scores
• consider time lapse between test administrations
• practice effects, maturation, treatment effects/ setting all impact scores

Question 18

Test-retest is an appropriate measure for I___ and E___

It is inappropriate for S___ A__ and W__ of a b_

Answer 18

Intelligence and Extraversion (stable over time)

State anxiety and weight of a baby

Question 19

Describe the parallel or alternative forms measure of evaluating reliability.

Answer 19

two forms of the same test developed; different items selected according to the same rules. e.g. alternative exam for PSY3041

Question 20

please select one of the two options:

• same
• different

parallel forms have ___ distribution across scores (means and variance equal)

Answer 20

same

Question 21

please select one of the two options:

• same
• different

alternate forms have ___ distribution of scores

Answer 21

different

means and variance may not be equal

Question 22

What are the similarities between parallel and alternate forms of reliability?

Answer 22

• both are matched for content and difficulty
• stable construct required
• two tests administered to the same group (looking for strong correlations between the versions)
• influenced by changes between testing times e.g. fatigue
• additional source of error: item sampling/ slightly diff items

Question 23

Describe the split half method of evaluating reliability.

Answer 23

test is divided into halves which are compared (randomly split, odd-even system or top vs bottom).

rationale: if scores on 2 half tests from single administration are highly correlated, scores on 2 whole tests from seperate administration should be highly correlated
- estimates of reliability will be smaller because smaller number of items

Question 24

What is the purpose of the Spearman-Brown formula/ correction?

Answer 24

As the reliability based on the split half is smaller due to a smaller number of items, the Spearman-Brown formula is applied to estimate reliability if each half of the test was the same length as the test.

-internal consistency tested

Question 25

Which reliability coefficient is used to measure internal consistency?

Answer 25

Cronbach’s alpha - a generalised reliability coefficient for scoring systems that are graded for each item.

• it is the mean of ALL possible split-half correlations, corrected by the spearman-brown formula
• ranges from 0 (no similarity ) to 1 (perfectly identical).

Question 26

What are acceptable levels of reliability?

Answer 26

.70 - .80 acceptable or good

greater than .90 may indicate redundancy of items –> high reliability is important in clinical settings when making decisions e.g. decision making capacity assessment

Question 27

What is the standard error of measurement? (SEM)

Answer 27

allows for the estimation of the precision of a specific (individual) test score.

The larger the SEM the less certain we are that the test score represents the true test score.
- Confidence intervals (CI) are often used

Question 28

A way of conceptualising validity is:

A test is valid to the extent that inferences made from it are, a___, m___ and u__.

Answer 28

A test is valid to the extent that inferences made from it are, APPROPRIATE , MEANINGFUL and USEFUL.

• validity must be relevant to the CONTEXT and POPULATION in which you are measuring a construct.

Question 29

What are the types of validity evidence?

Answer 29

• Face validity
• Content validity
• Criterion related validity
  Predictive evidence
  Concurrent Evidence
• Construct validity
  Convergent Evidence
  Discriminent evidence

Question 30

Explain face validity and how it’s measured

Answer 30

Does the test look like it measures the relevant construct?

• social acceptability issue
• must be an obvious link between construct and test times
• assessed using test-taker’s opinion

Question 31

Explain content validity and how it’s measured

Answer 31

The extent to which the items on a test represent the universe of behaviour the test was designed to measure

e.g. anxiety test examines all aspects of anxiety not just affect

• sampling issue
• commonly used in vocational settings/ achievement
• assessed by experts opinion on subject manner - logical deduction

Question 32

What are some issues for content-related validity

Answer 32

Construct underrepresentation
- failure to capture an important component of a construct e.g. depression scale which only measures emotions and thoughts but not behaviour

Construct- irrelevant variance

• measuring things other than desired construct
• e.g. wording of scale may cause people to answer in a socially desirable way

Question 33

Explain criterion-related validity

Answer 33

extent to which a measure is related to an outcome (criterion)

e.g. high school marks used to predict university performance or relationship satisfaction used to predict separation

Question 34

What is concurrent evidence? (criterion-related validity)

Answer 34

a comparison between measure in question and an outcome assessed at the same time

Question 35

What is concurrent evidence? (criterion-related validity)

Answer 35

How well a test predicts performance on a criterion. Compares measure in question with an outcome assessed at a later time. e.g. ATAR score used to predict uni marks

Question 36

What is construct validity?

Answer 36

A multi-faceted process concerned with establishing how well a test measures a psychological construct.

Question 37

What is convergent evidence? (construct validity)

Answer 37

The degree which two constructs which should be theoretically related are actually related. e.g. relationship between low self esteem and depression

• correlate test scores between two the two measures

Question 38

What is discriminant evidence? (construct validity)

Answer 38

a.k.a divergent evidence

aims to demonstrate that the test is unique. Low correlations should be observed with constructs which are unrelated to what the test is trying to measure

• also want to discern between similar but different constructs e.g. self-esteem and self efficacy
  e. g. scores on anxiety measure should be different from depression if both are being assessed within the same test.

Question 39

What is factor analysis ?(construct validity)

Answer 39

observe pattern using factor analysis.

• some items within a test may be highly related and form a set; whereas others may not be related to these and form a different set.

Question 40

Explain the two methods of factor analysis

Answer 40

Exploratory factor analysis
- don’t know how many underlying constructs/ clusters will be formed

Confirmatory factor analysis
- pre-developed test e.g. DASS, analysis occurs to confirm that there are actually multiple factors being measured