Assesment 1

Question 1

Measurement theory

Answer 1

A branch of applied mathematics that is useful in measurement & data analysis

The fundamental idea of MT is that measurements are not the same as the attribute being measured
Hence, if you want to draw conclusions about the attribute, you must take into account the nature of the correspondence between the attribute & the measurements

Question 2

Correlation

Answer 2

Magnitude & direction of the relationship between an IV (predictor) & DV (criterion)

Linear regression= y=ax+b

Shows a relationship, not a comparison between variables

Not a statistical technique for hypothesis testing, can be used after hypothesis testing to calculate effect size

Question 3

Curvilinear relationship

Answer 3

As one variable increases, so does the other variables, but only up to a certain point, after which, as one variable continues to increase, the other decreases
e.g. Yerkes Dodson law

Question 4

Spurious correlation

Answer 4

Coincidental correlation
Can be misleading
Correlations need to be theoretically driven

Question 5

Measurement theory: accuracy & precision

Answer 5

Accuracy= closeness of a measured value to a standard or known value e.g. sample with mean IQ score of 120 is not accurate

Precision= closeness of 2 or more measurements to each other e.g. IQ score in 3 different sessions

Question 6

Latent variables

Answer 6

Latent variables= constructs are unobserved, hidden or latent variables inferred from the data collected on related observable variables

SEM= multivariate stats analysis technique used to analyse structural relationships among observable & unobserved (latent) variables
Implies a structure for the covariences between the observed variables

Question 7

Latent variable models

Answer 7

The relationship between the observable & the unobservable quantities is described by a mathematical function

Question 8

Classical test theory

Answer 8

Behavioural perspective

Measures the overall score on a test
Manifest behaviour is the unique reason representation of a construct, with no consideration to latent traits

Assumes the existence of the measurement error
Therefore aims to elaborate strategies (statistics) to control or evaluate the magnitude of error
Unit of analysis is the whole test (item sum or mean)

Question 9

CTT evaluation

Answer 9

1) standard error of measurement applies to all scores in particular pop
2) longer tests more reliable than shorter
3) test scores obtain meaning by comparing their position in a norm group
4) unbiased assessment of item properties depends on having representative samples

Question 10

Item response theory

Answer 10

Cognitive perspective

The answer a subject gives to an item depends on his or her level on the latent trait, the magnitude of his or her theta

Proposes the validation of items & not of tests
This favours the composition of large groups of independent items that can be used to create or customise different tests for different purposes

Question 11

Unidimensional IRT

Answer 11

Premise that the interactions of a person with test items can be adequately represented by a mathematical expression containing a single parameter describing the characteristics of the person

Question 12

Assumptions 1 & 2 of unidimensional IRT

Answer 12

1) unidimensionality= a single latent trait variable is sufficient to explain the common variance among item responses
2) local independence= the response of any person to any test item is assumed to depend solely on the persons single parameter & the items vector of parameters, LI is evidence for unidimensionality if the IRT model contains person parameters on only 1 dimension

Question 13

Implications of local independence

Answer 13

Probability of a collection of responses can be determined by multiplying the probabilities of each of the individual responses

Question 14

Assumptions 3 & 4 of unidimensional IRT

Answer 14

3) the characteristics of a test item remain constant over all of the situations where it is used
4) monotonocity= probability of correct responses to the test item increases of does not decrease as the locations of examinees increase on the coordinate dimension

Question 15

IRT evaluation

Answer 15

1) standard error of measurement differs across scores but generalised across pops
2) shorter tests can be more reliable than longer ones
3) test score obtain meaning by comparing their distance from items
4) unbiased estimates of item properties may be obtained from unrepresentative samples

Question 16

Test blueprint (specifications)

Answer 16

Tells you exactly what skills will be tested & how many points each question worth

May include important details

Ensures that test assesses level or depth of learning you want to measure

Development of test specs more common for skills tests

Test specs indicate what dimensions & descriptors can be evaluated in the test & in what proportions

The operationalisation of subjective concepts enables their measurement from set of descriptors, which will represent in the future of the phenomenon under investigation

Question 17

Specific criteria for development of items

Answer 17

1) Behavioural= items must express a behaviour, not an abstraction
2) Objectivity= items should allow for a right or wrong response
3) Simplicity= items should express a single idea in order to avoid ambiguities
4) Clarity= items should be intelligible even to the lowest level of target pop, short sentences with simple & unambiguous expressions
5) Relevance= items must be consistent with the psych trait & other items covering the same construct
6) precision= items must have a position defined in the attribute continuum & be distinct from other items that cover the same continuum
7) neutrality= do not use extreme expressions e.g. excellent, miserable, the magnitude of the persons reaction is given in the response scale

Question 18

Specific criteria regarding test development

Answer 18

1) amplitude= range of simple descriptors to more complex descriptors
2) balance of amplitude

Question 19

Validity

Answer 19

The extent to which a test accurately measures what it is intended to measure

Question 20

Validity: CTT

Answer 20

1st period (1900-1950)= content-related validity (content & face validity)

2nd period (1950-1970)= criterion-related validity (concurrent & predictive validity)

3rd period (1970-current)= construct-related validity (convergent & discriminant validity)

Question 21

Content related validity

Answer 21

Systematic review of the test content to determine if items cover a representative sample of the universe behaviours to be measured & to determine if the choice of items is appropriate & relevant

Global, mostly non-statistical procedure

Question 22

Content-related validity: panel of experts (raters)

Answer 22

A consultation with experts in area of evaluated construct, analyse the representativeness of items in relation to theory on construct & assist in adequacy of items to the target pop

Average of 5 judges

Agreement rate of 80% expected between judges

Asked to make suggestions of improvement

Question 23

Content related validity: pilot study

Answer 23

Procedure seeks to verify whether items have been well understood by target audience

Content validity can be demonstrated statistically by item analysis

Question 24

Content related validity: face validity

Answer 24

Not a statistical or numerical technique
Regards whether a test is an apparent measure of its associated criterion
Involves language & layout- the way content is presented
The test should look good to test takers

Question 25

Content validity procedures

Answer 25

1) setting test goals
2) selection of the universe of behaviours that appear to measure the construct (pool of items)
3) item development
4) item analysis
5) final choice of test items

Question 26

Criterion-related validity

Answer 26

Extent to which a measure is related to either a present or future outcome

Such evidence is provided by high correlations between a test & well-defined criterion measure (ideal is above 0.5)

Criterion will depend on type of construct evaluated e.g. academic performance or group dynamics

Criterion is a standard on which a test will be compared against

Question 27

Criterion-related validity: concurrent validity

Answer 27

Derived from assessments of simultaneous relationship between year & criterion such as between learning disability test & school performance

Involves determining the current status of a person in relation to some classification scheme, such as diagnostic categories

Question 28

Criterion-related validity: predictive validity

Answer 28

The extent to which a score on a scale or test predicts scores on some criterion

Question 29

Construct related validity; construct validity

Answer 29

The degree to which a test measured what it claims or purports to be measuring

Whether a scale or test measures construct adequately

From the construct validity, the degree to which a person has a certain characteristic is inferred

Failures in the validation process may stem from instrument, it’s administration or the theory

Question 30

Construct validity

Answer 30

Content, criterion & construct

Question 31

Construct-related validity: convergent validity

Answer 31

Degree to which 2 instruments measuring the same construct are theoretically & empirically related

Ideal correlation is above 0.5

Question 32

Convergent-related validity; discrimination validity

Answer 32

If 2 measures of the same quality show higher correlations, then 2 measures that do not assess the same quality should not

2 tests of unrelated constructs should have low correlations- they should discriminate between 2 qualities that are not related to each over

May also be related to the ability of an instrument to discriminate groups of subjects that have a high magnitude in an attribute against subjects who have low magnitude

Ideal correlation in predictive validity is below 0.3

Also called divergent validity

Question 33

IRT validity

Answer 33

Can be used to investigate any type of test, whether measuring abilities or attitudes
Content validity & proper representation of latent trait under measurement is one of primary concerns of IRT, hence development of specifications matrix
Items are developed via TRI in systematic- based in descriptors- rather in an intuitive way as CTT does
The remaining steps for the test development & therefore content validity are similar to those deployed by CTT

Question 34

Content validity procedures

Answer 34

1) setting goals
2) identification of dimensions & descriptors
3) specification matrix
4) item development
5) item analysis
6) final choice of test items

Question 35

Criterion related validity

Answer 35

Not common but adoption of specific stats methods for study of criterion validity, non-linear correlations can offer more precision for correlation tests

Criterion could be behaviour or another instrument

Question 36

Factor analysis

Answer 36

A procedure for reducing scores on many variables to scores on a smaller number of ‘factors’

Purpose is to explore underlying variance structure of set of correlation coefficients
Thus, factor analysis is useful for exploring & verifying patterns in set of correlation coefficients

Question 37

Full info factor analysis

Answer 37

Most modern technique for study of construct unidimensionality

Does not require computation of intercorrelations between items since they are based on individuals pattern of response rather than on correlational structure of the multivariate latent response distribution
These models are therefore ‘full info’ models

To attest to construct unidimensionality, FA oblique rotation technique used with the extraction of 2 factors in order to verify if they’re correlated
Presence of high correlation between first 2 factors indicates that, even if there is a second order factor, only a single latent trait is being assessed, so the assumption of unidimensionality is met
Acceptable correlation is atleast 0.5

Question 38

Test info curve

Answer 38

Shows for which range of theta levels the test is particularly valid

Items above the validity range are too difficult & below the range, too easy

Question 39

Factorial components of variance

Answer 39

The variance of the observed variables in relation to their factors can be divided into several parts

A= specific variance 
B= common variance
C= error variance

Question 40

Common variance

Answer 40

Covariance between items & factors

The percentage of communality (common variance) defines the quality of the behavioural representation of the latent trait from the observable variables (test items)

Communality= the sum of squared factor loadings in all factors- the proportion of each variables variance that can be explained by the factors

Question 41

True variance

Answer 41

Reliability-> true variance
True variance= communality + specificity

Uniqueness= complement of communality, corresponds to difference between total variance & value of communality

Question 42

Factor analysis advantages

Answer 42

Most used technique for validation of psych instruments

Reduces large number of variables to manageable item pool

Question 43

Factor analysis disadvantages

Answer 43

Usefulness depends on ability of the researchers to develop a set of conditions that make the technique viable

Subjectivity of factors

Traditional FA assumes that the relations between the variables are typically linear

Question 44

Level of measurement

Answer 44

In conventional factor analysis, all variables must be quantitative

Important to test normality of variables, seeking to control large deviations of normality

Question 45

Reliability for CTT

Answer 45

For CTT; reliability was most investigated psychometric parameter & several statistical techniques were developed to estimate it

Corresponds to consistency of scores obtained by same individuals when they are re-examined with same test at different times, different sets of equivalent items or under other variable conditions of examination

Also concerns accuracy of measure or internal consistency (item-total correlation) between items of test

Question 46

Error variance

Answer 46

Any condition that is irrelevant to objectives of a test represents error variance

Factors reducing error variance= standardisation of test administration, environmental control, instructions, true limits, normative sample

Question 47

Test-retest reliability

Answer 47

Reliability coefficient is correlation between scores obtained by same individual in 2 different administrations of same test

Coefficient of stability or constancy

Ideal correlation is >0.6

Error variance source= interval between test administrations

Question 48

Split half reliability

Answer 48

Reliability coefficient is correlation between scores obtained by same individuals in same test divided into 2 equivalent halves

Coefficient of internal consistency

Ideal correlation is >0.6

Error variance source= content sampling

Question 49

Inter-rater reliability

Answer 49

Reliability coefficient is correlation between scores assigned by each rater

Error variance source= different raters perceptions

Question 50

Parallel (equivalent) forms

Answer 50

Reliability coefficient is correlation between scores obtained by same respondents in application or equivalent forms of test

Coefficient of equivalence

Ideal correlation is >0.7
True variance greater than error variance

Error variance source= content sampling

Question 51

Kuder-Richardson

Answer 51

Reliability coefficient is correlation between scores obtained in each item of test

Kuder-Richardson coefficient

Kuder-Richardson 20 formula used when items are dichotomous

Ideal coefficient is between 0.7-0.9

Error variance source= content sampling & content heterogeneity

Question 52

Cronbach’s alpha

Answer 52

Reliability coefficient is correlation between scores obtained in each item of the test
Similar to Kuder-Richardson technique

Cronbach’s alpha formula is modification of Kuder-Richardson equation & reflects magnitude of covariance among items
Varies from 0-1 (1=100% internal consistency)

Ideal correlation is between 0.7-0.9

Error variance source= content sampling & content heterogeneity

Question 53

Factors affecting reliability

Answer 53

> sample variability, >reliability= >test accuracy

> number of items, >reliability= >test accuracy

Question 54

Reliability for IRT

Answer 54

Reliability not as important as for CTT, though is part of validation process

Reliability can be obtained by estimating item information curve, which corresponds to graphical analysis that shows level of theta to which item brings maximum information

If amount of info small, ability can not be estimated with precision & estimates will be widely scattered about true ability

Item info function depends on item parameters a,b & x

Question 55

Standardisation

Answer 55

The meaning of test scores derives from the frames of reference we use to interpret them & from the context in which the scores are obtained

Standardisation refers to the need for uniformity in all procedures while using a valid & accurate test

Question 56

Norm

Answer 56

Created with scores from respondents who participated in normative study

Expected to be drawn from representative sample (normative group)

Question 57

Norm-referenced tests

Answer 57

Norms usually presented in form of tables with descriptive stats that summarise performance of groups or groups in question

When norms are collected from test performance of groups of people, these reference groups are labelled normative or standardised samples

Norms typically created from calculation of percentiles & standard scores

Mean & SD are main stats used to create norms

Question 58

Norm-referenced tests continued

Answer 58

Norms most widely used frame of reference for interpreting test scores

Used for test takers comparison

Score used to place test takers performance within a pre-existing distribution of scores or data obtained from performance of suitable comparison group

These scores can be obtained by linear/non-linear transformations of obtained data

Question 59

Z-scores

Answer 59

Non-normalised standard score (linear)

Transforms original scores of group measures in units of SD

Normally distributed scores of tests with different means, standard deviations & score ranges can be meaningfully compared once they have been linearly transformed into common scale as long as same reference group used

Question 60

T-scores

Answer 60

Non-normalised standard score (linear)

Same purpose as z-scores, with advantage of eliminating negative numbers & decimal numbers