Developing & Administering Psych Tests

Question 1

CTT

Answer 1

Test development is performed by the intuitive & more or less random collection of a sample of items from an infinite domain of potential items (face validity)

Question 2

IRT

Answer 2

Test development is carried out from the identification of behaviours that will constitute the empirical representation of the latent traits

Question 3

Dimensions (Factors)

Answer 3

Dimensions & factors sometimes come together

For geometry, dimension is the measure of the size of an object, usually given as length, width & height

Question 4

Unidimensional vs multidimensional

Answer 4

Unidimensional= 1 dimension, e.g. attitude- favourable unfavourable

Bidimensional= 2 dimensions, e.g. skill- verbal (insufficient sufficient) & non-verbal (insufficient sufficient)

Tridimensional= 3 dimensions, e.g. burnout- intensify, level & influence

Question 5

Guilford’s (1967) model of the structure of the intellect

Answer 5

Operation

Product

Content

Question 6

Descriptors

Answer 6

A descriptor refers to an observable behaviour states objectively & concisely

Are the significant & important features of the construct & provide condensed info on what should be evaluated

Question 7

Descriptors should

Answer 7

1) Should be representative & significant for the measurement of the psychological attribute
2) have validity, objectivity & consistency
3) be focused on clear, practical & easy to understand aspects
4) be easy to measure, based on easily available info
5) allow relationship with other descriptors without overlapping them
6) serve as a reference to the development of items

Question 8

Bloom’s taxonomy

Answer 8

Triangle

1) Remember
2) understand
3) apply
4) analyse
5) evaluate
6) create

Question 9

Test blueprint (specifications)

Answer 9

Tells you exactly what skills will be tested & how many points each question worth

May include important details

Ensures that test assesses level or depth of learning you want to measure

Development of test specs more common for skills tests

Test specs indicate what dimensions & descriptors can be evaluated in the test & in what proportions

The operationalisation of subjective concepts enables their measurement from set of descriptors, which will represent in the future of the phenomenon under investigation

Question 10

Items

Answer 10

A command or question requested to an individual for the investigation of a descriptor

Seems to examine perceptions, interests, opinions, attitudes, knowledge, skills & aptitude’s related to test content domain

Must represent a descriptor fully & accurately

Question 11

Sources of items

Answer 11

Theory & specifications matrix

Other measures of the same construct

Interview with target pop

Interview with experts in psychometrics or in the area whose test is intended for

Question 12

Specific criteria for development of items

Answer 12

1) Behavioural= items must express a behaviour, not an abstraction
2) Objectivity= items should allow for a right or wrong response
3) Simplicity= items should express a single idea in order to avoid ambiguities
4) Clarity= items should be intelligible even to the lowest level of target pop, short sentences with simple & unambiguous expressions
5) Relevance= items must be consistent with the psych trait & other items covering the same construct
6) precision= items must have a position defined in the attribute continuum & be distinct from other items that cover the same continuum
7) neutrality= do not use extreme expressions e.g. excellent, miserable, the magnitude of the persons reaction is given in the response scale

Question 13

Specific criteria regarding test development

Answer 13

1) amplitude= range of simple descriptors to more complex descriptors
2) balance of amplitude

Question 14

Number of items in test

Answer 14

Depends on complexity of the construct

CTT= 3 times more than the final number expected, pool of items

IRT= 10% additional items, theoretical validity

Question 15

Malhotra (2013); classification of scaling techniques

Answer 15

Scaling techniques (comparative scales & non comparative scales)

Comparative scales (paired comparison, rank order, constant sum/allocating points, Q-sort & other procedures)

Non-comparative scales (continuous rating scales & itemised rating scales)

Itemised rating scales (likert, semantic differential, Stapel)

Question 16

Advantages of Dichotomous item format

Answer 16

Ease & speed of application, processing & analysis

Statistical analyses can be performed to identify strengths & weaknesses of test items

Scoring system is ambiguous

Markers personal opinions & impressions can be controlled

Question 17

Disadvantages of dichotomous item format

Answer 17

Demand a lot of time & care while writing distractors

Reduced number of alternatives that can be chosen

Respondent May be influences by alternatives presented

Question 18

Advantages of polytomous item format

Answer 18

Can be considered as ordinal or internal variables

Since they have different levels or degrees to evaluate same attribute, polytomous items tend to be more accurate than dichotomous items when attitudes are measured

Question 19

Disadvantage of polytomous item format

Answer 19

Difficult to choose between category levels

Question 20

Itemised rating scale decisions

Answer 20

Number of categories; no single optimal number, traditional guidelines suggest between 5-9

Odd or even (forced vs unforced) number of categories

Question 21

Testing

Answer 21

Testing is a social construct in which all parties should seek a common shared understanding of the process

Testing procedures are normally designed to be administered under carefully controlled or standardised conditions that embody systematic scoring protocols

Question 22

Test administration should

Answer 22

Provide measures of performance & involve drawings of inferences from samples of behaviour

Include procedures that may result in the qualitative classification or ordering of people

Question 23

Rights & responsibilities of test givers & takers

Answer 23

1) Purpose & procedures for testing clearly stated & communicated to all parties involved in testing process
2) how test info will be used is clearly stated & communicated to all parties involved
3) procedures for dealing with enquiries & complaints about process of testing are clearly stated & communicated to all parties

Question 24

Tester & test administrators expect test takers to

Answer 24

1) ask questions prior to testing if uncertain about why test is administered, how administered, what they will do & what will be done with results
2) Inform about any condition they believe might invalidate results of which test takers would wish to have taken into consideration
3) follow instructions of administrator
4) be aware of consequences of not taking a test if choose not to & accept them

Question 25

After the test session

Answer 25

Tell test takers what to do with any equipment or materials they’ve used & what they make take away with them

Tell what will happen next, when told results & any feedback

Question 26

Open administration

Answer 26

Tears available for completion by anyone on demand

Question 27

Controlled administration

Answer 27

Provides with restricted access to test session, but administration is carried out without someone being present to supervise

Question 28

Supervised administration

Answer 28

Traditional mode for tear administration in group testing

Provides level of control needs for maximum performance testing

Question 29

Data protection

Answer 29

Personal data may only be kept in a form that permits identification of pp for no longer than is necessary for purposes for which it was processed

Can keep anonymised data for as long as you want

Question 30

Test maladministration

Answer 30

Any act that affects the security, confidentiality or integrity of the assessment or could lead to results that don’t reflect the pp’s real scores

Examiners should be aware that interaction with test takers can influence the results

Question 31

Stereotype threat

Answer 31

May have very important effects on test scores

May explain 50-80% difference between males & females on SAT maths section

Being aware of negative stereotype may inhibit performance on tests & academic performance