Methods and Statistics used in Research Studies and Test Construction

Question 1

an umbrella term for all that goes into the process of creating a test

Answer 1

Test Development

Question 2

brainstorming of ideas about what kind of test a developer wants to publish
- stage wherein the ff. is determined: construct, goal, user, taker, administration, format, response, benefits, costs, interpretation
- determines whether the test would be norm-referenced or criterion-referenced

Answer 2

I. Test Conceptualization

Question 3

preliminary research surrounding the creation of a prototype of the test

Answer 3

Pilot Work/Pilot Study/Pilot Research

Question 4

stage in the process that entails writing test items, revisions, formatting, setting scoring rules
- it is not good to create an item that contains numerous ideas

Answer 4

II. Test Construction

Question 5

reservoir or well from which the items will or will not be drawn for the final version of the test

Answer 5

Item Pool

Question 6

relatively large and easily accessible collection of test questions

Answer 6

Item Banks

Question 7

refers to an interactive, computer administered test-taking process wherein items presented to the testtaker are based in part on the testtaker’s performance on previous items

Answer 7

Computerized Adaptive Testing:

Question 8

occurs when there is some lower limit on a survey or questionnaire and a large percentage of respondents score near this lower limit (testtakers have low scores)

Answer 8

Floor Effects

Question 9

occurs when there is some upper limit on a survey or questionnaire and a large percentage of respondents score near this upper limit (testtakers have high scores)

Answer 9

Ceiling Effects

Question 10

ability of the computer to tailor the content and order of presentation of items on the basis of responses to previous items

Answer 10

Item Branching

Question 11

form, plan, structure, arrangement, and layout of individual test items

Answer 11

Item Format

Question 12

offers two alternatives for each item

Answer 12

Dichotomous Format

Question 13

each item has more than two alternatives

Answer 13

Polychotomous Format

Question 14

a format where respondents are asked to rate a construct

Answer 14

Category Format

Question 15

subject receives a longlist of adjectives and indicates whether each one if characteristic of himself or herself

Answer 15

Checklist

Question 16

items are arranged from weaker to stronger expressions of attitude, belief, or feelings

Answer 16

Guttman Scale

Question 17

require testtakers to select response from a set of alternative responses

Answer 17

Selected-Response Format

Question 18

Has three elements: stem (question), a correct option, and several incorrect alternatives (distractors or foils), Should’ve one correct answer, has grammatically parallel alternatives, similar length, alternatives that fit grammatically with the stem, avoid ridiculous distractors, not excessively long, “all of the above”, “none of the above” (25%)

Answer 18

Multiple Choice

Question 19

a distractor that was chosen equally by both high and low performing groups that enhances the consistency of test results

Answer 19

Effective Distractors

Question 20

may hurt the reliability of the test because they are time consuming to read and can limit the no. of good items

Answer 20

Ineffective Distractors

Question 21

less likely to be chosen, may affect the reliability of the test bec the testtakers may guess from the remaining options

Answer 21

Cute Distractors

Question 22

Test taker is presented with two columns: Premises and Responses

Answer 22

Matching Item

Question 23

Usually takes the form of a sentence that requires the testtaker to indicate whether the statement is or is not a fact (50%)

Answer 23

Binary Choice

Question 24

requires testtakers to supply or to create the correct answer, not merely selecting it

Answer 24

Constructed-Response Format

Question 25

Requires the examinee to provide a word or phrase that completes a sentence

Answer 25

Completion Item

Question 26

Should be written clearly enough that the testtaker can respond succinctly, with short answer

Answer 26

Short-Answer

Question 27

allows creative integration and expression of the material

Answer 27

Essay

Question 28

process of setting rules for assigning numbers in measurement

Answer 28

Scaling

Question 29

- involve classification or categorization based on one or more distinguishing characteristics - Label and categorize observations but do not make any quantitative distinctions between observations - mode

Answer 29

Nominal

Question 30

rank ordering on some characteristics is also permissible -median

Answer 30

Ordinal

Question 31

contains equal intervals, has no absolute zero point (even negative values have interpretation to it)

Answer 31

Ratio

Question 32

- has true zero point (if the score is zero, it means none/null) Easiest to manipulate

Answer 32

Interval

Question 33

- produces ordinal data by presenting with pairs of two stimuli which they are asked to compare - respondent is presented with two objects at a time and asked to select one object according to some criterion

Answer 33

Paired Comparison

Question 34

– respondents are presented with several items simultaneously and asked to rank them in order or priority

Answer 34

Rank Order

Question 35

– respondents are asked to allocate a constant sum of units, such as points, among set of stimulus objects with respect to some criterion

Answer 35

Constant Sum

Question 36

– sort object based on similarity with respect to some criterion

Answer 36

Q-Sort Technique

Question 37

– rate the objects by placing a mark at the appropriate position on a continuous line that runs from one extreme of the criterion variable to the other - e.g., Rating Guardians of the Galaxy as the best Marvel Movie of Phase 4

Answer 37

Continuous Rating

Question 38

– having numbers or brief descriptions associated with each category - e.g., 1 if your like the item the most, 2 if so-so, 3 if you hate it

Answer 38

Itemized Rating

Question 39

– indicate their own attitudes by checking how strongly they agree or disagree with carefully worded statements that range from very positive to very negative towards attitudinal object - principle of measuring attitudes by asking people to respond to a series of statements about a topic, in terms of the extent to which they agree with them

Answer 39

Likert Scale

Question 40

– a 100-mm line that allows subjects to express the magnitude of an experience or belief

Answer 40

Visual Analogue Scale

Question 41

– derive respondent’s attitude towards the given object by asking him to select an appropriate position on a scale between two bipolar opposites

Answer 41

Semantic Differential Scale

Question 42

– developed to measure the direction and intensity of an attitude simultaneously

Answer 42

Staple Scale

Question 43

– final score is obtained by summing the ratings across all the items

Answer 43

Summative Scale

Question 44

involves the collection of a variety of different statements about a phenomenon which are ranked by an expert panel in order to develop the questionnaire - allows multiple answers

Answer 44

Thurstone Scale

Question 45

the respondent must choose between two or more equally socially acceptable options

Answer 45

Ipsative Scale

Question 46

the test should be tried out on people who are similar in critical respects to the people for whom the test was designed - An informal rule of thumb should be no fewer than 5 and preferably as many as 10 for each item (the more, the better)

Answer 46

III. Test Tryout

Question 47

Risk of using few subjects = ______

Answer 47

phantom factors emerge

Question 48

A good test item is one that answered _________ by high scorers as a whole

Answer 48

correctly

Question 49

administering a large pool of test items to a sample of individuals who are known to differ on the construct being measured

Answer 49

Empirical Criterion Keying

Question 50

statistical procedure used to analyze items, evaluate test items

Answer 50

Item Analysis

Question 51

employed to examine correlation between each item and the total score of the test

Answer 51

Discriminability Analysis

Question 52

suggest a sample of behavior of an individual

Answer 52

Item

Question 53

a blueprint of the test in terms of number of items per difficulty, topic importance, or taxonomy

Answer 53

Table of Specification:

Question 54

Define clearly what to measure, generate item pool, avoid long items, keep the level of reading difficulty appropriate for those who will complete the test, avoid double-barreled items, consider making positive and negative worded items

Answer 54

Guidelines for Item writing

Question 55

items that convey more than one ideas at the same time

Answer 55

Double-Barreled Items

Question 56

defined by the number of people who get a particular item correct

Answer 56

Item Difficulty

Question 57

calculating the proportion of the total number of testtakers who answered the item correctly; The larger, the easier the item

Answer 57

Item-Difficulty Index

Question 58

__________ for personality testing, percentage of individual who endorsed an item in a personality test

Answer 58

Item-Endorsement Index

Question 59

The optimal average item difficulty is approx. _________ with items on the testing ranging in difficulty from about 30% to 80%

Answer 59

50%

Question 60

Omnibus Spiral Format

Answer 60

items in an ability are arranged into increasing difficulty

Question 61

provides an indication of the internal consistency of a test

Answer 61

Item-Reliability Index

Question 62

The higher ___________, the greater the test’s internal consistency

Answer 62

Item-Reliability index

Question 63

designed to provide an indication of the degree to which a test is measure what it purports to measure

Answer 63

Item-Validity Index:

Question 64

The higher Item-Validity index, the greater the test’s _________

Answer 64

criterion-related validity

Question 65

measure of item discrimination; measure of the difference between the proportion of high scorers answering an item correctly and the proportion of low scorers answering the item correctly

Answer 65

Item-Discrimination Index

Question 66

compares people who have done well with those who have done poorly

Answer 66

Extreme Group Method:

Question 67

difference between these proportion

Answer 67

Discrimination Index:

Question 68

correlation between a dichotomous variable and continuous variable

Answer 68

Point-Biserial Method

Question 69

graphic representation of item difficulty and discrimination

Answer 69

Item-Characteristic Curve

Question 70

one that eluded any universally accepted solutions

Answer 70

Guessing

Question 71

testtaker obtains a measure of the level of the trait; thus, high scorers may suggest high level in the trait being measured

Answer 71

Cumulative Model

Question 72

testtaker response earn credit toward placement in a particular class or category with other testtaker whose pattern of responses is similar in some way

Answer 72

Class Scoring/Category Scoring –

Question 73

compares testtaker’s score on one scale within a test to another scale within that same test, two unrelated constructs

Answer 73

Ipsative Scoring

Question 74

characterize each item according to its strength and weaknesses - As revision proceeds, the advantage of writing a large item pool becomes more apparent because some items were removed and must be replaced by the items in the item pool

Answer 74

IV. Test Revision

Question 75

revalidation of a test on a sample of testtakers other than those on who test performance was originally found to be a valid predictor of some criterion; often results to validity shrinkage

Answer 75

Cross-Validation

Question 76

decrease in item validities that inevitably occurs after cross-validation

Answer 76

Validity Shrinkage

Question 77

conducted on two or more test using the same sample of testtakers

Answer 77

Co-validation:

Question 78

creation of norms or the revision of existing norms

Answer 78

Co-norming

Question 79

test protocol scored by highly authoritative scorer that is designed as a model for scoring and a mechanism for resolving scoring discrepancies

Answer 79

Anchor Protocol:

Question 80

discrepancy between scoring in an anchor protocol and the scoring of another protocol

Answer 80

Scoring Drift

Question 81

item functions differently in one group of testtakers known to have the same level of the underlying trait

Answer 81

Differential Item Functioning

Question 82

test developers scrutinize group by group item response curves looking for DIF Items

Answer 82

DIF Analysis

Question 83

items that respondents from different groups at the same level of underlying trait have different probabilities of endorsing a function of their group membership

Answer 83

DIF Items:

Question 84

refers to an interactive, computer administered test-taking process wherein items presented to the testtaker are based in part on the testtaker’s performance on previous items

Answer 84

Computerized Adaptive Testing

Question 85

The test administered may be different for each testtaker, depending on the test performance on the items presented  Reduces floor and ceiling effect

Answer 85

Computerized Adaptive Testing

Question 86

occurs when there is some lower limit on a survey or questionnaire and a large percentage of respondents score near this lower limit (testtakers have low scores)

Answer 86

Floor Effects:

Question 87

occurs when there is some upper limit on a survey or questionnaire and a large percentage of respondents score near this upper limit (testtakers have high scores)

Answer 87

Ceiling Effects:

Question 88

ability of the computer to tailor the content and order of presentation of items on the basis of responses to previous items

Answer 88

Item Branching

Question 89

subtest used to direct or route the testtaker to a suitable level of items

Answer 89

Routing Test

Question 90

setting cut scores that entails a histographic representation of items and expert judgments regarding item effectiveness

Answer 90

Item-Mapping Method:

Question 91

– the level of which a the minimum criterion number of correct responses is obtained

Answer 91

Basal Level

Question 92

– standardized test administration is assured for testtakers and variation is kept to a minimum Test content and length is tailored according to the taker’s ability

Answer 92

Computer Assisted Psychological Assessment