Test Construction and Factor Analysis

Question 1

What are the 6 steps in test construction?

Answer 1

1. Defining the Test’s Purpose
2. Preliminary Design Issues
3. Item Preparation
4. Item Analysis
5. Standardization and Ancillary Research
6. Preparation of Final Materials and Publication

Polly Pocket Is A Silly Face

Question 2

What is involved with defining a test’s purpose?

Answer 2

Statement of purpose: traits to be measured and target audience (e.g., “the WMS-R is an individually administered clinical instrument for appraising major dimensions of memory functions in adolescents and adults”)

Question 3

What are the 6 components of preliminary design issues?

Answer 3

1. Background research
2. Mode of administration
3. Length
4. Number of scores
5. Question and response format
6. Administrator training

Rafael Ate Lard Sandwiches Timidly

Question 4

What is involved in background research (preliminary design issues)?

Answer 4

Literature search (theoretical and empirical), books. Ask yourself how have others defined the construct? Which assessments are already available? Subject matter experts: interview with individuals who study the phenomenon you are interested in, people who have specific knowledge about the construct, include different constituent groups

Question 5

Describe the mode of administration options (preliminary design issues).

Answer 5

group vs individual; pen and paper, scantron, online, oral

Question 6

What is the tradeoff involved with test length (preliminary design issues)?

Answer 6

tradeoff between reliability and efficiency, number of scores (constructs X dimensions within constructs)

Question 7

What are the different types of response format (preliminary design issues)?

Answer 7

self-report, informant rated, expert rated

Question 8

What are the different ways a question can be formatted (preliminary design issues)?

Answer 8

1. Dichotomously scored
2. Likert scored
3. Forced/multiple choice
4. Graded response options
5. Ranking
6. Visual Analog Scale
7. Open format
8. Perfomance assessment
9. Semi-structured interview

Question 9

What does it mean for a question to be dichotomously scored?

Answer 9

(Woodworth in 1915 with Army Recruits; little effort on answerer, do not distinguish differences among people who answer yes or no; require little effort on the part of the person answering questions; do not distinguish differences among the people who answered yes or no; e.g. are you currently depressed yes vs no)

Question 10

What does it mean for a question to be Likert scored?

Answer 10

(most widely used for assessing attitudes, beliefs, and feelings; agree or disagree on 5, 7 or 9 point scale; response options are typically symmetric or balanced because there are equal numbers of positive and negative positions; extended to assess frequency, importance, quality, and likelihood; a recent empirical study found that tiems with five or seven levels may produce slightly higher mean scores relative to the highest possible attainable score, compared to those produced form the use of 10 levels)

Question 11

What does a multiple choice question consist of? What is another name for it?

Answer 11

(typically used to test knowledge and ability; usually 4 or 5 responses, only one correct)

Question 12

What is a graded response option question?

Answer 12

options (every option from any test question, in which options are viewed as progressively more severe [e.g., strongly agree vs agree]; some tests utilize options that are more explicit, such as Beck Depression Inventory and IQ tests)

Question 13

What is a ranking question?

Answer 13

(respondents order elements from a group of objects, activities, characteristics, or conditions; advantage of forcing people to choose one over another—prevents ties among items)

Question 14

What is a visual analog scale?

Answer 14

(VAS is a method of capturing the degree or amount of some condition or attitude an individual has without the use of explicit numbers; described in 1921 and referred to as graphic rating method; actually measure distance)

Question 15

What are components and concerns when using an open format question?

Answer 15

(benefit of not controlling response; risk of not asking about certain aspects of the construct being assessed that may be important; allows respondent to provide any information they wish; must be rated by researchers using a scoring key)
*constructed response item

Question 16

What is a performance assessment and why would it be used?

Answer 16

(critically important to specifically define the target behaviors; identify the antecedent conditions that trigger the behavior; SME’s review the behaviors to make sure you are measuring the construct, e.g., customer service: sales associate must greet customer within 30s of customer entering the store)
*constructed response item

Question 17

Why do psychologists use semi-structured interviews? What are the two variance factors to consider when making these questions?

Answer 17

designed to help clinicians comprehensively assess the presence and absence of psychiatric symptoms; facilitate decisions about whether or not there were sufficient symptoms and impairment to render a formal diagnosis; designed to control information and streamline decision making:

1. Information variance (i.e. what information is collected and considered in deciding whether or not someone was depressed)
2. Criterion variance (i.e. how symptoms would be combined to reach a diagnosis). The goal is to ensure that every person making a diagnosis would ask exactly the same question about all of the relevant symptoms. Assumptions is that diagnoses are discrete entities—either you have it or you don’t.

Question 18

What are the two types of test items?

Answer 18

1. Selected-response times (e.g., T/F, multiple choice, Likert format)
2. Constructed-response items (open format [essay or oral responses] and performance assessments)

Question 19

What are some benefits of selected-response test items?

Answer 19

1. Scoring reliability
2. Temporal efficiency
3. Scoring efficiency

Question 20

What are some benefits of constructed-response items?

Answer 20

1. Behavioral observation
2. Exploring/explaining on responses
3. Development of study habits

Question 21

What are some considerations when training administrators (preliminary design issues)?

Answer 21

Scoring, Behavior assessment, Interviewing

Question 22

What are the four parts of a test considers in the item preparation step of test construction?

Answer 22

1. Stimulus
2. Response
3. Conditions governing responses
4. Scoring procedures

Question 23

What are some guiding principles when writing items?

Answer 23

deal with ONE central thought in each item, be precise, be brief, avoid awkward wording or dangling constructs, avoid irrelevant information, present items in positive language, avoid double negatives, avoid terms like all or none, avoid indeterminate terms like frequently or sometimes

Question 24

What are some item preparation recommendations?

Answer 24

prepare at least 2-3 times as many items as you intend to have. Review: spelling and grammar, content review, beware of stereotype

Question 25

What are the four ways to analyze an item?

Answer 25

1. item tryout
2. statistical analysis (item difficulty, item discrimination, distractor analysis)
3. factor analysis
4. item selection

Question 26

What are the two characteristics a perfect item has?

Answer 26

1. People who know the answer would always choose the right answer
2. People who do not know the answer would chose randomly among the possible responses

Question 27

What is item difficulty? How do you calculate it?

Answer 27

the percentage of students who took the test who answered items correctly; Item Difficulty (p) = # people correct/total

a p value is a behavioral measure, difficulty is a characteristic of the item and the sample, extreme p values restrict variability

Question 28

What is item discrimination? What are the two indices that determine discrimination?

Answer 28

determines how well that a single item on the test is measuring the same things the test itself; don’t want it too high, otherwise repetitive of there items; don’t want it 0 or negative because then it undermines the test
two indices to determine discrimination
1. Item discrimination index D
2. Discrimination coefficients

Question 29

How do you calculate item discrimination?

Answer 29

score and rank order; take 27% from both ends of distribution;
D=(# correct upper - # correct lower) / # people in largest group

Question 30

What does a D score tell us?

Answer 30

0. 40+=Good; 0.30-0.39=Reasonably good; 0.20-0.29=Marginal; 0.19 and below=poor
   e. g., 7 people in an exam were the lowest 27% lowest scorers, and 27% of high scorers had 8 people. On one item, only 3 of lower scores got it right, and 7 of the high scores got it right: 7-3/9 (of the leftover people, 100%-54%)

Question 31

How do you know if a distractor is good and what are the guidelines when writing them?

Answer 31

obtain discrimination index for each option to determine usefulness of distractor: should be low and preferably negative; be cautious of large D values as well

create distractors that are plausible; make all of the alternatives parallel in length and grammatical structure; keep the alternatives short; don’t write distractors that mean the same thing; alternate the position of the correct answer within the distractors; use the alternatives “all of the above” and “none of the above” as little as possible; make sure each alternative agrees with the stem

Question 32

What are the three core questions of dimensionality considered in factor analysis?

Answer 32

1. How many psychological attributes (i.e., dimensions) are reflected in the test’s items?
2. If a test’s items reflect more than one dimension, then are those dimensions correlated with each other?
3. If a test’s items reflect more than one dimension, then what are the dimensions?

Question 33

What are the three levels of dimensionality?

Answer 33

1. Unidimensional tests: conceptual homogeneity; one single score;
2. multidimensional tests with correlated dimensions: test with higher order factors and a variety of scores also have a total score which is the subtests combined with each other;
3. multidimensional tests with uncorrelated dimensions: no total score is computed

Question 34

Why is dimensionality important?

Answer 34

implications for appropriate scoring, evaluation, and interpretation of test scores; for example, the number of dimensions (each dimension should be scored separately [one person might receive more than one score from a test]; each such score requires its own psychometric evaluation; each score would be interpretable in terms of the psychological dimensions underlying the score)

Question 35

What is factor analysis? What are the two types?

Answer 35

a statistical procedure often used to evaluate a test’s dimensionality 1. Exploratory FA 2. Confirmatory FA

Question 36

What is the purpose of EFA?

Answer 36

EFA is generally used to discover the factor structure of a measure and to examine its internal reliability; EFA is often recommended when researchers have no hypotheses about the nature of the underlying factor structure of their measure

Question 37

What are the three basic points in EFA?

Answer 37

1. Decide the number of factors 2. Choosing an extraction method 3. Choosing a rotation method

Question 38

What are eigenvalues? How are they produced?

Answer 38

The total variance explained by a particular factor; produced by a process called principal components analysis (PCA) and represent the variance accounted for by each underlying factor; they are not represented by percentages but scores that total to the number of items (12-item scale with theoretically have 12 possible underlying factors, each factor will have an eigenvalue that indicates the amount of variation in the items accounted for by each factor; if the first factor has a eigenvalue of 3.0, it accounts for 25% of the variance (3/12=0/.25). The total of all the eigenvalues will be 12 if there are 12 items, so some factors will have similar eigenvalues)

Question 39

How do you decide the number of factors?

Answer 39

the most common approach to deciding the number of factors is to generate a scree plot (two-dimensional graph with factors on the x-axis and eigenvalues on the y-axis); on a scree plot, the eigenvalues are typically arranged in a scree plot in descending order

Question 40

What is a scree plot?

Answer 40

two-dimensional graph with factors on the x-axis and eigenvalues on the y-axis, eigenvalues usually arranged in depending order

Question 41

What is the Kaiser-Guttman Rule?

Answer 41

it states that the number of factors are equal to the number of factors with eigenvalues greater than 1.0; this approach is often not recommended because it tends to produce many factors

Question 42

What is factor extraction (EFA)?

Answer 42

once the number of factors are decided, the researcher runs another factor analysis to get the loadings for each of the factors; to do this, one has to decide which mathematical solution to use to find the loadings. There are about 5 basic extraction methods; regardless of the method chosen, the factor extraction will produce factor loadings for every item on every extracted factor; researchers hope their results will show a simple structure, with most items having a large loading on one factor but small loadings on other factors; Loading: how well an item correlates with a factor/dimension of construct

Question 43

What is a factor loading?

Answer 43

how well an item correlates with a factor/dimension of construct

Question 44

What is rotation? (EFA)

Answer 44

once initial solution is obtained, the loadings are rotated (it is a way of maximizing high loadings and minimizing low loadings so that the simplest possible structure is achieved; there are two basic types 1. Orthogonal (varimax, quartamax, equamax) assumes factors are not correlated 2. Oblique (oblimin, promax, direct quartimin) doesn’t make any assumptions

Question 45

What are the two basic types of item rotation?

Answer 45

1. Orthogonal (varimax, quartamax, equamax) assumes factors are not correlated
2. Oblique (oblimin, promax, direct quartimin) doesn’t make any assumptions

Question 46

How are EFA and CFA similar?

Answer 46

Both EFA and CFA are used to investigate the theoretical constructs, or factors, that might be represented as a set of items; either can be assumed the factors are uncorrelated, or orthogonal; both are used to assess the quality of individual items

Question 47

How are EFA and CFA different?

Answer 47

1. With EFA, reserachers usually decide on the numbers of factors by examining output from a principal components analysis (i.e., eigenvalues are used). With CFA, the researchers must specify the number of factors a priori
2. CFA requires that a particular factor structure be specified, in which the resracher indicates which item load on which factor. EFA allows all items to load on all factors
3. CFA provides a fit of the hypothesized factor structure to the observed data
4. in CFA, researchers typically use maximum likelihood to estimate factor loadings, whereas maximum liklihood is only one of a variety of estimators used with EFA