Week 3 Flashcards
Overarching theory of psychometric theory
Is classical testing theory
Classical testing theory
X is equal to t plus e
X (observed score), t (true score), e (error)
Content analysis
Getting content from individuals who are actually affected by the product you are showing
Critical incidents
Interviewing individuals about the Katrina hurricane, listening to the responses people saying there is not enough canned goods, these types of interviews can be helpful to future incidents
Direct observations
Directing peoples behavior
Step 2: Identifying behaviors that represent a construct
Content analysis, critical incidents, direct observation & expert judgments
Step 3 preparing the test specifications
Need to develop specific sub-areas that make up the larger construct and test developer needs to assign weights to each sub-area
Step 4- item construction
Select an item format that is feasible for the intended examinees and need to write appropriate items-also consider how long you want the test to be in terms of time
Aptitude and achievement test use what type of answer choices
Use true or false, multiple choice and matching
Attitude and personality traits answer choices
Agree-disagree, likert format, bipolar checklist
Step 1 identifying the purpose of the test
Need to identify both the material to be tested and for whom you are developing the test
Step 5 item review
Experts review items for accuracy grammar, appropriateness to the construct
Step 6 preliminary item piloting
Give test out a small group and debrief about test to determine points of difficulty
True score
Testing someone an infinite number of times
Two types of error
Random and systematic error
Random error
It doesn’t happen the same every time. Any error due to chance
What is chance based on?
Probability, which is based on infinity
Systematic error
Any error that reoccurs repeatedly based on the characteristic of the person being tested
Person R correlation
Association between two variables that are measured on interval and continuous scales
Variance
Variability (area)
Anything with an exponent its root is always
Variance =X^2
Non dichotomous scoring system variance
(X1-X)^2 + (X2-X)^2+…. Divided by N
Correlations matrix
Shows how related the items are to one another
0-.3 weak, .4-.6 moderate & .7 and above strong correlation
Variance item for dichotomous
Item mean times (1-item mean)
Phi formula
2 variables association on a categorical or nominal scale
Phi coefficient goes from what numbers
0 to 1
Phi formula actual formula
bc-ad divided by the square root of (a+b)x(a+c)x(c+d)+(b+d)
Which items go first on the phi formula
The lower number item always goes on top of the box and the higher number item goes on the left side
Where do you look when beginning the phi formula?
Make a square and look for the raw scores for the items compare them to one another and plug them into the formula
Item quality
Determining how good an item is and if we need to take them out, look at which items have better quality
Is the scale project non dichotomous?
Yes
Dichotomous response formats
Look at aptitude/achievement, either wrong or right answer, looks at variance, mean, phi coefficient, variance and covariance matrix
Non dichotomous response format
Attitude/personality, likert scale, look at mean, variance & correlation matrix
Non dichotomous variance you want items to be
Higher variance are better quality items! If you want to capture trait you need higher variance/variability
How do we determine a non dichotomous item to be appropriate?
The way we determine the item is appropriate if the mean is closest to the midpoint on a scale
Mean
Average responding for your sample for that item and denotes strength of sentiment
Variance related to standard deviation
Standard deviation ^2 is variance
Mean for dichotomous
The higher the mean the easier the question, the lower the mean the more difficult the question
Variance in dichotomous
There should be low variance between groups because there are only 2 options, we do not want variability we want similar variance scores, variance should be closest to zero
Do the negatives matter in variance-covariance matrix item scores?
The negative does not have a qualifier here
Covariance
How the items vary with each other
Two types of scoring
Dichotomous & Non-dichotomous
