Topic 5: Test Conceptualization Flashcards
___ an umbrella term for all that goes into the process of creating a test.
Test Development
Test Development: The process of creating ___ or ___ to measure specific constructs or traits in individuals.
assessments or tests
Reasons for Test Development (3)
- Need for research on a topic, but no extant measure suitable to carry out the research
- Diagnosis, assessment, employee selection
- Absence of suitable tests or inadequacies in existing ones
Stages of Test Development
- Test Conceptualization
- Test Construction
- Test Tryout
- Test Analysis
- Test Revision
The initial stage of test development where the idea for a test is conceived—defining of the construct and setting the direction for subsequent stages.
Test Conceptualization
Involves writing, revising, or re-writing test items, formatting items, setting scoring rules, and designing the test.
Test Construction
Administration of preliminary test to a representative sample of test-takers.
Test Tryout
Analyzing test-takers’ performance of the test items to determine which items are effective and which ones need revision or removal.
Test Analysis
Modification of test’s content or format based on the results of item analysis and feedback from the test tryout.
Test Revision
What is/are the purpose of test conceptualization?
- To clarify what the test aims to measure, why it matters, and how it will be implemented.
- To eradicate the possibility of confusion that arises when the key terminologies are perceived differently.
Significance of Test Conceptualization
- Guiding Development
- Face Validity
- Adapting to Change
Significance of test conceptualization, in which it guides every aspect of test development—from item generation to interpretation of results.
Guiding Development
A well-conceptualized test ___ of an assessment by ensuring that it measures what it intends to measure.
enhances (the) validity
A ___ facilitates the selection of appropriate items and criteria for scoring, minimizing construct-irrelevant variance.
clear conceptualization
Test conceptualization enables test developers to ___ societal needs and emerging trends.
adapting to change (or evolving)
Stimulus that begins any published test.
- Review of the available literature on existing tests
- Emerging social phenomenon or pattern of behavior
- Response to a need to assess mastery in an emerging occupation or profession
3 circumstances that lead to the development of successful tests
- Theoretical Advances
- Emperical Advances
- A Practical (Market) Need
Result if the test developer created a test that is not based on the 3 circumstances and caused by the duplication of tests.
Construct Proliferation and Jingle-Jangle Fallacy
A phenomenon wherein researchers come up with constructs thinking of these as “new” or “unique,” but are deemed theoretically or empirically indistinguishable from existing constructs.
KEY STATEMENT: REDUNDANCY OF CONSTRUCTS
Construct Proliferation
Occurs when a term or a word is used to refer to two different things.
Jingle Fallacy
Occurs when two different words are used to refer to the same thing.
Jangle Fallacy
Difference between Jingle Fallacy and Jangle Fallacy?
- Jingle F.: word used to refer to two different things
- Jangle F.: two different words are used to refer to the same thing
___ are often driven by prior empirical discoveries.
Theoretical advancements
Reason for the development of many tests and give rise to new constructs, involves the integration of theories, and modification of existing models.
Theoretical and Empirical Advances
Can existing frameworks be used for theoretical framework?
Yes, it can be used as long as current tests have not yet exploited these frameworks. (TF must be new or more logical than previous fw)
Interaction between theoretical and empirical advances, and it serves as a precursor to develop a systemic taxonomy of the domain.
Systemic Domain Mapping
Consists and identifies all primary and higher-level constructs that can be mapped onto a theoretically supported structure.
Systemic Domain Mapping
Collecting data on so many items is difficult that results in ___ of the data. (Limitations of systemic domain mapping)
burdensome analysis
Analyzing ___ requires the division of the total domain into manageable chunks. (Limitations of systemic domain mapping)
large groups of data
___ is vulnerable to errors of not being able to sufficiently capture some constructs. (Limitations of systemic domain mapping)
Grouping items
Not all ___ or ___ derived measures correspond to a practical need.
taxonomically, theoretically
Refer to tests containing many items, it is also often long and time-consuming, and results in long and detailed reports.
Tests with omnibus measures
Tests with omnibus measures that rarely correspond to a direct market need.
Woodcock-Johnson IV and NEO PI-R
Relatively short (user-friendly) and results are easily communicated and understood leading to practicality.
Myers-Briggs Type Indicator (MBTI)
For a test to address a market need, it should be…
- Technically sound in terms of theoretical grounding and psychometric properties
- Practically useful
Its main purpose is to obtain unbiased and correct parameter estimates for measurement artifacts, and it averages the value of an effect size across studies for a reliable summary estimate.
Meta-Analysis
The most important effect size in a selection context
Predictive validity
Limitation of Meta-Analysis
Can only be applied where there is already a substantial body of research.
Importance of Preliminary Questions
To help clarify, ensure an effective measure, identify, distinctively define, determine, establish, and serve as foundation
What is the 1st step in test conceptualization?
Describing the purpose and rationale for atest
What is the 2nd step of test conceptualization?
Describing the target population for the test
Relevant characteristics of target population
Age, educational status, language, literacy level, and disabilities
What is the 3rd step of test conceptualization?
Clearly defining the key variable of interest
Reference for key variable of interest
Psychological theories, empirical literature, cultural definitions, DSM-V or other diagnostic criteria
What is the 4th step in test conceptualization?
Create item specifications
Help define significant characteristics of items developed, and serves as a guide.
Item specification
Types of instruments (5)
- Self-report
- Performance assessment
- Observation schedule
- Checklist
- Examination/test
Participants report their own demographic characteristics, attitudes, beliefs, knowledge, feelings, and behavior
Self-report
Forms of self-report (2)
Questionnaire and Interview
Advantages of questionnaire
- Easy to administer and score
- Can be administered to larger numbers of test takers
Disadvantages of questionnaire
- Potential for biased responses
- Difficulty in interpreting open-ended questions
- Data security and privacy concerns
Advantages of interview
- Rich and in-depth information
- Expanding on unclear responses
- Can be used with any test-taker (disabilities)
Disadvantages of interview
- Takes time
- Subject to issues of inter-rater reliability
Types of items for self-report
Open-ended and close-ended
Test-takers give their own response
Open-ended
Advantage of open-ended
in-depth information
Disadvantages of open-ended
- Literacy skills is needed
- Subjective scoring
- Inter-rater reliability issues
- High chances of skipping items
Test-takers select their response from given categories on the test
Close-ended
Advantages of close-ended
- Easy and objective scoring
- Clear response
Disadvantages of close-ended
- Most common responses must be anticipated
- Response categories that is understandable to the population
- Response categories must be exhaustive
In which learner demonstrates his or her knowledge and understanding of concepts, and skills and technical abilities by producing something
Performance assessment
Analytical form/coding sheet, filled out by researchers during structured observations; Under specific conditions.
Observation schedule
Identify the frequency or presence of behaviors or characteristics
Checklist
Test participants’s knowledge of a topic
Examination or test
What is the 5th step in test conceptualization?
Choose item format
Grouping of words, statements, or symbols where the test-taker indicates judgments regarding the intensity of a specific trait, attitude, or emotion
Rating scale
Test participants are given pairs of stimuli and are asked to compare them and select one: Useful to place them into categories and avoid agreeing-disagreeing every time.
Paired comparison
It identifies how often behaviors or traits occur or whether they are present.
Checklists
Uses a semi-structured interview to identify the presence or absence of psychopathological symptoms.
Hare psychopathy checklist
Directly assessing performance on tasks
Performance Assessment
Standardized exams for the purpose of ranking and comparing test takers to one another.
KEY NOTE: DETERMINING POSITION IN COMPARISON TO OTHERS WHO ARE SIMILAR TO THEM
Norm-referenced test
Advantages of norm-referenced test
- Simple, affordable, and easy to administer
- Can distinguish between high and low performance in a larger group
Disadvantage of norm-referenced test
Cannot accurately assess the test-takers’ actual knowledge or skill level
Evaluates the person’s knowledge and abilities in relation to a predefined standard, learning objective, performance level, or other criterion.
Criterion-referenced tests
Advantages of criterion-referenced test
Specificness and subject proficiency
Disadvantages of criterion-referenced tests
- Costly and time consuming
- Cannot give a complete picture of a student’s abilities in relation to others
The initial study conducted in connection with the development of the test prototype and finding the most effective way to measure a specific component.
Pilot work or pilot study or pilot research
Importance of pilot work (3)
- Give important details on potential problems
- Find out moe about the potential expenses
- Assess the viability of your research
