Final Flashcards
Important information on groups is collected by which type of test?
Surveys
Outcomes or attributes are measured by what type of test?
Psychological test
What tests focus on individual outcomes?
Psychological tests
Results of a psychological test (individual) are reported at what level?
Test level, with overall scoring (high/low scores)
Results of surveys are reported at what level?
Question level, showing percentages per answered question
What are the 4 main steps to constructing, administering and using survey data?
- Preparing
- Pre-testing
- Administering
- Collecting and coding
What is a systematic examinations of published and unpublished reports/articles/studies on a topic?
A literature review
What are some important things to define when preparing for survey development?
Defining objectives, questions and plans.
Surveys and psychological tests must provide 3 sets of instructions for who?
- The one being tested
- The administrator
- The scorer
When developing administrator instructions, what must we take into account about each individual environment?
Testing environment
When developing instructions for the test takers, what should the developer assure about the test?
Participants are explained how to respond, questions are clear and concise, and ensure honesty
What instructions ensure that each person who scores the test will follow the same process?
The scoring instructions
Why should we write more questions than needed in the preparation phase of survey development?
You are likely to remove questions when pre-testing.
What type of language must you always avoid when developing questionnaires and surveys?
Slang and colloquial language
When preparing a survey, what face details are important?
Format, instructions and layout
When pre-testing a survey or psychological test, we should watch out for what type of measurement errors? Hint: errors
associated with the design and administration of the survey
Nonsampling measurement errors
When examining the amount of times a
question was not answered, what are researchers looking at?
Item non-response rate.
When administering a survey, a representative subset of the
population is known as a _____?
Sample (population)
What type of sampling uses statistics to ensure that a sample is
representative of a population?
Probability sampling
What type of sampling does not ensure
equal chance of being selected from the population?
Non-probability sampling
How can we code the Survey Question Responses, such as Likert scales?
We can assign numerical labels to add value to response choices.
Which form of presenting findings involves the use of research results, including dissemination, transfer, exchange, and co-creation or co-
production by researchers and knowledge users?
Knowledge mobilization
Defining the testing universe,
audience, and purpose; Developing a test plan; Composing the test items
; and Writing the administration
instructions are all a part of ____ development?
Test
Why should we be developing new tests if old tests already exist?
Needs are constantly evolving: behaviours change, some tests are no longer accurate, or do not properly evaluate what it is intended to
What helps define the testing universe of a survey or test?
Working definition
What identifiable measures help us most accurately define a target audience for a survey or psychological test?
Characteristics of a population
What a test will
measure, and how the test users will use the test scores defines the ____?
Test purpose
An _____ test question has one response that is designated as “correct” or that provides evidence of a specific construct.
Objective test
This test format does not have a single response that is designated as “correct.”
Subjective test
What is the most common model of scoring that determines an individual’s final test score? Eg. One point per “correct” answer
Cumulative model of scoring
What model of scoring places test takers in a particular group or class by looking for their pattern of scores (e.g., pattern of certain
symptoms = a diagnosis)?
Categorical model of scoring
What model of scoring scales items to distribute points, summing to a specific total?
Ipsative model of scoring
Conducting the pilot test and _____ are integral parts of the test development process.
Analyzing its data
What is the name for the scientific evaluation of the test’s performance?
A pilot test
Why do we call test items “items” rather than “questions”?
Questions are not always questions,
they can be statements, pictures, or
incomplete sentences.
What are 2 types of objective test items?
Multiple choice
Forced choice
What are super common, objective test items used for a variety of purposes including preemployment tests, standardized achievement tests, and classroom tests? These may include stems and distractors.
Multiple choice questions
What objective test items presents the test taker with little room for response variety? Eg. “this is more like me” OR “this is less like me”
Forced choice questions
Name 3 subjective test items
Essay questions
Interview questions
Projective techniques
Which subjective testing items are often lengthy and written?
Essay questions
Which subjective testing item involves verbal conversation and allows for follow-up questions and exploration of additional relevant topics?
Interview questions
What techniques involve using highly ambiguous stimulus to elicit an unstructured response? E.g., show a child a picture, ask them to describe it
Projective techniques
What refers to the tendency of test takers to respond inaccurately to questions?
Test bias
What patterns of responding can result in false or misleading
information?
Response sets
What is the term for the tendency of some test takers to provide or choose
answers that are socially acceptable or that present themselves in a favourable
light?
Social desirability bias
What is the name for the tendency to agree with any ideas or behaviours presented on test items?
Acquiescence
What is the name for responding to items in a random way by marking answers, without reading or considering them?
Random responding (random response patterns)
What is the act of answering items in a way that will cause a desired outcome or diagnosis?
Faking
What are 4 elements that may contribute to response bias?
Social desirability bias
Acquiescence
Random responding
Faking
What is the term for evidence based on
content?
Validity
What is the term for logically examining and evaluating the content of a test (including the test questions, format, wording, and tasks required of test takers) to (1) determine the extent to which the content is representative of the concepts that the test
is designed to measure … (2) without including elements that are irrelevant to their measurement?
Evidence based on test content
Why must we use concise and exact language when writing tests and test items?
Brevity reduces errors.
When writing effective test items, you should ensure to include the following 8 things:
Brevity
Complete sentences
Relevant and realistic time periods
Accessible language (avoid jargon)
No leading questions
Avoid double barrelled questions
No double negatives
Avoid assumptive questions
What is the name for a question where biased language has the effect of pushing a test taker toward a particular answer option?
Leading questions
What is the name for a question that assumes that the taker is already familiar with something and disregards the possibility that the test taker
may not be familiar with the concept?
Assumptive questions
How can developers evaluate the
performance of each test item? This is important during pilot testing
Item analysis
How can we calculate item difficulty?
By dividing the number of persons who answered correctly by the total number of persons who responded to the
question
If an item difficulty scores at .5 yield, what does this mean?
There is a lot of variation in responses (pvalue)
This can be calculated for tests of personality, and shows the percentage of test takers who
respond correctly.
Item difficulty
What index compares the
performance of those who obtained very high test scores
(the upper group) with the performance of those who
obtained very low test scores (the lower group) on each
item?
Discrimination index
The discrimination index will range from ____?
-1.0 to +1.0
True or false: As a general rule the more positive the discrimination index the better the quality of the question.
True
Which of the following two discrimination index examples involve a better test? A DI of 0.6 or -0.1?
0.6
What is a measure of the strength
and direction of the relation between the way test takers responded to one item and the way they responded to all of the items as a whole
Item total correlation
What would a low item-total correlation signify?
We should probably drop the question
What matrix displays the correlation of
each item with every other item?
Inter-item correlation matrix
What measures the correlation of item responses with a criterion measure?
Item criterion correlation
True or false: Criterion validity asks: is the test related to other
tests?
True
What is the name for a measure of the relationship between individuals’ performances on one test item and the
test takers’ levels of performance on the overall measure of the construct the test is measuring?
Item response theory
What is the name of the theory that lets us relate how a test taker did on each individual item to a statistical estimate of the test taker’s ability on the construct being measured?
Item response theory
How can quantitative item analysis be done at an individual level?
Test takers are given
a survey/questionnaire about the actual items
How can quantitative item analysis be done at a group level?
An expert panel is
convened where they may provide feedback on the items
Why is it important to revise the test you just came up with?
You want to ensure that items are good by scoring well on the Item Statistics Matrix
What is a validation study?
To establish evidence of validity based on test content.
What is the purpose of a validation study?
To make sure the test can provide meaningful results.
What is it called to administer the same test to another sample of test takers from the target audience?
Replication
What is the replication crisis in psychology?
We are finding that
not a lot of studies actually replicate… This means many theories and findings (and measures!) we believe are
“valid” may not be
What is the statistical measure that expresses the extent to which two variables are related? This often shows relationships are measured in a linear way, meaning that they change
together at a constant rate.
Correlation
“What does the one item contribute to the overall test” is a great example of what theory?
Item response theory
What are the 5 components of quantitative item analysis?
Item Difficulty
Item Discrimination
Item-Total Correlation
Inter-Item Correlation
Item-Criterion Correlation
What is the name for an interview collection method that involves a rgid set of questions, interviewer cannot
deviate from questions or ask follow-ups? Often delivered in a
standardized way
Structured interviews
What is the name for an interview collection method that involves informal, free-flowing questioning? The interviewer may have a general guide but is free to go in any direction
Unstructured Interviews
What is the name for an interview collection method that is more open than structured interviews? Interviewer may ask follow-up questions
Semi-Structured Interviews
What is a probing question?
To follow-up where you seek more detail
What is a prompt in semi-structured interviews?
To follow-up where you give them more info to help them answer
Who mainly oversees ethics of
psychological testing?
Psychological associations
Why is the Canadian Code of Ethics for Psychologists meant as a guide?
Ethical decisions are complicated
What are the 4 principles of the Canadian Code of Ethics for Psychologists?
- Respect for the Dignity of Persons and Peoples.
- Responsible Caring.
- Integrity in Relationships.
- Responsibility to Society
What is the name for a professional credential individuals earn by
demonstrating that they have met predetermined qualifications?
Certification
What is the name for a mandatory credential individuals must obtain to
practice within their professions?
Licensure
How do governing bodies enforce their ethical codes of
conduct?
Licensure can be revoked or suspended.
What are some responsibilities of test publishers?
Ensuring professionalism and ethics
Attention to distribution
Providing user manuals
What is the name for a person who responds to test questions or whose
behavior is measured or observed?
Test taker
What are some responsibilities of test takers?
Must understand the consequences of their decision to take
the test (or not take it), must ask questions if anything is
unclear, must protect test security.
What are 4 test taker rights?
Issue 1: Right to Privacy
Issue 2: Right to Informed Consent
Issue 3: Right to Know and Understand Results
Issue 4: Right to Protection From Stigma
How can groups can further be minoritized by testing? e.g., if the test doesn’t take into account unique
cultural conditions that may impact results
Marginalization
What is the job of a psychometrist?
To test, score and analyze psychological tests.
To truly be consent, it must be:
Free, informed and ongoing
What do we call when prospective participants are recruited by individuals in a position of authority or otherwise pressured to
participate?
Undue influence
What is a more extreme form of undue influence, involving a threat
of harm or punishment for failure to participate or remain in the project?
Coercion
What is the name for anything offered to participants, monetary or otherwise,
for participation in research?
Incentives
What could be a consequence of breaking confidentiality in research?
Institutional consequences (e.g., losing job at uOttawa)
What could be a consequence of breaking confidentiality in clinical contexts?
Losing/suspending licensure
What to do when there is a breach in proper consent or confidentiality procedures?
We must follow pre-determined steps, often under supervision of a superior or
in consultation with a colleague
What is the main goal of Research Ethics Boards?
To review the ethical acceptability of all research involving humans
What do REBs review?
Research conducted by
faculty, staff or students, or
members of the institution (e.g.,
hospital, university)
Who is on a REB?
- two members must have relevant expertise;
- one member is knowledgeable in ethics;
- one member is knowledgeable in the relevant
law; - one community member has no affiliation with the institution.
How many people should be on a REB to ensure competent, independent review?
REB must consist of at least 5 members
What is the name of forms that explain study purpose, risks, benefits, how confidentiality will
be protected, how data will be
conserved, and any
compensation?
Consent forms
How does application
review work for a study?
An REB will consider the possible level or risk
associated with possible ethical issues and then
choose an appropriate level of review
How do REBs ensure that researchers will stick to the plan that has been approved?
Modifications require reapplication and review
What is open science?
A movement to make
research transparent,
accessible, verifiable, by
anyone
True or false: Open access articles tend to get cited less.
False. They are cited more.
