HowToDesignAResearchStudy Flashcards
What are the steps to designing a research design? (How Venerial Disease Overtook My Sexy 3 Aunts Royally)
- Hypothesis
- Variables
- Design
- Operationalizations
- Manipulation
- Setting and Participants
- 3rd Variables (Mediators and Moderators)
- Analyze Data
- Report Results
Hypotheses should be worded as ________ and should make explicit _____ between groups
statements
comparisons
What are the 8 threats to internal validity
- Maturation
- History
- Instrumentation
- Regression Toward the Mean
- Testing
- Selection
- Mortality
- Interaction of Selection x Maturation
Hypotheses should be worded in the past/present/future tense
present or future tense
Describe the internal validity threat: Selection
- Differences in posttest scores exist because of preexisting group differences
- possible unequal distribution of subject-related variables (e.g., sex, race, age, intelligence)
Hypotheses can/cannot be proven
You cannot prove a hypothesis
Describe the internal validity threat: History
- simultaneously occurring outside events may influence subjects
- Events that take place during the course of research that influence the posttest scores
If you include interactions as part of your hypothesis, you need to be explicit about each experimental condition in the interaction. Give an example.
People who work in homogenous groups will perform better than people who work in heterogenous groups
Describe the internal validity threat: Maturation
- Naturally occurring changes in person over time that influence the posttest scores
- Permanent: biological growth
- Temporary: fatigue
What question does the Moderator answer
Answers the question when or for whom does the IV have an influence on the DV
Describe the internal validity threat: Repeated Testing
- Ss may become sensitized
- pretest measurement of the DV may affect the results obtained from subsequent measurements
What question does the Mediator variable answer
- Answers the question how or why does the IV influences the DV
- often a variable that describes an internal psychological mechanism
- Often introduced when there is already a strong relationship between IV and DV
Describe the internal validity threat: Instrumentation
Changes in the way pretest and posttest scores were measured may cause changes in outcome
What should you include when discussing the design of a study
- Name of the design (be specific)
- why it is the best design
- flaws or limitations and how you will attempt to mitigate them
- determine if btwn or within Ss design (state why)
- discuss feasibility (economically, logistically, ethically)
- discuss limitations of feasibility
- design is usually a 2x2 factorial or 2 btwn-Ss factors
Describe the internal validity threat: Regression to the mean
Ss with extreme scores on the pretest will tend to have scores closer to the mean on the posttest
Why do you want to use a 2x2 factorial design
- Economy: the design provides more information from the same amount of work
- Experimental control
- Increases Generality of the results
- A way to investigate interactions among Ivs
*the factorial design allows us to investigate more realistic situations
Describe the internal validity threat: Mortality
Loss of Ss during the experiment that is different btwn the experimental and control groups (e.g., dropping out)
*Differential mortality cannot be ruled out by random assignment
Why is it important to test the interaction of Ivs
Because the effect of an IV rarely occurs in isolation
In the real world, many variables operate simultaneously
Describe the internal validity threat: Selection x Maturation
Differences between groups that cause changes in the groups at different rates
Independent Variable
A manipulation to which participants are randomly assigned (if possible)
What is external validity
The extent to which results fo a study can be generalized from the sample to the population or other settings, samples, and variables
If you cannot manipulation the IV, what types of studies can you do?
quasi-experimental or correlational
Threats to external validity are anything that makes the experimental situation ____
unique
For all levels of the IV, you need to _____
identify the operationalization (even for “absence”)
*be specific (e.g., give quotes of what experimenters will say, if relevant)
_____ experiments have lower external validity; ____ experiments have higher external validity
true, field
Why is it important to run a pilot-test
- Determine if manipulations (IV) have intended effect (i.e., did you manipulate the IV effectively?)
- to find a balance between impact and control
What is ecological validity
whether the research conditions resemble at all the way humans behave in the real world
why is it important to include a manipulation check
to determine if manipulations (IV) have intended effect
What are the 5 threats to External Validity
- Selection x Treatment
- History x Treatment
- Interferences from Multiple Treatments
- Testing x Treatment
- Reactive Arrangements
You should specify what type of data will be produced by the IV.
True/False
True.
Describe the External Validity threat: Selection x Treatment
results are only generalizable to the specific sample studied, not to the general population (e.g., resarch volunteers tend to be different from non-volunteers)
When operationalizing the IV, what should you do to reduce systematic error
- Carefully operationalize variables so that extraneous variables don’t occur with the IV
- Randomly assign subjects to conditions
Describe the External Validity threat: History x Treatment
Because of events going on in the world, the results of the study can’t be generalized to every day life (e.g., doing a study during wartime)
What is the dependent variable
The variable being measured; the variable used to assess the affected construct
Threats to External Validity: What are the three Interferences from Multiple Treatments (in within-subjects designs)
- Carryover effects
- Order Effects
- Practice Effects
When defining the DV, what should you include
- measurement, scale, type of data produced, meaning of high/low scores, reliability and validity of measurement
- be unobtrusive in measurement
- if judges/raters are used, describe criteria for competence and inter-rater reliability
Threats to External Validity: Interferences from Multiple Treatments: What are Carryover Effects
Effects of one treatment contaminate the next (e.g., drug A is still in Ss system when he takes drug B)
What should you do during the manipulation phase of your study?
- describe the research setting
- consent participants
- describe the cover story
- reduce sources of error
Threats to External Validity: Interferences from Multiple Treatments: What are Order Effects
Treatment A alters participants’ response to Treatment B
Threats to External Validity: Interferences from Multiple Treatments: What are Practice Effects
Treatment A teaches participants how to respond to Treatment B
During the manipulation phase, how do you reduce sources of error when designing conditions?
All conditions should be identical except for the IV
Describe the External Validity threat: Selection x Treatment
- The sample is primed by testing and thus not representative of the population
- i.e., the pretest increases or decreases the participants’ sensitivity or responsiveness to the experimental variable such that the pretested sample is no longer representative of the untested population (to which results are to be generalized)
During the manipulation phase, how do you reduce sources of error in terms of experimenter bias?
- Experimenters should be blind to the hypothesis or subject condition
OR
- reduce contact btwn experimenter and subjects
Threats to External Validity: What are the four Reactive Arrangements
- Hawthorne Effect
- Experimenter Expectancy
- Evaluation Apprehension
- Demand Characteristics
During the manipulation phase, how do you reduce sources of error in terms of practice and order effects?
You should consider practice and order effects.
Threats to External Validity: Reactive Arrangements: What is the Hawthorne Effect
Change is a result of being observed and would not really occur in unobserved populations
During the manipulation phase, how do you reduce sources of error when describing the study to Ss?
- Give clear, specific instructions
- explain the purpose of the study
- emphasize its importance
- ensure confidentiality
Threats to External Validity: Reactive Arrangements: What is Experimenter Expectancy
The experimenter communicates their expectations to the subjects implicitly or explicitly
During the manipulation phase, how do you reduce sources of error in terms of rumors about the study?
Control rumors about the experiment
Threats to External Validity: Reactive Arrangements: What is Evaluation Apprehension
Subjects try to avoid negative evaluations
What two things should you do at the end of the manipulation phase
- Manipulation check
- Debrief participants
Threats to External Validity: Reactive Arrangements: What are Demand Characteristics
- Cues provided by the experimental setting that help the subject to develop naïve hypotheses about the experiment’s purpose and to behave accordingly
- When participants behave as they think the experimenter would want them to
Describe what is done during a manipulation check? Give an example of how you can do a manipulation check.
You make sure the IV was perceived by participants in the intended way (e.g., give a questionnaire at the end of the treatment asking Ss about their experience of the study)
What is reliability
How precise is our measure? (the extent to which it is free from random error)
What does it mean to “debrief” participants
You ask them how their participation in the study went; ask them if they have any questions
What are some ways you can increase reliability
- Use heterogenous group of Ss
- Make test items more similar
- Standardize administrations (and avoid context effects and priming - start with neutral, non-threatening items)
- Increase the length of your measure
- Make items and instructions clear (don’t use reverse-worded items, give cover story in everyday language)
After you determine your population of interest, you can get a representative sample through _________
random sampling
Name four ways of assessing reliability
- Test-retest
- Parallel Forms
- Split-Halves
- Internal Consistency Method
When describing your setting and participants, how do you determine how many subjects you need per cell
You do a power calculation. (Jess says you need at least 40-50 Ss per cell)
Describe Test-Retest reliability
- You give the same test twice
- the correlation between the two scores is the reliability (the reliability coefficient)
When deciding who you will and will not include in your study, you should describe __ and __ criteria.
inclusion and exclusion criteria
What are the assumptions and limitations of Test-retest reliability
- assume true score didn’t change
- there are memory effects
- reactivity (from practice effects)
- you assume the tests were parallel
What are some sample demographics you should describe
gender, race, age, other relevant characteristics
Describe Parallel Forms reliability
- You use two different forms
- the correlation between the scores on the two forms is reliability
If you are using a between-subjects design, you can ____ Ss to groups
randomize
What are the Advantages of Parallel Forms reliability
You don’t have to worry about memory effects
If there are important variables that can be identified and measured and equivalence among groups is a problem (even given randomization) what are other ways of equalizing groups using subject variables
matching or blocking
What are the limitations of Parallel forms reliability
- assume true score didn’t change
- assume your error variances are equal
- reactivity (from practice effects)
- it can be difficult to come up with a second test
Define and give examples of moderators
- Moderators can be measured or manipulated
- Usually a demographic-like variable
- answers when or for whom
- (e.g., gender, boring/intersting task, simple/complex task)
Describe Split-Halves reliability
- Administer a single test
- Use 1/2 of the test for one score and 1/2 for the other score
- using the correlation, you will get the “split half” reliability
How do you test for a moderation effect
By creating an interaction term and seeing if it is significant
What are the advantages of uisng Split-Half reliability
You don’t have to worry about:
- stability of the true score (since you only give test once, your score can’t change btwn tests)
- memory effects
- reactivity (practice effects)
it’s cheaper and easier
Define and give examples of mediators
- Mediators can be measured
- Often a psychological proces variable or an internal psychological mechanism
- answers the question how or why the IV influences the DV
- (e.g., anxiety, stress level, self-esteem)
*
What are the disadvantages of using Split-half reliability
- The split-halves have to be parallel
- there are many ways to split the test
- diminishing returns (as you keep increasing the length of the test, the reliability goes up, but more and more slowly)
When analyzing data for a true experimental design, you will usually run an
ANOVA
Describe Internal Consistency reliability
compute coefficient alpha/cronbach’s Alpha
When analyzing data to test for moderation, use a _________
factorial ANOVA
What is alpha
the reliability over all split halves
When analyzing data to test for mediation, you should run ___, ___, or ___
several regression equations, partial correlation, or path analysis
Cronbach’s alpha has a range of ___ to ___
0 to 1
When reporting results, don’t report the main effect when
the interaction effect is significant
Standard Error of Measurement measures the ____ of an individuals score (thus related to _____) - that is, how much the score would vary if tested repeatedly (but it is only based on one testing)
Precision, reliability
When reporting results, state the results in words with their appropriate _______
statistics
The Standard Error of Measurement (SEM) is ____ when reliability goes up
smaller (i.e., you want a small SEM)
When reporting results, how should you visually display results?
- Show a line graph for the interaction or main effect
- Show a diagram (possibly with correlations) for the mediation effect
Validity
The extent to which a measure actually measures what it is meant to measure, and nothing else
When reporting results, what alternative explanations (limitations) should you consider
- limited generalizability (e.g., are you just using college students)
- IV manipulation is ineffective (i.e., not powerful enough to detect a difference
- IV operationalization is not effective (i.e., you are measuring something else, another influece diluted your efect)
- systematic error/systematic differences btwn groups that covary with manipulation
- confounds
If a measure is not _____it cannot be valid
reliable
After reporting results, you should give ideas for future _____
research
If a measure is not reliable it cannot be _____
valid
What is the purpose of a cover story
- Answers the question: Why are you here?
- Gives participants the sense that they know what is going on, gives an overall framework in which to interpret the events and tasks called for by the research
- Keeps pts interested and motivated and to keep them from speculating about what the researcher really wants
*It is best to avoid deception when possible
What is face validity
- Not really validity
- does a test measure what it superficially appears to measure
What components should you include when answering Caryn Block’s question on the CERT?
- Design
- Variables
- Hypotheses
- Setting and Participants
- Operationalizations
- Describing Moderation first
- Describing Mediation
- Alternative explanation of results
What is content validity
- Whether the measure fully represents the domain of the content it is supposed to represent
- established through a rational analysis of the content fo a test
- its determiniation is based on individual, subjective judgment
What design is usually used on the CERT for the Caryn Block question?
2x2 factorial ANOVA
What are the two types of content validity
- face validity
- logical validity
Hypotheses: Give an example of an explicit comparison that can be made between groups.
People who work in homogenous groups will perform better than people who work in heterogenous groups.
)could also say as compared to)
What is criterion validity
The degree of relationship between a measure and a criterion (e.g., 1st yr college GPA, job perf compared to a test you took when you first applied)
Operationalizations should be _______. If relevant, give ___ of what research assistants will say.
specific
quotes
What are two types of criterion validity
- Predictive Validity
- Concurrent Validity
When describing the moderation effect, the ___ variable goes on the X axis and the ____ variable goes on teh Y axis.
independent
dependent
What is concurrent validity
- scores discriminate btwn groups that are known to differ at time of testing
- (data is available at the same time, even if you don’t give measures on the same day)
How do you describe the mediation effect for the Caryn Block question?
Demonstrate that the mediator is correlated with the DV
In a regressin, demonstrate that the interaction term is significant in predicting the mediator
In a regression with the main effects, interaction effect, and mediator all predicting the DV, demonstrate that the interaction effect is not significant
What is predictive validity
- measures predictor first, then criterion (the criterion becomes available in the future)
(e. g., getting SAT scores and later getting 1st yr college GPA)
What alternative explanations of results do you want to describe for the Caryn Block question?
Any systematic differences btwn groups that covary with manipulations
confounds
history effect (e.g., one room is hotter than the other)
What is construct validity
the degree to which the test measure the theoretical construct or trait that it was designed to measure
*You need to demonstrate that
- Your measure is related to other measures of your construct
- Your measure is related to measures of constructs that are related to your construct
- Your measure is unrelated to measures of constructs that are unrelated to your construct
What are the 3 desiderata of research?
Control (behaviors)
Realism (Context)
Generalizability (Actors)
What falls under construct validity
- convergent validity
- discriminant validity
- multitrait-multimethod matrix
What is the 3 horned delimma (desiderata)
If you maximize one or two of the desiderata you will, by definition, minimize the third
What is convergent validity
- measures the same trait by different methods
- should be correlated with your measurement of the construct
A theory can never be ___ / ___ / ___, but rather it is ___ based on data
proven, confirmed, disconfirmed
modified
What is discriminant validity
- different measures of an unrelated construct are uncorrelated with your measure
- the correlation between different traits measured by the same or different methods (should be smaller than convergent validity)
- can be used to differentiate your construct from other similar constructs
variables are concrete partial representations of ____
constructs
What is the multi-trait multimethod matrix
- A table of correlations that establish both convergent and discriminant validity of a measure
- used when 2+ traits are being measured by 2+ methods
How can hypotheses be worded for categorical variables
- Condition A1 has a positive effect on B as compared to Condition A2.
- Condition A1 will result in less B than Condition A2.
In terms of Internal Validity, by using a more precise test, you will have less ___ ___ the ___ from pretest to posttest
regression towards the mean
How can hypotheses be worded for continuous variables
- The higher A, the lower B.
- As A increases, B decreases.
- A is negatively related to B.
This is the only way to make two or more groups equal on all possible differences
Random assignment
What does the moderating variable do
- Partitions the IV into subgroups that differentially impact the DV
- Can be subject variables (age, race, and socio-economic status)
- Can be situational variables (task conditions, environmental conditions)
- Often introduced when an unexpectedly weak relationship between IV and DV had been found.
Random assignment is not random ______
sampling
Provide an example of a moderating variable (drawn out)
Threats to Internal validity: What 3 factors do not change participants but do change average scores on the posttest
Instrumentation
Regression
Mortality
The operational definition of a construct is
the set of procedures we use to measure or manipulate it
Threats to Internal Validity: What 3 Non-treatment factors
change participants and their scores
Maturation
History
Testing
When considering researchable ideas, mediators may provide ___ ___ for the relationship between the IV and the DV
alternative explanations
Threats to Internal Validity: What 3 factors challenge
the assumption of equal groups
Selection
Selection X Maturation
Mortality
When considering researchable ideas, the moderator will tell you ___ the relationship is more likely to hold
for what type of people / for who
What are 4 problems of lab experiments
- Some variables cannot be experimentally varied
- Some variables cannot be ethically varied
- Some variables occur over a long period of time and cannot be looked at in the context of the lab
- Some variables might be hard to elicit in a lab
Describe the partial mediation model (image)
Field experiments have:
High ___
High ___
___ Validity
___ Validity
High realism
High control
Internal Validity
External Validity
Describe how mediating variables work (image)
Field experiments have ___ control over conditions and ___ random and systematic error than in a lab
Treatment dilution
Treatment confounding
Treatment contamination
Changes in treatments
less, more
Treatment dilution
Treatment confounding
Treatment contamination
Changes in treatments
Give an exaple of a model that includes both mediators and moderators (image)
What is a factorial design and what does it allow us to do?
- Looks at how 2 IVs jointly effect a DV
- Allows us to do two experiments at one time (main effects)
- Allows us to look at interactions
Of the IV, DV, mediators and moderators, which can be manipulated and which can be measured (image)
What are some examples of factorial design notation?
2x2 (2 IVs both with 2 levels)
2x3 (2 IVs - one with 2 levels, one with 3 levels)
2x2x2 (3 IVs - all with 2 levels)
What is an example of an interaction (image)
People who receive pay perform better than those that do not when the task they are working on is boring, whereas people who do not receive pay perform better than those who do when the task they are working on is enjoyable.
What is random error and what does it do
- extraneous variables whose average influence on the outcome is the same in all conditions
- obscures the relationship between the IV and the DV
For factorial designs, when creating graphs, the DV goes on the ___ axis, one IV goes on the ___ axis and the other IV __
The DV goes on the y-axis
One IV goes on the x-axis
The other IV is plotted
What does the main effect of an IV tell us
The main effect of each IV tells us about the relationship between that IV and the DV
What does it mean if we have an interaction
An interaction indicates that the effect of one IV is different at different levels of the other IV
How can you tell if you have a main effect (using a table)
A main effect of Word Type tells us that more words are recalled when they are concrete
A main effect of Rehearsal Type tells us that more words are recalled when imagery is used
How can you tell if you have an interaction (using a table)
To calculate interactions we are interested in differences
If the differences are different then you have a two-way interaction
What do graphs of main effects look like?
What do graphs of interactions look like
What is systematic error and what is it a threat to?
- Systematic error: extraneous variables whose influence on the outcome is different across conditions of the IV
- Systematic error variables are a threat to internal validity
Systematic error results from ____ that cannot be separated from the IV
and distorts the relationship between the ___ and the ___
extraneous variables, IV, DV
In a Between-subjects design, each subject is exposed to a ___ and ___ ___ are compared
single condition
group means
In a Within-subjects design, each subject is exposed to ___ and ___ ___ are compared
several or all conditions
individual differences
What are the advantages of a within Ss design
- Requires fewer subjects
- Higher sensitivity
- Reduces random and systematic error (subject serves as his or her own control)
What are the disadvantages of a within Ss design
- Inter-treatment contamination / Order effects
- Practice effects
- Sensitization
- Carry-over
The cover story should be
- Simple
- Involving
- Unrelated to manipulations
- Inducing the same psychological state (in the experimental and control participants?)
Operationalizing the IV:
Describe the relationship between control and impact
- We need as much control as possible over manipulation so that error is minimized
- As we increase the impact of a manipulation, we are likely to decrease our control over it.
- Our IV should have enough impact to show up against background noise, but not so much as to have multiple meanings or alternative explanations
What is experimental realism (a way in which manipulation can be realistic)
- ‘Psychological reality’
- Manipulation is realistic to subjects and has an impact to them
What is mundane realism (a way in which manipulation can be realistic)
- ‘Physical reality’
- Manipulation is similar to real-life events
Demand characteristics are a source of ___ error
systematic
What are Strategies for minimizing
demand characteristics
- A deceptive cover story
- Collect DVs unobtrusively
- Expose subjects to only one condition
- Separate IV manipulation from experimental session
- Subjects are unaware that they are in an experiment
- Use Behavioral Measures as DVs
How can you Minimize experimenter expectancy
- Experimenter is unaware of condition
- Experimenter is unaware of hypothesis
- Multiple experimenters within sessions
- Multiple experimenters across sessions
- Experimenter is unaware of condition as long as possible
- Randomly assign experimenters to conditions
- Experimenters are trained
- You can also Minimizing experimenter’s role in the experiment by using media or limiting experimenter-subject interactions
What are Alternative methods of establishing equivalence when randomization to groups is not possible
- Matching (controlling subject variable)
- Blocking (controlling subject variable)
- Nested design (controlling contextual variable)
- ANCOVA (statistical post-hoc procedure)
Describe matching in experiments (image)
What are the pros and cons of using matching
- Pro: Equivalence most likely established on matched variable
- Con: No longer have random assignment - differences are likely to exist after matching on other subject variables
- Con: Eliminating cases that cannot be matched raises questions about external validity
- Con: Difficult to choose which subject variable is the only important one
Describe blocking in experiments (image)
What are the pros and cons of Blocking versus Matching
- No unconditional random assignment so that other important subject variables are not controlled for, but better random assignment than with matching.
- Blocking assigns subjects to a few blocks, differences between conditions might still exist after blocking.
- More subjects are required for successful blocking than for successful matching!
Describe Nested designs (image)
What are the pros of nested designs
- Accounts for an important naturally occurring variable that is contextual and is likely to affect the DV
- Random assignment does exist within each block
- Equivalence most likely established within each of the blocked groups
What is ANCOVA
- Statistical post-hoc procedure that removes influence of third variable
- a ‘what if’ analysis: What would happen if all cases scored equally on the covariate?
- Randomly assign subjects to conditions OR Create two conditions, not through random assignment
- Measure the variable (covariate) you want to control for (e.g., pretest score)
- Remove influence of this variable statistically with ANCOVA procedure
What are the pros of ANCOVA with random assignment
Removes random error due to the control variable, after random assignment
Allows you to quantify the effect of this variable while maintaining equivalence through random assignment
What are the issues of ANCOVA without random assignment
- Attempts to remove systematic error due to the control variable, to establish equivalence between groups
- Groups are not equivalent on many other variables
- Can’t use ANCOVA to establish equivalency between groups post hoc
What is internal validity
- the extent to which conclusions can be drawn about the causal effects of one variable on another
- Allows us to make the statement We can make the statement: IV causes DV
In correlational research, the IV is ___ not ___
measured
manipulated
What are some reasons why you would do correlational research, measuring rather than manipulating the IV
- Variable of interest cannot be manipulated
- It is unethical to manipulate the variable
- It is not practical to manipulate the variable
What are the three criteria for establishing causality:
- Association (correlation)
- Time Priority (cannot be established if IV and DV are measured simultaneously)
- Non-spurious Relationships (A third variable causes the relationship between IV and DV. The correlation between IV and DV is entirely determined by the third variable.)
Establishing Causality: What is a spurious relationship? Give an example.
- A third variable causes the relationship between IV and DV. The correlation between IV and DV is entirely determined by the third variable.
- IV and DV share no variance after removing the covariance of a third variable.
(e.g., The rel btwn the number of Storks (IV) and Number of Babies (DV) is caused by population density (EV))
Establishing Causality:What is full mediation? Give an example.
The third variable mediates the relationship between IV and DV. The correlation between IV and DV is entirely determined by the third variable.
e.g., The rel btwn Counseling (IV) and Well-being (DV) is mediated by Social support (EV)
Establishing Causality: What is Partial Explanation/Isolation? Give an example.
The third variable influences the relationship between IV and DV. But it does not entirely determine the relationship
e.g., the rel btwn Satisfaction (IV) and Performance (DV) is partially explained by Rewards (EV)
Establishing Causality: What is Partial mediation? Give an example.
The third variable mediates the relationship between IV and DV. But there is also a direct relationship between IV and DV.
e.g., the rel btwn Counseling (IV) and Well-being (DV) is partially caused by Social support (EV), but there is also a direct rel btwn the two
What is validity
Does the measure reflect the construct we want to measure and no other systematically varying constructs
A measure can’t be v___ if it is not r___
valid
reliable
*Reliability is a necessary, but insufficient condition for validity
What type of wording should you avoid when writing test items
- Ambiguity
- Jargon
- Length
- Double-barreled
- Leading
- Loaded
- Threatening
- Over-demanding
- Over-specificity
- Relevance
What is the Multitrait Multimethod Matrix (construct validity)
A table of correlations that establish both convergent and discriminant validity of a measure
