Topic 7: Behavioral Research and Experiment Design Flashcards
How is Research defined?
Research is a systematic process of
collecting,
analyzing and
interpreting data to increase our understanding of a phenomenon about which we are interested or concerned.
Explain the difference between Objectivism and Constructivism
Constructivism, There can be multiple truth, depending on the perspective/view.
Objectivism: Discover the objective truth
What is the goal of behavioral research?
To examine and understand human behavior through measurement and interpretation.
What characterizes a good research question or problem?
Most Important: It has to be a new and novel question that is practical significant (useful)
1) The problem should be handleable and feasible to test (practicality)
2) It should be related to the existing research foundation
3) It should be replicable and repeatable.
Paper: What constitutes a publishable theory paper?
- What’s new? - new/novelty, value-added
- So what? - practically significant, change the practice
- Why so? - logic, theory development
- Well done? - well executed, conceptually well-rounded
- Done well? - easy to understand
- Why now? - timing, topic of contemporary interest (not redundant, unconnected, or antiquated)
- Who cares? - percentage of interested academic readers in this topic (not too specialized, which benefits 1.&2. but lacks generalizability)
Describe the research process
- Problem is identified
- Question is posed
- Question is converted to a clearly stated research problem –> problem statement
- Hypotheses are created
- Literature is searched
- Data is collected
- Data is analyzed against the hypotheses
- Conclusions are made
How to create a good problem Statement?
The problem statement is like aiming for a gun. You must carefully aim by clarifying your problem statement in a clear format.
What is a Hypothesis?
It is a phrased expectation of the finding! It is more specific that a general theory. It is a precise problem statement that can be directly tested through an empirical investigation.
What are the four qualities of a good research project?
1) (Universality) What we find should be applicable to the industry, to regular consumers
2) (Replication) Someone else should get the same finding when following the method
3) (Control) I have to be in control of the influencing factors to rule out external effects that distort the results. Eliminate influencing optional hypothesis
4) (Measurement) alternative perspective on how to measure the results
What are the 4 fundamental questions of data collection?
Data:
- what is needed
- where to gather
- how to obtain
- how to analyze
What is the difference between primary data and secondary data?
Primary data is gathered by you, and secondary data is measured by someone else
What is meant by nature of measurements and why is it important?
Reliability
- to which extent the measurement items yield a consistent result
Validity
- to which extent the instruments measures what it is intended to measure
What tool measures the reliability of data?
Cronbach’s Alpha
(Inter-item correlation Analysis)
Internal consistency is usually measured with Cronbach’s alpha.
What is the difference between reliability and validity for measurements?
If the measured results are close to each other, they are reliable. They produce nearly the same results over and over again. If these results are off from what I wanted to measure, they are not valid.
What are the 3 types of research and what is their focus?
Descriptive: describe a situation
Relational: identify relations between variables
Experimental: identify causes of a situation
What are the requirements for causal relationships of two effects?
- Covariation (If X changes positively Y changes as well in either direction)
- Time precedence (First X moves, then Y moves)
- Non-concomitance (No other factor is influencing the effect)
What is internal consistency?
Typically a measure is based on the correlations between different items on the same test. Usually really high within a controlled lab environment.
What is a Null Hypothesis, and why is it beneficial to disapprove of it?
The Null Hypothesis represents the common idea or belief. Instead of carefully trying to prove this belief, it is easier to prove that it can be wrong.
Null Hypothesis: The pill has no effect on the health of the patent
Alternative Hypothesis: The pill has a effect on the patent
What is the difference between the null & alternative hypothesis?
The alternative hypothesis is mutually exclusive with the null hypothesis.
What is goal of an experiment?
To find statistical evidence to refute or nullify the null hypothesis in order to support the alternative hypothesis
What should a hypothesis specify?
It should define the independent variables (IV) and the dependent variables (DV).
Define what independent variables (IV) and dependent variables (DV) are
IV: the factors that researchers are studying (the possible “causes”) –> they are usually the treatment/condition that the researchers control
DV: the outcome/effect the researchers are interested in
(goal: show that DV is dependent on changes in IV)
What are typical Independent Variables (IV)?
1.) Related to Technology:
a) Types of devices (e.g. keyboard vs. tablet)
b) Type of design
2.) Related to Users:
a) age, gender or computer experience
3.) Related to context:
a) physical status
b) user status
c) social status
What are typical Dependent Variables (DV)?
Efficiency
Accuracy
Subjective satisfaction
Ease of learning and retention rate
Physical or cognitive demand (e.g. NASA task load index)
Why is it important to have a randomization of a pretest?
Randomization allows a balance of multiple factors that can act as noise or influence that were unknown in advance.
–> allows to balance out hidden factors
What are 3 randomization methods?
Simple: random generator, no constraints
Block: select out of blocks with equal representation
Stratified: random selection out of prior made clusters
What is quasi-experimental design all about?
Creation of a valid experiment without the randomization of the test group.
Like a true experiment, a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable.
However, unlike an actual experiment, a quasi-experiment does not rely on random assignment. Instead, subjects are assigned to groups based on non-random criteria.
A quasi-experimental design is a useful tool in situations where true experiments cannot be used for ethical or practical reasons.
What problems could arise with simple randomization?
For example, there can be a possibility of an uneven distribution of gender.
Why do we need Significance Tests?
1) When the population is large, can we only sample a sub group of people from the entire population?
2) it allows us to determine how confident we are that a result form the sample group can be generalized to the entire population
What is a type 1 and 2 error?
Type 1 (alpha) error - “false positive”:
- mistake of rejecting H0 when it is true
–> commission error
Type 2 (beta) error - “false negative”
- mistake of not rejecting H0 when it is false
–> omission error
Is type 1 or type 2 error worse?
Type 1 errors, as the society is pooled on H0 (can cause a condition that is worse than current state)
(Type 2 errors can be seen as blindness - it’s there but we didn’t see it –> lost opportunity for improvement)
What is the statistical power of test and how can it be improved?
The statistical power of test refers to the probability of successful rejecting H0 when it is false and should be rejected –> 1-ß
It can be improved by a:
- higher α-error –> lower ß-error
- bigger sample size n
- larger effect size
How are α- and ß-error related?
They are interrelated, meaning decreasing α (alpha) reduces the chance of making Type 1 error but increases the chance of making Type 2 errors
What is ment by p value? and what is popular used value for p?
The p value describes the probability that Type 1 occurs.
Typically p should be lower than 0.05
What is essential for a research experiment?
To have well-defined, testable hypotheses that consist of a limited number of dependent and independent variables.
What is a core characteristic of experimental research?
At least one treatment (manipulation)
Need to decide about:
- treatment/conditions (what you want to compare),
- units (objects on which to apply the treatmend) and
- assignment method (the way units are assigned to different treatments/conditions)
What is a control variable?
X has an influence on Y, therefore, is Y dependent on X. A controlled variable W could have an influence on Y as well but is shown to be unchanged and under control.
–> isolate or include into the assesment the control variable
Why are pretests important?
- to assess equivalence before the treatment
- discover confounding variables that covariate (may include into analysis with ANCOVA)
- assess the status quo to measure the treatment effect in combination with a post test
What are the key considerations for internal validity?
Experimental design:
- Pretest-post test (measure treatment effect)
- Control group (measure effect of independent variable)
- Randomization (balance concomittance among all groups | quasi-experiments don’t use randomization)
Measurement:
- Reliability (have consistent results - low variance)
- Validity (have correct results - measure what we want to measure)
Study validity:
- Threats to validity
What is the purpose of a control group?
A control group is a reference to the test group and receives no treatment or change.
What are the 5 experimental design types and their core characteristics?
- pre-experimental design (no randomization, one treatment, control group & multiple observations possible)
- true-experimental design (random assignment, one treatment, control group & multiple observations possible)
- quasi-experimental design (no randomization, at least 2 observations, at least one treatment)
- ex post facto design (no radomization, only 1 post observation)
- factorial design (random assignment, can study effect of multiple variables)
What are pre-experimental designs?
one-shot experimental case study: (1 group, 1 treatment, 1 posttest) –> cannot substante cause-and-effect relationship
One-group pretest-posttest design: (1 group, 1 posttest, 1 treatment, 1 posttest) –> measure of change, but no conclusion about cause
Static group comparison: (2 group incl. control group, 1 treatment, 1 posttest) –> fails to determine pretreatment equivalence
What are true experimental designs?
Pretest-posttest control group design: (randomization, pretest & posttest, control group) –> controls threats of internal validity
Solomon Four-group design (add two more groups without pretest) –> test if pretest is affecting the results
Posttest-only control group design (only posttest, randomization, control group) –> randomization essential for max group equivalence + useful when pretesting cannot or should not occur
Within-subject design
What are quasi-experimental designs?
never randomized:
- Nonrandomized control group pretest-posttest design
- Simple time-series design
- Control group time series design
- Reversal time-series design
- Alternating treatments design
- Multiple baseline desing
What are ex post facto and factorial designs?
- ex post factor design
- factorial designs –> discovering effect of multiple variables
What is a within-subject design?
In a within-subjects design or a within-groups design, all participants participate in every condition.
What is a between-subject design?
It’s the opposite of a between-subjects design, where each participant experiences only one condition.
What is a Solomon Four-Group Design?
This design adds groups in which a pre-test and a post-test are not done under the assumption that they have an influence on the study outcome. This is useful if there is a chance of the pre or post-test influencing the results.
What are threats to validity?
Statistical conclusion validity: low statistical power, violated assumptions of statistical tests
Construct validity: inadequate preoperational explication of constructs
Internal validity: ambiguity about the causuality direction
External validity: interaction of setting (or selection) and treatment
