Topic 7: Behavioral Research and Experiment Design

Question 1

How is Research defined?

Answer 1

Research is a systematic process of
collecting,
analyzing and
interpreting data to increase our understanding of a phenomenon about which we are interested or concerned.

Question 2

Explain the difference between Objectivism and Constructivism

Answer 2

Constructivism, There can be multiple truth, depending on the perspective/view.
Objectivism: Discover the objective truth

Question 3

What is the goal of behavioral research?

Answer 3

To examine and understand human behavior through measurement and interpretation.

Question 4

What characterizes a good research question or problem?

Answer 4

Most Important: It has to be a new and novel question that is practical significant (useful)

1) The problem should be handleable and feasible to test (practicality)
2) It should be related to the existing research foundation
3) It should be replicable and repeatable.

Question 5

Paper: What constitutes a publishable theory paper?

Answer 5

1. What’s new? - new/novelty, value-added
2. So what? - practically significant, change the practice
3. Why so? - logic, theory development
4. Well done? - well executed, conceptually well-rounded
5. Done well? - easy to understand
6. Why now? - timing, topic of contemporary interest (not redundant, unconnected, or antiquated)
7. Who cares? - percentage of interested academic readers in this topic (not too specialized, which benefits 1.&2. but lacks generalizability)

Question 6

Describe the research process

Answer 6

1. Problem is identified
2. Question is posed
3. Question is converted to a clearly stated research problem –> problem statement
4. Hypotheses are created
5. Literature is searched
6. Data is collected
7. Data is analyzed against the hypotheses
8. Conclusions are made

Question 7

How to create a good problem Statement?

Answer 7

The problem statement is like aiming for a gun. You must carefully aim by clarifying your problem statement in a clear format.

Question 8

What is a Hypothesis?

Answer 8

It is a phrased expectation of the finding! It is more specific that a general theory. It is a precise problem statement that can be directly tested through an empirical investigation.

Question 9

What are the four qualities of a good research project?

Answer 9

1) (Universality) What we find should be applicable to the industry, to regular consumers
2) (Replication) Someone else should get the same finding when following the method
3) (Control) I have to be in control of the influencing factors to rule out external effects that distort the results. Eliminate influencing optional hypothesis
4) (Measurement) alternative perspective on how to measure the results

Question 10

What are the 4 fundamental questions of data collection?

Answer 10

Data:
- what is needed
- where to gather
- how to obtain
- how to analyze

Question 11

What is the difference between primary data and secondary data?

Answer 11

Primary data is gathered by you, and secondary data is measured by someone else

Question 12

What is meant by nature of measurements and why is it important?

Answer 12

Reliability
- to which extent the measurement items yield a consistent result

Validity
- to which extent the instruments measures what it is intended to measure

Question 13

What tool measures the reliability of data?

Answer 13

Cronbach’s Alpha
(Inter-item correlation Analysis)
Internal consistency is usually measured with Cronbach’s alpha.

Question 14

What is the difference between reliability and validity for measurements?

Answer 14

If the measured results are close to each other, they are reliable. They produce nearly the same results over and over again. If these results are off from what I wanted to measure, they are not valid.

Question 15

What are the 3 types of research and what is their focus?

Answer 15

Descriptive: describe a situation
Relational: identify relations between variables
Experimental: identify causes of a situation

Question 16

What are the requirements for causal relationships of two effects?

Answer 16

1. Covariation (If X changes positively Y changes as well in either direction)
2. Time precedence (First X moves, then Y moves)
3. Non-concomitance (No other factor is influencing the effect)

Question 17

What is internal consistency?

Answer 17

Typically a measure is based on the correlations between different items on the same test. Usually really high within a controlled lab environment.

Question 18

What is a Null Hypothesis, and why is it beneficial to disapprove of it?

Answer 18

The Null Hypothesis represents the common idea or belief. Instead of carefully trying to prove this belief, it is easier to prove that it can be wrong.

Null Hypothesis: The pill has no effect on the health of the patent
Alternative Hypothesis: The pill has a effect on the patent

Question 19

What is the difference between the null & alternative hypothesis?

Answer 19

The alternative hypothesis is mutually exclusive with the null hypothesis.

Question 20

What is goal of an experiment?

Answer 20

To find statistical evidence to refute or nullify the null hypothesis in order to support the alternative hypothesis

Question 21

What should a hypothesis specify?

Answer 21

It should define the independent variables (IV) and the dependent variables (DV).

Question 22

Define what independent variables (IV) and dependent variables (DV) are

Answer 22

IV: the factors that researchers are studying (the possible “causes”) –> they are usually the treatment/condition that the researchers control

DV: the outcome/effect the researchers are interested in

(goal: show that DV is dependent on changes in IV)

Question 23

What are typical Independent Variables (IV)?

Answer 23

1.) Related to Technology:
a) Types of devices (e.g. keyboard vs. tablet)
b) Type of design
2.) Related to Users:
a) age, gender or computer experience
3.) Related to context:
a) physical status
b) user status
c) social status

Question 24

What are typical Dependent Variables (DV)?

Answer 24

Efficiency
Accuracy
Subjective satisfaction
Ease of learning and retention rate
Physical or cognitive demand (e.g. NASA task load index)

Question 25

Why is it important to have a randomization of a pretest?

Answer 25

Randomization allows a balance of multiple factors that can act as noise or influence that were unknown in advance.
–> allows to balance out hidden factors

Question 26

What are 3 randomization methods?

Answer 26

Simple: random generator, no constraints
Block: select out of blocks with equal representation
Stratified: random selection out of prior made clusters

Question 27

What is quasi-experimental design all about?

Answer 27

Creation of a valid experiment without the randomization of the test group.

Like a true experiment, a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable.

However, unlike an actual experiment, a quasi-experiment does not rely on random assignment. Instead, subjects are assigned to groups based on non-random criteria.

A quasi-experimental design is a useful tool in situations where true experiments cannot be used for ethical or practical reasons.

Question 28

What problems could arise with simple randomization?

Answer 28

For example, there can be a possibility of an uneven distribution of gender.

Question 29

Why do we need Significance Tests?

Answer 29

1) When the population is large, can we only sample a sub group of people from the entire population?
2) it allows us to determine how confident we are that a result form the sample group can be generalized to the entire population

Question 30

What is a type 1 and 2 error?

Answer 30

Type 1 (alpha) error - “false positive”:
- mistake of rejecting H0 when it is true
–> commission error

Type 2 (beta) error - “false negative”
- mistake of not rejecting H0 when it is false
–> omission error

Question 31

Is type 1 or type 2 error worse?

Answer 31

Type 1 errors, as the society is pooled on H0 (can cause a condition that is worse than current state)

(Type 2 errors can be seen as blindness - it’s there but we didn’t see it –> lost opportunity for improvement)

Question 32

What is the statistical power of test and how can it be improved?

Answer 32

The statistical power of test refers to the probability of successful rejecting H0 when it is false and should be rejected –> 1-ß

It can be improved by a:
- higher α-error –> lower ß-error
- bigger sample size n
- larger effect size

Question 33

How are α- and ß-error related?

Answer 33

They are interrelated, meaning decreasing α (alpha) reduces the chance of making Type 1 error but increases the chance of making Type 2 errors

Question 34

What is ment by p value? and what is popular used value for p?

Answer 34

The p value describes the probability that Type 1 occurs.

Typically p should be lower than 0.05

Question 35

What is essential for a research experiment?

Answer 35

To have well-defined, testable hypotheses that consist of a limited number of dependent and independent variables.

Question 36

What is a core characteristic of experimental research?

Answer 36

At least one treatment (manipulation)

Need to decide about:
- treatment/conditions (what you want to compare),
- units (objects on which to apply the treatmend) and
- assignment method (the way units are assigned to different treatments/conditions)

Question 37

What is a control variable?

Answer 37

X has an influence on Y, therefore, is Y dependent on X. A controlled variable W could have an influence on Y as well but is shown to be unchanged and under control.
–> isolate or include into the assesment the control variable

Question 38

Why are pretests important?

Answer 38

1. to assess equivalence before the treatment
2. discover confounding variables that covariate (may include into analysis with ANCOVA)
3. assess the status quo to measure the treatment effect in combination with a post test

Question 39

What are the key considerations for internal validity?

Answer 39

Experimental design:
- Pretest-post test (measure treatment effect)
- Control group (measure effect of independent variable)
- Randomization (balance concomittance among all groups | quasi-experiments don’t use randomization)

Measurement:
- Reliability (have consistent results - low variance)
- Validity (have correct results - measure what we want to measure)

Study validity:
- Threats to validity

Question 40

What is the purpose of a control group?

Answer 40

A control group is a reference to the test group and receives no treatment or change.

Question 41

What are the 5 experimental design types and their core characteristics?

Answer 41

• pre-experimental design (no randomization, one treatment, control group & multiple observations possible)
• true-experimental design (random assignment, one treatment, control group & multiple observations possible)
• quasi-experimental design (no randomization, at least 2 observations, at least one treatment)
• ex post facto design (no radomization, only 1 post observation)
• factorial design (random assignment, can study effect of multiple variables)

Question 42

What are pre-experimental designs?

Answer 42

one-shot experimental case study: (1 group, 1 treatment, 1 posttest) –> cannot substante cause-and-effect relationship

One-group pretest-posttest design: (1 group, 1 posttest, 1 treatment, 1 posttest) –> measure of change, but no conclusion about cause

Static group comparison: (2 group incl. control group, 1 treatment, 1 posttest) –> fails to determine pretreatment equivalence

Question 43

What are true experimental designs?

Answer 43

Pretest-posttest control group design: (randomization, pretest & posttest, control group) –> controls threats of internal validity

Solomon Four-group design (add two more groups without pretest) –> test if pretest is affecting the results

Posttest-only control group design (only posttest, randomization, control group) –> randomization essential for max group equivalence + useful when pretesting cannot or should not occur

Within-subject design

Question 44

What are quasi-experimental designs?

Answer 44

never randomized:
- Nonrandomized control group pretest-posttest design
- Simple time-series design
- Control group time series design
- Reversal time-series design
- Alternating treatments design
- Multiple baseline desing

Question 45

What are ex post facto and factorial designs?

Answer 45

• ex post factor design
• factorial designs –> discovering effect of multiple variables

Question 46

What is a within-subject design?

Answer 46

In a within-subjects design or a within-groups design, all participants participate in every condition.

Question 47

What is a between-subject design?

Answer 47

It’s the opposite of a between-subjects design, where each participant experiences only one condition.

Question 48

What is a Solomon Four-Group Design?

Answer 48

This design adds groups in which a pre-test and a post-test are not done under the assumption that they have an influence on the study outcome. This is useful if there is a chance of the pre or post-test influencing the results.

Question 49

What are threats to validity?

Answer 49

Statistical conclusion validity: low statistical power, violated assumptions of statistical tests

Construct validity: inadequate preoperational explication of constructs

Internal validity: ambiguity about the causuality direction

External validity: interaction of setting (or selection) and treatment