3.2 Flashcards

1
Q

Basic steps of conducting experiments

A
  1. Define purpose of the experiment
  2. Define concise research question
  3. Define dependent variable
  4. Define independent variable
  5. Manipulate independent variable/design treatment
  6. Define participants
  7. Theory work (research model & hypotheses)
  8. Identify control variables
  9. Craft experiment design
  10. Craft experiment instructions/define incentive
  11. Conduct pretest
  12. Refine experiment design
  13. Conduct experiment
  14. Analyze experiment
  15. Write your work down
1-12 = work before experiment
13-15 = work after experiment
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q
  1. Define participants
A
  • Can be in a physical lab, then it will be mostly students
  • Can be through an online micro-task platform (Amazon Mturk, Prolific, etc.)
  • If it is a field experiment, then it can be employees, customers, a lot of possible options
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q
  1. Craft experiment instructions/define incentive
A
  • Write a clear and concise introduction for the participants.
  • DO NOT write exactly what your aim is (seeing if people rate differently when they are exposed to chatbots). Just give a high-level intro (we are interested in your evaluation of this video).
  • Workers were paid 0.13£ per minute upon completion of the task.
  • We also implemented attention checks (asking random questions), to check if participants actually pay attention.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q
  1. Conduct pretest
A
  • Conduct pre-tests for all the important variables in your experiment
  • Independent variable (survey vs. chatbot):
    Do people understand the survey, are questions clear (more on that in the survey session)
    Does the chatbot work?
  • Dependent variable (rating):
    Which scale to use (5-point, 7-point, 10-point)?
  • Mechanism (interactivity & social presence):
    What are the accepted scales in the literature to measure interactivity & social presence?
  • When you are all set and done, launch the experiment.
  • Platforms like Amazon Mturk or Prolific will be able to recruit a couple of hundred participants within a few hours.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q
  1. Analyze experiment
A
  1. Randomization check
  2. Manipulation check
  3. Main results / hypothesis test
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

13.1 Analyze experiment - Randomization check

A
  • Probably the most important task is to check if your randomization has worked properly. If not, everything is in danger.
  • You do this by testing if your control variables are significantly different between your treatment and control groups with a t-test, for instance
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

13.2 Analyze experiment - Manipulation Check

A
  • Very important: You want to test whether your treatment actually had the desired effect. In our case, you want to see if chatbots actually had higher interactivity and social presence than conventional review forms.
  • T-test for the variable between treatment and control group
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

13.4 Analyze experiment - Tests to use

Data normally distributed

A
  • Independent t-test

- Dependent t-test

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

13.4 Analyze experiment - Tests to use

Data not normally distributed

A

Family of non parametric tests:

  • Mann-Whitney U test (instead of independent t-test)
  • Wilcoxon Signed Rank Test (instead of dependent t-test)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly