3.2 Flashcards
1
Q
Basic steps of conducting experiments
A
- Define purpose of the experiment
- Define concise research question
- Define dependent variable
- Define independent variable
- Manipulate independent variable/design treatment
- Define participants
- Theory work (research model & hypotheses)
- Identify control variables
- Craft experiment design
- Craft experiment instructions/define incentive
- Conduct pretest
- Refine experiment design
- Conduct experiment
- Analyze experiment
- Write your work down
1-12 = work before experiment 13-15 = work after experiment
2
Q
- Define participants
A
- Can be in a physical lab, then it will be mostly students
- Can be through an online micro-task platform (Amazon Mturk, Prolific, etc.)
- If it is a field experiment, then it can be employees, customers, a lot of possible options
3
Q
- Craft experiment instructions/define incentive
A
- Write a clear and concise introduction for the participants.
- DO NOT write exactly what your aim is (seeing if people rate differently when they are exposed to chatbots). Just give a high-level intro (we are interested in your evaluation of this video).
- Workers were paid 0.13£ per minute upon completion of the task.
- We also implemented attention checks (asking random questions), to check if participants actually pay attention.
4
Q
- Conduct pretest
A
- Conduct pre-tests for all the important variables in your experiment
- Independent variable (survey vs. chatbot):
Do people understand the survey, are questions clear (more on that in the survey session)
Does the chatbot work? - Dependent variable (rating):
Which scale to use (5-point, 7-point, 10-point)? - Mechanism (interactivity & social presence):
What are the accepted scales in the literature to measure interactivity & social presence? - When you are all set and done, launch the experiment.
- Platforms like Amazon Mturk or Prolific will be able to recruit a couple of hundred participants within a few hours.
5
Q
- Analyze experiment
A
- Randomization check
- Manipulation check
- Main results / hypothesis test
6
Q
13.1 Analyze experiment - Randomization check
A
- Probably the most important task is to check if your randomization has worked properly. If not, everything is in danger.
- You do this by testing if your control variables are significantly different between your treatment and control groups with a t-test, for instance
7
Q
13.2 Analyze experiment - Manipulation Check
A
- Very important: You want to test whether your treatment actually had the desired effect. In our case, you want to see if chatbots actually had higher interactivity and social presence than conventional review forms.
- T-test for the variable between treatment and control group
8
Q
13.4 Analyze experiment - Tests to use
Data normally distributed
A
- Independent t-test
- Dependent t-test
9
Q
13.4 Analyze experiment - Tests to use
Data not normally distributed
A
Family of non parametric tests:
- Mann-Whitney U test (instead of independent t-test)
- Wilcoxon Signed Rank Test (instead of dependent t-test)