Survey Methods

Question 1

To determine the quality of the measure for your purposes, you must ask:

Answer 1

Is is appropriate for research question and design?
Does it have good psychometric properties?
- validity (construct, content, criterion)
- reliability (internal consistency, test-retest reliability)

Is the questionnaire appropriate from a practical sense?
- is it free?
- do i require training?

Question 2

What are some advantages and disadvantages about using google scholar to find measures?

Answer 2

Adv:
- captures a lot of literature
- no prior knowledge of creating searches needed
- linked to MQ library

Dis:
- captures a lot of literature (too much)
- difficult to easily judge quality of results
- time consuming

Question 3

Using electronic data bases:

Answer 3

search within other research papers that use the same variables and population that you are interested in
- good if you make a note of the measures used when you are doing background reading
- information of the most commonly used measures (aid in comparability)
- provides you with reliability and validity info
- very time consuming

Question 4

Where you can find measures in MQ library?

Answer 4

We have a library separate to the main one with psych tests and measures

Library home –> study support –> subject and research guides –> browse library guides –> psych sciences –> test and measurements

Question 5

What’s another database for tests to look at?

Answer 5

APA PsycTEST
shows us details for us to judge it with

Question 6

Determine whether to adapt an existing measure or create a new one:

Answer 6

Depends on how big of a change you want to make:
- small change in wording
- omission / addition of item
- change response scale
- what does that mean for the comparability of results

Another common issue is having different response scales within the same survey –> in this case, you would change it

Question 7

What are the rules of thumb for whether you should adapt an existing measure or create a new one

Answer 7

Keep the response scales consistent across measures where possible

This is not possible in some cases (i.e. if one is measuring attitudes and the other is measuring freq)

Question 8

Using adapted / an existing well-validated instrument will:

Answer 8

save time
save resources
allow you to compare results with previous studies
means you only have to give an overview of the measure in your method section
make it easier to publish your study

Question 9

If you decide to make your own, where do you start?

Answer 9

Existing measures
Theories on the research topic
Focus groups

Draft your survey items to maximise reliability (will they be interpreted the same way by your respondents in the way you intended) and validity (will the items tap into the constructs they need)

Pilot and edit your items –> ask for feedback from your colleagues, edit, pilot your draft survey and ask for feedback, edit

Question 10

What are the 3 types of surveys

Answer 10

Mail survey
+ precise access to target population
+ inexpensive and convenient
+ increased honesty of responses
+ avoids training of research, staff and technical issues
- poor response rate (3-15%)
- not suitable for certain populations
- no control over admin
- manual data entry

Interview
+ suitable for all populations
+ captures nonverbal cues
+ minimises missing data
- high cost, time consuming
- limited sample size
- manual data entry
- anonymity compromised

Online survey
+ convenient, accessible
+ time and cost effective, large samples
+ design flexibility
+ automatic collection and storing of data
- can’t access no-internet populations
- increased fraud
- incentives may encourage falsification

Question 11

Is your survey highly sensitive, sensitive or general?

Answer 11

Highly sensitive: identifiable / re-identifiable, information related to health, finances, criminality, race, religion, political opinion, genetics

Sensitive: identifiable / reidentifiable, does not contain any of the sensitivity indicators above

General: not identifiable / reidentifiable, data has been anonymized and is publically available
- is in aggregated form

Question 12

Use a data sensitivity calculator to help you ascertain which

Answer 12

research platform browser to use

Question 13

What is the preferred platform to create surveys?

Answer 13

Lime Survey

Followed by red cap and qualtrics

Question 14

What is the preferred platform to store data of surveys?

Answer 14

Australian Data Archiva

followed by Sharepoint/Onedrive, then Open Science framework (which is never preferred and DO NOT USE for highly sensitive data)

Question 15

Where to disseminate / advertise new survey to participants?

Answer 15

• relevant organisations (care homes, sporting groups, schools)
• personal / professional email
• research participant pools (Mq SONA, Prolific, Mturk, Qualtrics)
• social media
• word of mouth

Question 16

Statistics on survey response validity

Answer 16

3-9% of respondents engage in highly inattentive / careless responding (10-12% in ug)

introduces more error variance into data set reducing data quality, decreases statistical power, and obscures meaningful results

Question 17

What are some other types of invalidity

Answer 17

• faking good
• faking bad
• social desirability
• slackers: lack proper motivation
• straight liners: select the same answer
• speeders: work through studies as quickly as they can
• survey bots: automatically fill in question bubbles

Question 18

How to detect invalid responding: inattention

Answer 18

• inclusion of specific items (directly or indirectly) designed to detect inattention

Question 19

How to detect invalid responding

Answer 19

Multivariate outlier analysis: assessing statistically unlikely response patterns

Survey response time: completing the survey in a suspiciously short amount of time

Long string analyses: measuring the tendency of participants to repeatedly choose the same answer within a block regardless of item content

Self-reported diligence: self-report response style measures (“how often do you read each question carefully”)

Question 20

How to handle invalid responses in your dataset

Answer 20

1. Delete all cases with inappropriate responses
2. Create a cut-off score (>80% incorrect response on attention items)
3. Conduct a sensitivity analyses (conduct the analyses with and without suspected invalid responses and see if the results change substantially)
4. may be appropriate to code their invalid responses as missing and then apply the most appropriate method for handling missing data (to retain more of your sample)