Developer study design

Question 1

On the ecological validity of online sec dev studies Setup

Answer 1

Deception study:
Deception might cause unknown behavior if participants know deception study
Only use after careful consideration
Does it affect developers if they know they are producing code for a company instead of a study

Question 2

On the ecological validity of online sec dev studies - Differences

Answer 2

Real project had no study announcement and only allowed JSF
Study allowed for JSF and Spring.

Question 3

On the ecological validity of online sec dev studies - Procedure and Request

Answer 3

First submission. If plaintext storage (SeqRequest-P) if SecScore < 6 (SeqRequest-G)
Finally Survey.

Question 4

On the ecological validity of online sec dev studies - Results

Answer 4

Could check for attempted security (online setting) => check for achieved security
H-P1: Prompting has positive effect on achieving security YES
H-F1: Framework has an effect on the security score of participants achieving security NO
Years of Java experience have an effect on the security scores NO
If participants state that they have previously stored passwords, it affects the likelihood that they store them securely NO
Takeaways:
Study with deception and replication with announcement yielded same results
=> Deception not always necessary
Instruct to use security guidelines! If not known prompting does help aswell.

Question 5

Code Reviewing as Methodology - Setup

Answer 5

Let participants review code instead of writing yourself.
RQ1: How do developers behave when reviewing code in a security-critical task such as password storage?
RQ2: Which factors have an influence on developers’ security awareness?
RQ3: How much time do developers dedicate to security and do they feel
responsible for security in a code review?
RQ4: Comparing the results of a programming and a code reviewing task on
password storage, which methodological implications can we conclude?
IV1: Prompt half of developers to look for security issues.
IV2: Different password-storage issues (MD5, Base64, plaintext) in between-subjects study
Delete Comments and add generic comments to all code
Arrange function order two distraction task for all snippets

Question 6

Code Reviewing as Metholodgy Results

Answer 6

Half wanted to release insecure code. Prompting made a significant difference
Participants feel resposible but couldnt act accordingly.
Not suggesting to replace coding studies with code review studies.