Participant Recruitment Flashcards
Software Developer Requitment Motivation
Some studies cant be conducted with CS students because they lack context
Platforms to reqruit software developers exists but unclear how and where requitment happens
Important facts about study often not mentioned
Recruitment and Solana Setup
● RQ1: How do study factors affect the motivation of company developers to
participate in security studies?
● RQ2: How and where do company developers prefer to be contacted for
participant recruitment?
● RQ3: Which concerns do company developers have with study data collection
Presurvey for filtering, 30 semi structured interviews, post survey
Agreed on codebook by discussion (using intercoder agreement): R1 R2 coded 15 interviews, R1 R3 15 interviews
Recruitment and Solana Recommendations
● Pay participants appropriately and value their time
● Provide as much information as possible during recruitment
● Address different types of motivation in the study invitation
● Data collection and trust
● Conduct your study online and keep your survey short!
Active > Passive Recruitment, Allow for asynchronous communication, allow to make it personal, recruit through trusted source, social media, study platform
Engaging Company Developers in Security Research
Studies - Motivation
Lack of Data: very little reported on how to recruit
Need for Insights: need to know what motivates to participate
Engaging Company Developers in Security Research
Studies - Setup
● RQ 1: What factors influence the security study participation of company
developers?
● RQ 2: Are there differences in attitudes toward study participation between
first-time and repeat participants?
● RQ 3: Compared to existing recruitment platforms concerning participants’
programming and security experience, skills, and knowledge, is Qualtrics
suitable for recruiting company developers?
Reviewed Literature to identify research gaps
Quantitive Survey: Recruit 340 professionals
Engaging Company Developers in Security Research
Studies - Results:
Most effective: Targeted emailing (4/5) and recommendations (4.5/5)
Least effective: Unsolicited emailing (2.5/5)
Mean study length prefs:
- Surveys: 25.61 minutes
● Interviews: 31.56 minutes
● Implementation Tasks: 56.89 minutes
● Code Reviews: 56.61 minutes
More time on practical tasks.
Payment: survey and interview < lab and field < coding and review
EFA:
Motivators
● Altruism: High importance placed on contributing to broader security
knowledge
● Transparency: Clear communication about the study’s purpose and data use is
crucial
● Personalization: Acknowledge participants expertise and address them
personally
● Personal Development: Opportunities for skill enhancement and knowledge
gain are strong motivators
Barriers
● Commitment: Significant concern about the time and effort required
● Privacy: Participants have strong concerns about how their data will be
handled and protected
● Time Constraints: Many participants are constrained by their professional
responsibilities and limited free time
● Uncertainties: Study participation can be daunting. Some participants want to
know exactly what to expect before participating
Security Attitudes
● Responsibility: Participants feel a strong sense of duty towards maintaining
security
● Risk Awareness: High awareness of potential risks and the importance of
security measures
● Task Difficulty: Security tasks are perceived as challenging but important
Results for Qualtrics Reqruitment
Upsides:
Better security proficiency, higher female rate, easy and efficient to use, replaced removed or discarded participants
Downsides:
Longer time, high rejection rate, very expensive
Engaging Company Developers in Security Research
Studies Takeaways
Need to be clear about study goals and offer potential for participants development
Compensation: Fair compensation required, higher pay for harder tasks, shorter more flexible tasks prefered
High willingness to participate (96%), many people prefer security tasks
Lack of opportunities
Solana: Defying the Odds Setup
● RQ1: Do Solana smart contract developers recognize prominent security
vulnerabilities in smart contracts?
● RQ2: What challenges do developers encounter that impact the development
of secure smart contracts?
● RQ3: Given these challenges, what is the prevalence of vulnerabilities in
Solana smart contracts?
Code review of rust based marketplace study.
Post task questionaire => Interview
Solana: Defying the Odds Recruitment
Over multiple platfroms, 55 did survey
Problems: Spammers(identified by giving wrong information), bots(detected by qualtrics)
Solana: Defying the Odds Analysis
Thematic analysis using MAXQDA, independent coding, code system based on interview data
Procedure: Discussions to refine codebook, Final codebook included 400 codings
No participant found all security weaknesses:
RQ1: Only 20% found atleast one security vulnerability, 82% would have released vulnerable file
RQ2: High demand leads to young less experienced devs wanting to earn money
Testing and reviews often neglected, Rust is hard
Solana devs prioritize functionality over security, rely on anchor and dont understand rust,
