Lecture 5 Flashcards
Recruitment of Software Developers not easy
Github, directly from companies, students, programming groups
Online services: Clickworker, Amazon Mturk, Qualtrics
Screening task
Use screening task to filter out non programmers
Paper “Do you really code?” Goals
Filter out non developer with quick questions that real programmers can solve:
Effectiveness (decide between programmers and non programmers)
Efficiency (fast)
Cheating Robustness
Language independence (dont depend on certain language)
Paper “Do you really code?” Results
Some questions were filtering out actual devs and some not enough non devs
Best: Source.Usage, Function.Param, Boolean, Recursive
Also time to solve is a factor.
Questions with code did well.
Paper “Do you really code?” Attack
Non programmers get money for each completed question with explicit online help.
Paper ‘‘Testing Time Limits in Screener Questions’’ Goals
Analyze how time limits allow for screening:
3 Phases.
1 Base: To compare to no time usage study
2 No Limit: Ask to solve as quickly as possible but no time limit
3 Countdown: Use countdown for each task
Paper ‘‘Testing Time Limits in Screener Questions’’ Results
Correctness between the three approaches was mostly similar.
=> Time usage was different => Choosen countdown
Paper ‘‘Testing Time Limits in Screener Questions’’ Attack
Sources Usage, Recursive, Boolean, IDE.Known, Prime, Array tasks where most efficient
Paper “ChatGPT-Resistant Screening’’
May use pictures to disallow copy paste but Chat GPT still breaks old tasks.
=> new questions that resist chat gpt
Paper “ChatGPT-Resistant Screening’’ Results
Programmers with ChatGPT were better in almost all tasks
Non programmers very similar
Three adjusted tasks from before were recommended
Use of ChatGpt increased time pressure
Recommended: Compiler, Recursive, Detect-Funcname, Queue, Websites, Detect loop
Surveys
Can use limited open questions, be clear etc
