In our case? Call it ~6 hours, over the course of several (potentially discontiguous) days, offsite and at the candidate's leisure.
Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE).
We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too.
Some other things to know about that guy:
* No prior crypto experience
* No prior software security experience
* Less than 2.5 years dev experience across both jobs
* From flyover country
* Physically uncomfortable during the interview (something noted by interviewers on his feedback)
How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess.
Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf.
Yes, I think someone is going to make a lot of money correcting that market inefficiency.
† If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.
If this comment shows up hereafter on every "developers need passion" and hiring thread, I'll up vote it. every. single. time. Alternately, a blog post expanding on the thesis, if we could draft you to write it...
Wait -- there are two questions here. One is whether developers need passion, and the other is whether an interview process is a good way to tell whether they have it (and the other qualities required -- I haven't seen anyone say that passion is sufficient). I think most of us are speaking to the former -- I certainly have been.
I agree that work sample tests can be quite valuable.
Work sample testing has been extremely powerful for us. In particular: it has enabled us to screen candidates for aptitude instead of experience, and the result has been a series of hires with no prior experience in our (very specialized) field who have completely blown us the fuck away, like the former line-of-biz insurance company .NET guy who, a month or so in, looked at Rails for the first time and found a serious security hole (which now has a CVE), or the other line-of-biz .NET guy who now runs our crypto review board and co-wrote our crypto challenges, or the math major we hired out of college who ripped through the MSP430 CTF in a couple days (and also has a Rails CVE).
We picked up a guy who had done a couple months of Android development after college, and then moved to what I think (from his resume) was a devops job --- I never asked him in the whole course of our hiring process what that job was about, so I don't know. Six months later, he implemented a CRI elliptic curve DSA exploit that involved BKZ lattice reduction and an FFT filter step. The lattice reduction steps on that attack took something like SIX HOURS to run, and when they finished, you still had to get the FFT step working to figure out if you got the BKZ step right.† Months later and I'm blown away just thinking about how he could have gotten that working. In case you're wondering, he kicks ass on the more boring parts of our work too.
Some other things to know about that guy:
* No prior crypto experience
* No prior software security experience
* Less than 2.5 years dev experience across both jobs
* From flyover country
* Physically uncomfortable during the interview (something noted by interviewers on his feedback)
How compelling do you think his "greatest achievement so far" story would have been? We weren't dumb enough to ask, so we'll have to guess.
Before we did work-sample testing, overcommunication about hiring process, demoted the importance of phone screens, and standardized face-to-face testing, we never would have hired these people. Any process that filters outs Seans, Alexes, Peters, and Ryans over things like "passion" or "what is your greatest achievement" is a stupid fucking process. But there it is, enshrined as the default process for our whole industry. And here's a thread full of people arguing on its behalf.
Yes, I think someone is going to make a lot of money correcting that market inefficiency.
† If you're wondering, number of other people on our team who could have helped him debug lattice reduction C++ code: zero.