News Flash: Tech Interviews Don’t Work

There’s been a lot of buzz around the blogosphere lately about various issues with interviewing programmers.

One of these days I’d like to do a study. I’d find a very brave employer who needs to hire 100 people. We would use traditional interview techniques to fill 50 positions. For the other half, we’d pluck reasonable resumes from a pile, choose 50 at random, and just hire them.

After a year, I would return and talk to management. My guess is there would be no significant difference in the performance of the two groups.

If you are anything like me, you’ve been on both sides of the “hiring table” enough times to know that my hypothesis is probably right. There are just too many ways that an interview can fail.

If you don’t ask any tech questions at all, you can end up hiring someone who literally cannot code. Most employers are terrified of these “false positives” and so, instead, they make their interviews highly technical. They only hire people who can remember, on the day of the interview, how to solve the dining philosophers problem, or how heapsort works. However, nerd brains have aggressive garbage collectors which quickly expunge unused information. In English: they forget stuff. This leads to a high “false negative” rate — which is a more insidious problem because it tends to inhibit diversity.

A team with 5 heapsort experts is likely to suck for the same reason that a rock band with 5 drummers would suck. But I digress.

The typical whiteboard question is highly ego-centric. We go into a little room with a question that we’ve asked a dozen times and understand fully. We pose this question to a “lesser” nerd, in a high pressure environment, and carefully note all the missed semicolons. We chuckle to ourselves, secure in our supreme geekhood.

Ego deflation alert: 100% of these candidates could turn around and immediately ask an “interview question” that we could not answer. Oops. Who’s the idiot now?

By focusing on the whiteboard, interviews typically fail to evaluate candidates on many orthogonal — but critically important — axes. For example, how well does the candidate learn? How well do they teach? Do they ask good questions? Are they able to speak and write clearly?

Almost regardless of what questions you ask, there are a number of things that your interview cannot possibly measure. These are things which require months, not hours, to ascertain: work ethic, initiative, team skills, personality, team compatibility, self-motivation, ability to compromise, creativity, etc.

I think the only way to fix this system is to restructure the entire employment process. The root cause is that it’s currently way too hard to fire someone. If professional programmers were more like professional athletes — hired on a contract basis for a specific duration — things would be much different.


13 Responses to “News Flash: Tech Interviews Don’t Work”

  1. 1 John Knox February 28, 2007 at 3:00 pm

    You could write a book about this: A Random Walk Down Geek Street.

    You’re dead right that programmers/engineers tend to latch on to “ego-centric” interview questions.

    Unfortunately, the alternative to puzzles and whiteboard programming seems to be a bunch of mumbo jumbo interview questions.

    Questions like “Lets say your program doesn’t work. What would you do?” The answer he was looking for was “You should conclude that the hardware was broken.” True story. Of course, this was at a company that spent most of their eight and a half hour interview telling me how smart they were.

  2. 2 Mike March 1, 2007 at 12:02 am

    You can have a technical interview without it being like you describe. We ask candidates to do technical problems not to see if they memorized a heapsort, but to see how they think.

  3. 3 Occasional Interviewer March 1, 2007 at 5:54 am

    Well, the point of the recent discussion is that you can filter out quite a few candidates with a very low bar–FizzBuzz requires a ‘for’ loop and preferably a modulus operator, which isn’t a lot to ask. And yet I’ve seen candidates who would fail it.

    Slightly harder screening questions involve trivial problems with pointers and/or recursion. Again, these questions filter out large numbers of candidates, many of whom have entirely plausible resumes (and who sound fine in a phone screen). About 75% of people who make it to an interview will sit there for 10 minutes or an hour trying to solve some trivial problem. The successful candidates solve it in about a minute.

    Do you _really_ think that “Hey, write some code to reverse a linked list” is too much to ask? Do you really think we’d be better off hiring people who can’t? Because the candidate pool is full of these people, and they’ve figured out how to write decent resumes.

    I do agree with your larger point, though–really hard technical questions will eliminate too many competent candidates, because everybody can answer a different set of questions.

    (Of course, if a candidate claims guru-level skills in a particular area, then I’ll ask harder questions. Got a PhD in distributed systems? Dining philosophers it is, then.)

  4. 4 Mark March 1, 2007 at 6:58 am

    No, I’m not actually suggesting that we not interview, or that we not ask simple technical questions when we do. I was just lamenting the fact that, no matter what we do, it won’t work very well.

    I mean, look, I’m a professional programmer. I don’t use — let alone reverse — linked listed on a regular basis. In an unfamiliar environment, under high pressure, I could easily screw up that question.

    When I do, you certainly haven’t “seen how I think.” Maybe you’ve seen how I *panic*. Is that relevant?

  5. 5 Occasional Interviewer March 1, 2007 at 9:39 am

    “Maybe you’ve seen how I *panic*. Is that relevant?”

    Well, for some jobs. :-/ Even in the best-run shrink-wrap companies, you can discover a crashing C++ bug within 48 hours of the gold master. So being able to solve pointer problems in a state of total panic isn’t _entirely_ irrelevant, sadly.

    But “Please reverse a linked list,” is really a way of asking, “Please prove to me that you can easily cope with recursive structures that involve pointers.”

    Programmer aptitude tests show an interesting “double hump”. For example, the AP Computer Science AB examination used to show the following breakdown (higher is better):

    1: 15% 2: 15% 3: 30% 4: 10% 5: 30%

    The 1s and 2s will never make it as programmers. The 3s can kind of muddle along, but won’t get any placement credit when they get to college. The 5s can eat pointer problems for breakfast, and they’re generally more productive across a huge range of tasks.

    When I’m interviewing system programmers, my goal is to screen out everybody who isn’t a 5. We could just give the AP exam, but that takes over an hour. So instead, I ask candidates to implement a simple algorithm over the simplest possible data structure, in the language of their choice.

    I don’t think it’s unreasonable to expect candidates to answer an easy second-term CS problem in a programming interview. This is more-or-less standard at Microsoft, Google, Amazon and Yahoo, so it shouldn’t come as too big a surprise.

    If you know you can do this stuff, but you’re rusty and you’re concerned about screwing up under pressure, I’d recommend a good AP CS study guide or a book like “Programming Interviews Exposed”.

  6. 6 Mark March 1, 2007 at 10:32 am

    I wasn’t aware of the “double-hump” — that’s interesting. Thanks.

    I have a sort of ethical problem with studying for interviews, but I don’t deny that it works. Thanks for those suggestions as well.

    If there’s a standardized test you can give candidates which correlates well with on-the-job performance, I’d highly recommend giving it. I’ve heard that IBM used to do this. Color me skeptical.

    I suspect that the typical on-site for MS, Google, Amazon or Yahoo is a lot more difficult than you suggest. It all goes back to the nerd ego problem. We love to ask hard questions for which we already have answers.

    For example, this one is used by some on the MS C/C++ compiler team:
    Sub Array With Maximum Sum

    Switching gears, here’s something to ponder. What is more important on the job: effort, or intelligence? This is an unfair question because, clearly, they are both important. However, I’d take a hard worker over a genius any day. Geniuses tend to be assholes, and I hate working with assholes 🙂

  7. 7 Occasional Interviewer March 1, 2007 at 1:07 pm

    Well, I think it’s perfectly fair to review core CS classwork at any time, interviewing or not. 🙂

    More seriously, if the candidates are all up to speed on the basics, it prevents competent candidates from stressing out and bombing on a question they could easily answer.

    I don’t much like the “subarray with maximum sum” question. It requires a non-trivial insight into a specific problem, and testing for that sort of thing in an interview is hit or miss. Another question I don’t like is “implement malloc”, because there’s a couple of tricks that you need to discover to find a simple solution.

    Of course, modern optimizing compilers do involve a lot of hairy algorithms and non-trivial math, so “subarray with maximum sum” may be a fair question for the MS compiler team to ask. I mean, compared to register allocation, that’s a walk in the park.

    Still, my preferences run towards basic algorithms over recursive structures. They’re all part of the standard CS curriculum (so there’s no secret trick), and they do seem useful in distinguishing the stronger programmers.

  8. 8 Mark March 1, 2007 at 1:51 pm

    We can agree on your first point for sure. I love autodidacticism.

    I think FizzBuzz or simple recursion are fine high-pass filters. Just don’t expect to avoid problem hires. Some of the “most selective” interview processes I’ve seen still result in folks who *literally* sleep on the job.

  9. 9 John Knox March 1, 2007 at 2:51 pm

    > What is more important on the job: effort, or intelligence?

    To a large extent, I think a hard worker should get preference over a smart person. What is easier to change: intelligence or work ethic and persistence? Studies show that you can improve your intelligence. I’m not sure about work ethic though.

    So how do we interview for hard worker?

    It seems like a trick, but I suppose the interviewee could be interrogated about their knowledge of what the interviewing company does or sells. Perhaps a hard worker would be interested in the company’s needs while non-hard workers would just be interested in brushing up on pointers and recursion the night before the interview. I’m just making this up…

    Volunteer, open source, and entrepreneurial buzzing on the resume would probably indicate hard workers as well.

  10. 10 Occasional Interviewer March 1, 2007 at 3:09 pm

    Definitely! There’s a lot more to hiring than just technical screening.

    Thank you for this excellent discussion.

  11. 11 mikee March 1, 2007 at 5:09 pm

    Mark – you are spot-on with your observations. I’ve never found it very productive asking or answering geek-trivia questions for an interview. I’d love to see a broader set of questions that help categorize candidates in terms of personality traits and likely strengths/weaknesses. Didn’t the military used to give tests like this (and the promptly proceed to ignore the results)?

  12. 12 Mark March 24, 2007 at 10:27 pm

    For posterity, here’s a reference for the “double hump” that “Occasional” mentioned previously:

    The Camel Has Two Humps (Working Title)

  13. 13 Mark February 1, 2009 at 4:32 pm

    Programmers who misunderstand REQUIREMENTS cause a lot more defects in systems, than does poorly written code. In addition, spending the time to write better test cases with higher code coverage can highlight poorly written code faster. Shouldn’t interviews be more about how a programmer understands the mapping of requirements and test cases to the implementation? In fact, programmers often spend less time actually writing code than interfacing with users and business stakeholders to build the right solution.

Comments are currently closed.

%d bloggers like this: