Debunking the 10x Productivity Myth

You can’t work in this industry for very long before being confronted with the “fact” that super-productive programmers are an order of magnitude faster than you.

(Seriously, how many geeks hear this statistic and rejoice? Finally someone has quantified my amazing ability! No. Nobody does that. Everyone despairs.)

There is good news, however. Upon hearing the 10x number again recently, I started Googling and discovered that, lo and behold, it’s bullshit! The original Sackman study, despite being cited by Fred Brooks and Steve McConnell, is actually a deeply flawed assesment of productivity. Subsequent research indicates that the best performers are merely a binary order of magnitude (just 2x) better.

For more info, check out this great series of articles over at Best Webfoot Foward.

PS: I’m curious about something that I haven’t seen a study directly address so far: what is the typical productivity difference between a person’s best performance and that same individual’s worst performance?

Advertisements

8 Responses to “Debunking the 10x Productivity Myth”


  1. 1 Kaitlin Duck Sherwood April 8, 2007 at 2:43 pm

    Not that I think it probably isn’t even 2x — the studies show it’s ~2x between the best and the median FOR ONE TASK UNDER CONTROLLED CONDITIONS. I bet that when you have a bunch of tasks over time, the difference is lower. I bet that if you look at an uncontrolled study — when the amount of time each programmer actually puts in is much more to their own discretion — then you have another confounding factor.

    Note that the slowest programmer can be infinitely slower than the median. I am infinitely faster than a fruit bat, for example. So the biggest message I take away is that getting stuck is very very bad.

  2. 2 Mark April 8, 2007 at 7:01 pm

    Thanks for visiting Kaitlin. I dig your blog.

    Isn’t it true, also, that the biggest observed difference has been on the sort of ground-up programming problem none of us have done since university? Not a very realistic scenario.

  3. 3 Kaitlin Duck Sherwood April 23, 2007 at 3:23 pm

    Yes and no.
    Ko’s study (which reported the number of “actions”, not the time) had five tasks which included some bug fixing and some enhancing a small (~500 LOC) paint program.
    Robillard’s study had people modify the autosave feature of JEdit (an existing large code base).
    The Sackman study and the DeMarco-Lister studies do appear to be code-from-the-ground-up.

    Note that these studies also don’t try to account for how many hours each coder actually codes per day. I suspect that there is enormous variation in that as well: some people work 18 hrs/day; some people spend 7.5 hours at work, of which 7.25 are at the water cooler. (I think trying to factor the hours-per-day metrics in right now is too challenging! I want to just look at the productivity per coding hour right now.)

  4. 4 Kaitlin Duck Sherwood May 9, 2007 at 12:45 pm

    Mark —

    I’ve decided that it’s even harder to draw conclusions about programmer productivity based on one test than I was thinking.

    I recently watched two different people navigate through code looking for the same thing. There was one place where there were three obvious candidates for further exploration. One person chose to follow the calls on line A; the other chose to follow the calls on line B. A led to the interesting method, while B did not.

    The person who followed A took ten minutes; the other person wasted 30 minutes off in the weeds, chasing down a dead-end. The interesting thing is that BOTH A and B were reasonable choices.

    This might argue for a more “breadth-first” search. Furthermore, problems that are selected such that people can finish them in the time alloted for a typical user study would definitely be biased towards that. However, in real life, maybe sometimes a depth-first approach would be better. I suspect that you really want to have some people around who singlemindedly dog a trail until the bitter end, who don’t get distracted or discouraged.

  5. 5 Mark May 26, 2007 at 3:26 pm

    Kaitlin,

    I just re-read this post, and I was suddenly struck by something you said in your very first comment: “So the biggest message I take away is that getting stuck is very very bad.”

    That totally feels right.

    When I am *most* effective, I always have — either on paper or in my head — a whole series of “next steps”. Some steps on my list are sets of alternatives. In other words, “give X the old college try, but if that doesn’t seem to work, try Y or Z.”

    When I am *least* effective, I find myself wondering what I should do next. I run out of options. I spin my wheels. I get stuck.

    I think this is a really killer insight.

    –Mark

  6. 6 Paul May 26, 2007 at 7:30 pm

    My rule of thumb on programmer efficiency is a great programmer can knock out 10,000 lines of code per year. Take two and ask them to cooperate, you are lucky to get 5,000 lines of code per year jointly.

    Programming is a team effort for all but relatively trivial projects and standouts with 10x numbers don’t usually take to scaling very well.

    Most managers and analysists like myself just aren’t good enough to come up with 10,000 line capsules for lone wolves. I’m lucky if I can state requirements clearly. I wouldn’t get all worked up over “productivity” numbers. It’s still a Dilbert world out there.

  7. 7 Nathan Zook June 11, 2007 at 6:01 pm

    Well, I’m not the least bit ashamed to admit that I’m more than 2x as productive today than I was ten years ago. I hope to cover that again in the next ten–and I consider myself to be well above average.

    Productivity comes from a number of sources, and you’re not going to measure some of them easily (or cheaply–important consideration) in the lab.

    As mentioned hours spent thinking about a problem is a big deal. Those of us who get irritated at bosses who won’t shell out for laptops so we can code from home (just to get the fscking problem out of our mind) from time to time are going to look a lot better than those who actually have lives.

    But the A route vs B route is killer. Going down a wrong path can and will regularly add hours, days, weeks, or even months to a project. A good programmer will have fewer of these, and they will be of the shorter variety.

    It gets even worse. A poor programmer is much, much more likely to hit a cognative limit and be effectively incapable of making progress past a certain point. Really good programmers never do. Not that they are necessarily that much more intelligent, but that they are sufficiently good at organizing the problem space that they never hit their limit.

    A what about the productivity of a coop who requires more oversight than it would require for the overseer to do the project himself? I’ve had one of those. I know of a geographic region which is notorious for producing that sort of programmer.

    Are elite programmers 10x more productive than average? Perhaps. Especially if you ignore that many or most are geeks with no real lives.


  1. 1 Programmer Productivity, Again at mark++ Trackback on March 27, 2008 at 9:22 pm
Comments are currently closed.




%d bloggers like this: