software darwinism and technology bubbles

  • Some developers are 10 times as productive as others. That’s one factor of 10.
  • 50% of software projects fail. That’s a factor of 2.
  • Bigger teams are less productive. Let’s assume a 16 people team is a factor of 4 less productive than a 1 person team.
  • 80% of software projects are not actually aligned with the market; only 20% of software projects really build the right thing. Due to competition the eventually get replaced by better software. That’s a factor of 5.

Multiplying the multipliers, taking the worst case everywhere, the productivity of your team is 400 times less than in the best case scenario.

Add a few statistics, and then it follows that one successful scenario could be to hire 99 one-person teams working on semi-random solutions, and 1 person to pick the right solution out of all of those. It’s quite likely most of those solutions are not worst-case scenarios. You would probably still be about 4 times more productive than in the worst case scenario.

The math above is easy to argue with, but it’s the general principle of having extremely skewed productivity that matters.

Applying this principle is sometimes called software darwinism, which is a term sometimes used to explain the success of open source. It’s probably also one reason why companies like google and microsoft are as consistently successful as they are.

The silver bullet in this story would be a way to somehow measure and predict which of those many teeny tiny tiger teams will end up doing the really successful project. If you are really good at betting on the right startup, and also get really really lucky, you might just get back 400 dollars for every dollar you invest. The odds are really tempting. This is why so much of the software industry is controlled by venture capitalism, and this is why we have bubbles like the dot com bubble or the web 2.0 bubble.

Software development, then, is something that attracts cowboys. I do hope that someday software engineering will consistently and provably offer better ROI than software cowboyism, but I’m afraid that’s empirically not true today. You cannot be in the software business today without accepting that you are part of an ecosystem is full of cowboys, some of whom are quite successful.