- Some developers are 10 times as productive as others. That’s one factor of 10.
- 50% of software projects fail. That’s a factor of 2.
- Bigger teams are less productive. Let’s assume a 16 people team is a factor of 4 less productive than a 1 person team.
- 80% of software projects are not actually aligned with the market; only 20% of software projects really build the right thing. Due to competition the eventually get replaced by better software. That’s a factor of 5.
Multiplying the multipliers, taking the worst case everywhere, the productivity of your team is 400 times less than in the best case scenario.
Add a few statistics, and then it follows that one successful scenario could be to hire 99 one-person teams working on semi-random solutions, and 1 person to pick the right solution out of all of those. It’s quite likely most of those solutions are not worst-case scenarios. You would probably still be about 4 times more productive than in the worst case scenario.
The math above is easy to argue with, but it’s the general principle of having extremely skewed productivity that matters.
Applying this principle is sometimes called software darwinism, which is a term sometimes used to explain the success of open source. It’s probably also one reason why companies like google and microsoft are as consistently successful as they are.
The silver bullet in this story would be a way to somehow measure and predict which of those many teeny tiny tiger teams will end up doing the really successful project. If you are really good at betting on the right startup, and also get really really lucky, you might just get back 400 dollars for every dollar you invest. The odds are really tempting. This is why so much of the software industry is controlled by venture capitalism, and this is why we have bubbles like the dot com bubble or the web 2.0 bubble.
Software development, then, is something that attracts cowboys. I do hope that someday software engineering will consistently and provably offer better ROI than software cowboyism, but I’m afraid that’s empirically not true today. You cannot be in the software business today without accepting that you are part of an ecosystem is full of cowboys, some of whom are quite successful.
LSD::RELOAD 
I wouldn’t say that Microsoft’s consistent success is down to darwinism. They certainly know how to manage software products better than most, but their success can in large part probably be attributed to their ability to copy good, but flawed products and make them better as well as their already dominant position which they leverage.
They’re sort of like the equivalent of that bastard in 8th grade who copied your French homework **AND** got a higher mark than you did.
Google I would not say is consistently successful. They’ve had one hit – search, and have developed a number of other apps from which they have mostly failed to derive any significant revenue (significant compared to what they get from adwords at least).
Both companies copy an awful lot from small startups (which I consider to be the true drivers of innovation in this industry), as well as buying them out. Both have served them well.
I guess maybe you’re right – it’s a form of capitalistic darwinism, but with the ultimate beneficiaries being Google and MS.