Monocultures

This is a post about programming, but I am going to draw on industrial agriculture and its resultant monoculture to use as an analogy for software development, so bear with me as I take some time to lay down a common context.

The popular story is that subsequent to famine after famine, India embarked on a revolutionary program to modernise her agriculture sector. Scientific methods of agriculture replaced the centuries old methods that had remained largely unchanged for millennia. Yields skyrocketed and farm productivity soared in the Punjab. Famine has been banished from the subcontinent and now granaries are full to bursting. The official line is that the Green Revolution has been an unqualified success and it’s progenitor, Dr Norman Borlaug was even awarded the Nobel Peace Prize. (Of this last fact, you can make what you wish.)

On the other side of the argument are some pretty prominent environmentalists. In her book ‘The Violence of the Green Revolution’, Vandana Shiva sheds some highly unflattering light on the claims made by industrial agriculture. She does not argue that industrial monocultures do not yield more crop than conventional agriculture. In fact, she agrees wholeheartedly. Trouble is that crop yield is a very narrow metric to measure farm output by. Farms are not factories that churn out grains but are complex ecosystems which do not submit to a teleological approach.

Shiva argues that food crops are but one of the things that conventional farms produce, and the increase in yield is at the cost of these other things. For example, rice with thinner stalks yields more rice per acre but the chaff and stalks which used to feed the farm animals does not obtain any more and has to be bought for money by the farmer, making the farm and the farmer a little less self sufficient. The natural pesticides that used to grow in the fields do not grow any more and now industrial fertiliser needs to be used. Seed banks are useless as now only seeds resistant to the chemical pesticides can survive in this environment. The inevitable result is that what used to be a self-sustaining ecosystem has now turned into a monoculture that requires vast amounts of inputs from external sources in order to sustain itself. The self-sufficient farmer has become dependent on forces outside his control. Monocultures are less resistant to pests and climate change. The whole system, in effect, works until the day it doesn’t at which time it collapses hard. Never bet against the second law of thermodynamics, it seems.

Which of these is true? Which is better? These are all questions that can be endlessly debated. What should be reasonably uncontroversial is that when one choses one over the other, one should do it with full knowledge of the choice. The danger is in choosing without knowing what one chooses. And it’s this unconscious choice that I’d like to address.

Just like agriculture has it’s 'fake’ metric of yield, the startup software delivery mechanism is also measured in similar terms. Things like lines-of-code are of course too barbaric to merit much discussion these days, but the more dangerous metric in startup-land is, of course, time-to-market. We’ve all heard these old saws before - Startups win when they get to market first. No startup ever failed because of bad code. Get Shit Done. Ship it. etc. etc.

All very well, but who stops to consider the cost of a feature? At one client, the outsourced developer team said they’d have X feature set 'ready by Friday even if we have to work all night’. And who, my friend, is going to maintain the code you wrote at 3 am after 5 straight 16 hour days? Startup product teams think they are selling kitchen appliances. Actually they are more like Agriculture Minister of a developing country in a time of climate change. When you don’t know whether you will have a drought or a flood next year, when you don’t know whether temperatures will be three degrees above or seven below long term averages, what do you optimise for? The answer seems to be obvious - resilience. You build the farm sector that works under the widest variety of possible outcomes. Yet when delivering MVPs, startup product owners often worry more about specific feature sets than about their ability to deal with change. And this is where the danger of the monoculture bites.

If you have little money, many competitors, low barriers to entry and a shifting technological landscape, will you still optimise for a given feature set? Or will your optimisations be more general? Optimise perhaps for team cohesion over team size? For feature ownership rather than feature delivery? For sustainability of processes over lines of code written? For reputation over outcomes? For adaptability over expertise? For relationships over transactions?

Are you playing with new languages and platforms or is it ruby ruby ruby rails rails rails all the time? Are you experimenting with Service Oriented Architectures or is it Rails monoliths all the time. How many different message queues have you experimented with? How many conference videos did you see last week? Did you get time to sit quietly and just think about that tough problem? Did you attend a javascript meetup where they talked about testing front-ends? Have you booted up a raspberry pi…..

Time to market is important. It’s just not the only important thing. Here are some other things that are important

  • developers that stay on the team a long time - developers are not fungible and good developers that stay on a team for a long time can and do contribute heavily. Are you optimising for keeping your best developers around for ever?
  • side projects - “hey boss, I just deployed my side project this weekend and Capistrano 3 works great. Let me take our legacy deploy system and upgrade them.” Is this a conversation worth having? You can’t have it if you’re working 100 hour weeks.
  • “We need a new internal facing webapp for customer support. How long will it take?” - “Two weeks”. - “Take four. Build it in clojure.”
  • “Take the rest of the day off from putting code into production and prepare your slides for the local javascript meetup”

Most of the imaginary conversations above revolve around taking the focus off code in production and focussing instead on the processes that lead to good production code. Effective software development teams, like farms, are complex ecosystems that do not yield to teleological approaches. Code is just the stuff that sustains startup revenues, but if your development processes start becoming monocultures, you will need increasing amounts of energy to keep it productive. Project managers, quarterly reviews, Gantt charts and so on will eat up your resources much more surely than giving developers time to explore new technologies and sending them to conferences.

Startups fail for a variety of reasons. Getting to market second is the biggest fear of most entrepreneurs, but they’d do better to develop a healthy fear of developer churn and excessive managerial overhead.

As my final point, I’d like to point to HelpShift, a product built by a company initially known as InfinitelyBeta. I only have an outsiders view on events, and what happened is well known - they started with a product called paisa.com, which failed. They then iterated with a help-desk system which pivoted into a help-desk solution for mobile apps. This happened over a period of 4 or so years. In all these years, never once did I meet a stressed developer, never once did I hear of 'discipline’ being imposed, of Gantt charts and product managers and so on. Notwithstanding early failure, investment from well known funds, setting up offices in the Valley and eventual success, I never felt any change in my friends at HelpShift. Always cool, always more interested in the path than the outcomes, generally not worried about specific instances of success or failure and instead working to build an organism that thrives in many environments. As far as I can tell (I’m no insider btw), they seem to have invested into building an awesome team of A+ developers that bring a wide variety of skills to the table and can adapt to changing circumstance.

So, when you choose to 'hire 20 developers and churn out features fast’, know what you’re buying into. Monocultures can last a long time and even help you win. But they are fragile. And over anything more than the very short term turn out to be very expensive.

Hacker News Link