Perfection...
...and how to avoid it
In 1795, Laplace, the French polymath, published an essay on his Analytical Theory of Probabilities. In the second chapter, he writes:
We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.
— Pierre-Simon de Laplace, Essai philosophique sur les probabilités
Today, we know that this is false: you cannot know both the position and the momentum of an object.
But the idea that it is possible to capture enough information about the state of something to accurately predict its future state, that you can determine all the parameters of something, this idea seems almost primal in us. We need to feel we have that control. And, to be fair, most of the time it works, at least to the degree of accuracy we need.
But it cannot work when we don’t know the rules. Even ignoring quantum effects, Laplace’s calculations about the state of the universe would gradually drift away from reality thanks to the effects of relativity and cosmic expansion and the limitations of numerical accuracy.
And if there’s one area where we most certainly do not know the rules, it’s software development.
Model It, and We’re Done
During the 1980s and 90s, the folks who managed the construction of software were obsessed with creating models. From SADM in the 70s to OOAD in the 80s and UML in the 90s, we were never short of tools that would help us model the problem, its domain, and our application. And model it we did. It was not uncommon to walk into an office and face an entire wall covered with diagrams: data flow diagrams, entity-relationship diagrams, OMT diagrams, and the 14 different types of UML diagram.
…twenty seven eight-by-ten color glossy photographs with circles and arrows and a paragraph on the back of each one explaining what each one was…
— Arlo Guthrie, Alice’s Restaurant
All these diagrams were produced at the start of the project. The feeling was that the system could be modeled so accurately, and to such a level of detail, that the code required to make it real could be mechanically generated, either by the lowest of the low, the coders, or by machine.
The trouble is, it didn’t work. Software development got worse and worse; by the mid 1990s, the Standish group reported that only one in six projects were delivered on time and to budget, with all the required functionality.
Yup—you were betting that $12M project on the roll of a die.
The fault, of course, is that no model is one-hundred percent accurate; the map is not the land. But the teams kept trying to make their models more accurate, more detailed, more perfect.
And this constantly disappointing drive made projects worse.
To search for perfection
Is all very well
But to look for heaven
Is to live here in hell— Sting, Consider Me Gone
Why weren’t the models perfect? In retrospect, the reasons were obvious:
Most of the time, people don’t actually know what they need.
The model is not reality
You Can’t Always Want What You Need1
What people think they want and what they actually need are often very different. Back in the old days, I wrote specs, the client signed off on them, and I delivered software that accurately implemented them. It was disheartening the number of times we presented the client with software that exactly mirrored their stated requirements, only for them to say “yes, that’s what I said I wanted, but it isn’t what I need.”
A while back, we remodeled a bathroom. “What size floor tile do you want?”, asked the contractor. “Ummm…",” we replied. We talked for 20 minutes about options, and we were no closer to having an answer. He gave up, and the next day turned up with a couple-of-hundred pounds of tile samples which we lay on the floor until we could understood the various parameters and make a choice.
No matter how perfectly you model represents the clients stated requirements, no matter how many iterations you go through, and how many sign offs you get, it won’t capture what they actually need. Because they don’t know.
The Model is Not Reality
“On Exactitude in Science” by Jorge Luis Borges, translated by Andrew Hurley:
…In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other relic of the Disciplines of Geography.
Any model that is perfect would, like the ultimate map of the Empire, be as big and as complex as the software it was describing.
And, like the map, it would just be a picture, not the real thing. The client couldn’t experiment with it, or use it with live data to see if it behaved as they needed. It would just lie there, doing nothing.
And Along Comes…
Agility was a reaction to this search for perfection. The manifesto’s values were informed by the fact that this is an imperfect and ever changing world, and the only reliable way to deliver value is to make mistakes and iterate them towards something usable.
But that’s another story.
Ring the bells that still can ring
Forget your perfect offering
There is a crack in everything
That's how the light gets in.— Leonard Cohen, Anthem
With apologies to the Rolling Stones.