I ignored my own advice, burned a day, and proved—once again—that
I am my own worst client.

Our new book distributor has a talent for inventing things to charge us money for. Dozens of them. We have to account for each of these, assigning costs to individual titles so we can correctly calculate royalties. Easy, right?

Except the “per-book” expenses show up on one statement, and the lump-sum deductions that we end up paying come on another. And each of these lumps can hide a grab-bag of individual expenses, none labeled. All we get is a couple of dates and one random-looking total. We then have to match that total to some combination of expenses.

In coding-interview lore, that’s the Subset Sum problem: take a pile of numbers and figure out which combinations add up to a given total.

It’s an NP problem; big, scary running time.

I knew that. And that was the trap.

Armed with my one dangerous fact, I set out to optimize. Pruning strategies. Grouping likely pairs. A truly deranged idea involving only the last digit of each number, because apparently I hate myself.

Hours vanished. Lunch went cold. I kept convincing myself the next trick would crack it.

At last I had something that looked promising, but I needed to be sure it caught all the subsets. So I wrote a brute-force version to test against.

I set it running and shoved my chair back, ready to walk the dog while it ground away for hours. Before I’d even stood up, the prompt blinked back.

That couldn’t be right—must be a bug. But the output looked fine.

I hammered it with property-based tests. All green. It finished in milliseconds.

I’ve stood on countless stages quoting Knuth’s immortal line: “Premature optimization is the root of all evil.” And then I went and sacrificed a day on the altar of exactly that.

Physician, heal thyself.

This last week, the Ruby world was jittery. What happened, though, has far wider ramifications.

The whole situation is still unclear, but here’s my current understanding.

The Ruby Central organization is partly funded by corporate sponsors. Some of these companies apparently became concerned about so-called supply-chain vulnerabilities, where malicious parties were injecting bad code into libraries.

It seems that maintainer access to the RubyGems archive was fairly informal. Over the years, access had been given to folks who needed it. Sometimes this access was still open even though the individuals no longer worked on gems. The financial sponsors of Ruby Central (quite rightly) viewed this as a threat, and demanded that Ruby Central take action.

Ruby Central had apparently been given a deadline. The day that deadline expired, they seem to have panicked. They moved the ownership of the RubyGems repository to their own organization, and they seem to have removed the access of maintainers who did not work for (or have a formal contract with) Ruby Central.

Some of those maintainers posted their displeasure, and the socialverse did what is supposed to do, overreacting and inflaming the situation.

I have to stress, this is all supposition, based on culling through other people’s posts.

It is clear that Ruby Central botched their handling of this. It shouldn’t have come down to the wire without having some public consultation, and the takeover of the repository should likewise have been explained as it happened.

But that doesn’t matter. A month from now, this will all be history. But that doesn’t mean we write this off as “geeks don’t communicate well *shrug*.” There’s an important warning buried in all the agita.

Linux powers 96% of the worlds one-million largest servers.

The modern web relies on JavaScript, and JavaScript relies on the NPM repository.

Shopify, Netflix, AirBnB, and many other household names have systems build on Rails. Laravel probably powers more sites than Rails.

This is all open source software, maintained by and large by unpaid volunteers.

These people are dedicated and largely unthanked. They put in hour after hour, adding features and fixing bugs. In a very real sense, these people keep the modern world running.

These people would never dream of adding malicious code to their projects.

But what happens if they decide it just isn’t worth it? Or what if the various foundations who are the stewards of larger open source projects accidentally mess things up?

What happens if the open source framework on which we build our palaces of code starts to rot? Do you have insurance? Because the open source model will change.

Open Source Insurance

Here are some things you can do to mitigate your exposure to these types of issue:

Do You Really Need It?

Before adding a dependency to your project, ask if you really need it.

Sometimes the thing your are adding is so trivial, you might as well just implement it yourself. The left_pad fiasco of 2016 is a great example. It was literally a dozen or so lines of code that left-padded a string, and yet thousands of projects decided to add it as a dependency rather than write it themselves. Then the developer deleted his code, and projects the world over stopped building.

Sometimes you add a dependency because it lets you do something cool: fancy UI affordances are a great example. Does the customer really need the functionality?

Do You Really Need All of It?

Ruby on Rails is a proud, large, monolith. A brand new Rails app installs over 700 dependencies, which means your project is depending on 700 separate projects over which you have no control or insight.

Sometimes you need the whole thing. But quite often, you can get by with smaller and more targeted libraries.

This isn’t just a Rails issue. If anything, front end JavaScript frameworks are even worse, often starting out with literally thousands of installed libraries.

Do You Have It in Version Control?

NPM suddenly goes dark. Ruby Central runs out of money and shuts down Gems. What happens the next time you check out your project to make a change and npm or bundle install 404s?

Maybe it’s time to run a local mirror of the stuff you use, or add your installed libraries to your project’s repo.

Is This Really Necessary?

Do I think that any major library repository will just vanish? No, I don’t.

But, it is still a possibility. And if that happens, will your project survive? Insurance is always a hard sell, but it’s often a prudent purchase.

And…

It is well past the time where we should have come up with a fair and transparent way of dealing with the burden of developing and maintaining the software that underlies our whole world.

Seems to me that open source contributions are a proof of work, and are a more worthy basis of value than some cryptographic bit juggling. Maybe there’s a new cryptocurrency lurking in there. Just sayin’.

Share

Subscribe now

Leave a comment

Some of the chatter over on LinkedIn rubbed me up the wrong way, so I posted the following:

The true measure of a software developer is being able to deliver value in the simplest possible way.

It isn't knowing the latest languages, frameworks, or techniques; it's knowing when not to use them.

It isn't delivering lots of sparkly new features; it's delivering the one feature that gives its users the leverage they need.

It isn't sitting behind a screen 16 hours a day; it's knowing who to talk to and building relationships.

It isn't being right; it's being an inspiration.

I thought it might be fun to dig into this a little.

… deliver value in the simplest possible way.

In the daily grind of producing code, it’s easy to forget that we’re not paid to program, or write tests, or stand in daily scrums; we’re paid to create something that helps someone else to something better. We’re paid to deliver value.

End users don’t care how many design patterns we used, or our test coverage ratio, or our velocity. They care that it works, that it was delivered in a reasonable time, and that, if necessary, it can be changed.

Which means it’s our job to discover where the value is, and to produce that value using simple code. That’s a skill founded in experience and confidence, and it’s not something AIs can currently do.

… knowing when not to use (languages, frameworks, and techniques)

If we are going to create simple solutions, we need to think hard about every technology we use: is it carrying it’s load? Sure, it’s fun and educational to try a new language, but is it necessary to do it on production code? We all love a good framework or library, but are they strictly necessary? Every external piece of code in a project adds dependencies and complexities, and each adds to the vulnerabilities of the app. Typical JavaScript projects now have thousands of dependencies, and each of those acts as a little anchor, holding you back when you want to change stuff.

…deliver the one feature that gives its users the leverage they need

Let’s face it: features are fun. All those little bells and whistles we add to our code are our way of showing off a little, of showing our mastery. But that’s not what our customers notice (or want). These people likely use your applications a lot: they know how to drive them and what to expect. They don’t need flair and features: these just get in their way. Look at some of the most popular systems on the Web. Wordpress admin screens look like they were designed by the same people who did Craigslist and Hacker News; in your face 1990s HTML. Doesn’t stop them being popular, does it?

…building relationships

Programming has never been about coding. That’s a good thing, because code is becoming a commodity. No, a good developer is an intermediary between the messy, poorly specified real world and the tightly constrained and unforgiving world of machines. The defining characteristic of a good developer is that they are comfortable in both of these worlds, and they know how concepts translate between them.

More and more, that role depends on interacting with people: customers, managers, other developers. Building relationships not only helps you do your job better; it most likely will help you find your next job.

It isn't being right; it's being an inspiration

In the end, no one cares if you’re right or wrong. They care about how you make them feel. They don’t need to be told what to do; thy need to see you doing it, and to be inspired to try it, too.

Am I wrong? Please comment,

Leave a comment

Subscribe now

Share

The magician leaves the apprentice with a chore: “sweep up the workshop.” The apprentice, now alone, starts the task, but soon gets bored. They use a spell they’d heard the magician cast that causes the broom to start cleaning on its own. But the broom soon gets out of control, spilling water everywhere, and the apprentice doesn’t know that incantation to stop it. Everything else the apprentice does makes things worse. Eventually, the magician returns and order is restored.

Goethe’s poem was written in 1797. The story is acting out today, 230 years later.

Be Careful What You Wish For

In the modern world, the coding LLM is the magic, the prompt is the spell, and the unfortunate developer is the apprentice.

All too often, we hear stories where developers invoke a prompt: “write a program that …” and the AI obliges. But the developer doesn’t understand what they’ve asked for, or what the AI produces. It’s all just magic.

And because they don’t understand, they have no way of knowing if it is correct. They don’t know if it is a reasonable implementation, whether it will fit in with the rest of the company’s infrastructure, whether it will be maintainable. They are just passengers, along for the ride. And they better hope that there’s a sorcerer somewhere in the background who can dig them out of whatever mess their magic creates.

I normally hate rules, but here’s one I can get behind:

The AI Rule

AI is a tool that amplifies your ability to code. It does not replace your understanding of what needs to be done, and it won’t take the blame if what it produces is wrong.

You have to know the basics. You have to know design, and techniques, and performance, and the thousand other things, both large and subtle, that make you a developer.

And all that comes from doing the work up front—from experience and judgement.

AI is not deskilling development: if anything it requires more skill of us.

Jan turns up at her 12th interview. The interviewer hands her a marker and points to the whiteboard. "Go ahead and write out the code to calculate the weight of minimum spanning tree of a connected, weighted, and undirected graph." Jan sighs: third time this month. She sets to work.

In my opinion, she shouldn't get the job based in this performance, however practiced the answer.

For me, a good candidate would put the pen down and ask, "Could you give me the context for this, because I'd hope you'd want programmers to use an AI (or a library) to handle this kind of problem. I was looking for a job that lets me do things your AIs can't do."

Challenge-based coding interviews were always a bad idea: they put the focus on the least important skill of a potential hire, memory. Let's list some of the more important skills that the challenge interview doesn't test:

• communication skills, including the ability to question authority in a polite, productive way;

• context building skills: framing a problem in its larger setting before blindly coding;

• knowledge of a spectrum of available approaches to solving a problem, and the tradeoffs between them;

• the ability to work in a team;

• and so on.

I’ve never understood why anyone would consider the ability to write down some random algorithm would be a predictor of success in a job. I have a little more sympathy for questions which ask about selecting a good algorithm or data structure, but, even then, why wouldn’t you let the candidate use resources that would be available to them in their when they were working for you?

Ask yourself: who would be a better fit on your team: someone who’d memorized all four volumes of The Art of Computer Programming but who refused to work with other people, or someone who’d cheerily admit they didn’t know something, but who did know how to look it up, and then would happily share what they’d learned with others?

Flip The Interview

Some secondary schools run a flipped curriculum. Rather than learning in the classroom and practicing at home with homework, they ask the student to study the material at home and then come in with questions, prepared to try some examples in class, where help is at hand

That sounds like a great model for job interviews. Send the candidate a real problem that they are likely to encounter in the first six months of their job. Express it in the way that a team member would normally be given a task. Tell them you don't want a solution. Instead, when they come in, they'll be sitting in a room with you and a couple of team members.

Their task is to ask questions that will lead them to enough of an understanding to be able to start attacking the problem. When they are satisfied, ask what their plan would be to implement it. Make sure they're incorporating small steps and feedback. You don’t want them actually to do the implementation, if for no other reason than the problem is likely to be big enough that this wouldn’t be practical. Instead, you’re look for a realistic approach, with small steps and lots of feedback. The first few steps will be fairly specific, but should then quickly become fuzzier and conditional on the results of prior steps.

Finally, tell them that, two weeks into coding, the requirement X has changed to Y. What impact will that have? For more senior candidates, make the requirements change something that is clearly a bad idea and see if they push back.

Hire The Person, Not The Skills

Today, basic coding skills can be recruited by buying more AI tokens. You shouldn't be hiring based on that.

But AI does not (currently) have the higher level, more global skills that people have. An AI is less likely to remember that Joe in fulfillment once mentioned that you should never schedule deliveries to Topeka on a Wednesday, or remember that someone at the local user group mentioned issues they’d had with a library.

Recruiting based on performance executing a challenge is an incredibly Taylorist1 approach to hiring: it reeks of stopwatches and clipboards. We’ve moved on from that over the last hundred years.

You're hiring a person. Make sure the process you use evaluates the candidates as people, and not as algorithm-writing automata.

I think that’s a very deep question. Maybe stop for a minute and think how you’d answer.

Motivation

I found it most useful to recast the question.

In an age where children have constant access to their phones, do they need to learn arithmetic?

I’ve heard folks online say “no” to this. “They’ll never have to manually add up numbers or calculate a percentage. Let them explore more interesting things.”

I disagree with this, but not because I believe everyone should know what seven times nine equals.

My thinking is that numbers are the first step towards building an intuition about more useful things: ratios and percentages, speed vs. acceleration, compound growth—these are all intensely practical things. Until you have some kind of feel for them, you will be unable to make meaningful decisions when faced with a bunch of important life choices. We learn arithmetic not because we need to know what twelve divided by three is, but because we need the intuition that twelve divided by three is larger than twelve divided by six. We live in a world ruled by numbers; if you don’t know how those numbers behave, you’re at a disadvantage.

I could go on, or I could draw other parallels (do kids need to learn to history? geography? languages? …?). But every point would come to the same conclusion. Some skills are important because they are foundational to building an intuition about the world in which we operate.

Intuition

The aim of learning is not to remember what you’re taught, but instead to make that experience accessible when it is needed.

If I say “two plus two equals,” the word “four” will pop into your head without you thinking about it. There’s no recall; it’s just intuitive. If you’re driving past a street sign your mind will likely read what it says, even if you don’t ask it to. It’s just an inate ability, learned through repetition and practice when we were young.

It is this ability to handle things below the level of conscious thought that allows us to function.

Look out the window. Maybe you see trees with leaves rustling in the wind. Maybe you hear cars driving by. Maybe you smell that bakery down the street.

How much information is arriving via your senses? What is the bandwidth we handle when we simply exist? As raw data, it must be many, many megabits each second. If we had to process all that consciously, we’d melt.

That’s why the level of thought below consciousness is so important. It’s the LLM we have been training all our lives to recognize patterns and make suggestions based on what it finds.

Back to coding.

Do You Use AI When Learning to Code?

My answer is that “it depends.” Why are you learning to code? Do you have some cool idea for a killer app that you need to get to MVP? Vibe away. Do you want to build a career in software development? The basics will help.

If learning to code simply means getting better at prompting Claude to write something for you, then you are choosing to limit your experience; you are choosing not to build an intuitive understanding of what the code does and why it does it. And that’s absolutely fine, but you have to know going in that you’re building on an unknown foundation. If cracks appear later, you won’t really be in a position to know why.

If, instead, you decide to forego the AI, then you’re going to go slowly. You’re going to make mistakes and experience frustration, both made worse by the knowledge that an LLM could tell you what’s wrong in seconds.

But those mistakes and frustrations are necessary if you are trying to understand how things work; these are the feedback paths that build and cement intuition.

The vast majority of experienced developers use AI every day. But they all tell me the same story: you have to keep an eye on what it’s doing, and you have to guide it towards solutions you want it to use. To do that, you have to have an idea when you start of how things will play out; you must have the experience to be able to see the overall structure of what you’re building, and to be able to predict likely issues and choices that will mitigate them. To use AI effectively, you pretty much have to know what it is doing for you.

The AI Paradox

If want a career when you get computers to create value for people, then you need to build the intuition; you need to do things manually and make mistakes.

But there’s the paradox. AI is an incredible tool in the hands of an experienced developer, but how do you get to be an experienced developer without using AI? Companies are unlikely to hire someone who does something as old-fashioned as writing all their own code.

So here’s what I’d suggest. If you really want to get good at doing what we used to call “programming,” adopt a hybrid approach.

You must build intuition, and that means learning. It means trying stuff, making mistakes, rereading the same paragraph in an article a dozen times until it clicks. There’s no fast-path to experience.

But… as you start to cement that understanding into your intuition, you can start to use AI tools to facilitate using what you’ve learned. Once you know the multiple ways of summing a list of numbers, you could probably get an LLM to write that code for you, letting you focus on the higher level things you are learning.

Discipline

Once you allow an AI into your editor, then you are living with constant temptation. “I could write that for you, you know” is a dangerous thing to hear when you’re struggling to master something. It might be possible to tell your ASI assistant that you only want help up to a certain level: “please help me write basic control structures, but leave the higher level function organization to me.” I’ve played a little with this, but without much success.

So, as with most things, becoming good requires motivation and discipline.

Personally, I find the rewards are worth it. What do you think?

Leave a comment

Share

Subscribe now

I’ve been working with a company in Belgium, helping them refine their team structures and development processes. This week, the topic turned to job titles. They asked what I thought about assigning titles like Junior Developer, Developer, and Senior Developer.

I recoiled. I wasn’t sure why—so I did what any self-respecting consultant does and deflected with a question:

“What’s your intent here?”

We want to give people a set of objective requirements for each role, in order to give them something to aim for.

“Could you give me some examples?”

Well, a junior developer on the modelling team has to be able to create and manipulate simple 3d objects in Blender, and they have to be able to perform basic image manipulation tasks in Photoshop.”

That clarified things. There wouldn’t be one set of requirements per role. Instead, each specialty—modeling, coding, and so on—would have its own separate expectations at each level.

So I got to thinking. Suppose there are half a dozen high-level requirements in each area: Junior/Modeling, Senior/Coding, and so on. What do you do with a modeler who’s world-class in Blender but hopeless in Photoshop? Are they a Junior because of the Photoshop gap? A Senior because of their Blender mastery? Or maybe a Middling Modeler because you couldn’t decide?

We Label People When We’re Too Lazy To Know Them

We live in a world where people are dropped into buckets based on labels: Republican/Democrat, Liverpool/Everton, Junior/Senior Developer. And that labeling encourages the insularity of opinion we see everywhere.

Reality is messier and more nuanced. Someone might align mostly with Republican values but support parts of the Democrat platform. A Liverpool fan might secretly admire a Toffee’s striker. And a so-called junior developer might already have senior-level skills in a niche area.

These high-level labels are a management convenience. But they shortchange both the individual and the organization.

What If We Gamified It?

On the call, I suggested an alternative to these blanket titles. I asked if they could go back to their original requirements and break them down into individual skill challenges: Basic Blender, Photoshop Novice, Blender Wizard, and so on.

Each challenge would have specific, objective criteria. And when someone completed one, they’d earn a badge, trophy, or silly award. Over time, folks would collect trophies across specialties and levels—physical tokens they could display on their desk or laptop.

You could even lean into the game aspect. Every quarter, host a light-hearted showdown: two Photoshop Wizards go head-to-head on a tricky challenge, with their screens projected for all to see. A bit of fun—and a great way to pick up new tricks.

To my surprise, they jumped at this idea. They’re now rethinking the Junior/Mid/Senior hierarchy, replacing it with individual skill tracks. Their new system has three goals:

• Identify people’s actual skills, so projects can tap into the right talent.

• Create personalized career paths, instead of promoting people based on checklists.

• Encourage learning through playful competition, sparking curiosity and motivation.

It will be interesting to see how that works out over the coming year.

What do you think?

Facing an interesting challenge? I’m at dave@pragdave.me

Leave a comment

Subscribe now

Share

I was one of the people who helped create the Manifesto for Agile Software Development. I’m proud of what we did, and I still believe that the four values are as important today as they were 25 years ago. I had hoped that it might improve the lives of developers, and for a while it did. No longer. What people now call Agile has become the kind of overbearing set of practices that we were rebelling against when we drafted those values.

For the values of the manifesto to work, we have to change the world. And the world has other things on its mind at the moment.

But I’d still like to find ways to improve the developer’s lot.

Take Two

We can’t control our world, or how our company runs. But I’m startinging to think we don’t have to. We still control and incredibly potent assets: ourselves.

We developers tend to forget just how much control we have over our daily lives. Sure, we have structure imposed on us, but that’s typically at the per-day, per-week, and per-sprint/release level.

So let’s stop trying to change the world, and instead focus on changing our world.

Acknowledging that we can do that is step one. Step two is setting some goals.

To my mind, there is no bigger goal right now than reducing the insane levels of complexity we work with.

Some folks may be thinking that the incursion of AI into our working lives is the number one threat. I disagree. AI is a powerful tool, but a lot of its power comes from the fact that it relishes complexity; when you see programs of a whole, as a mesh of parts that interact, you have no need to be limited by human design principles. Who needs DRY when you can immediately see and change every use of a function or some data?

But that’s a tactical view. There comes a point where you cannot afford to ignore the structure of what is being built. If interactions in your system grow exponentially with the number of parts, there’s a point at which no technology can manage it. And by that point, it is too late.

One of the reasons AI often produces bloated code is that it merely extrapolates from the code it has consumed during training; it is the ultimate but we’ve always done it that way engine. This is just cargo-culting.

The sad thing is that human developers do that, too. We look for answers from people who seem to know what they are doing, adopting their designs, tool choices, and often their code to create complex patchworks of code, code that is hard to change, and code that is vulnerable to the inevitable changes in the tools and libraries on which it is built.

So I come back to the idea of reducing complexity being the most important thing we, as individuals, can do if we want to reclaim control over our daily lives.

I believe this so strongly that I spent four years writing three books about it. The first two I threw away; I discovered you cannot tell people how to simplify what they do—simplicity is always contextual. The third book took a different (and, for me, uncomfortable) tack. Rather than tell folks what to do, I give 29 examples of ways I have found, in my particular world, to make what I do simpler. The idea is not to copy me: that wouldn’t work. Instead, I hope the examples illustrate the thinking behind these simplifications, the feedback loops I used to notice that something was getting complicated and then to monitor my progress as I experimented with simplifying it.

Is this article just a lame-ass attempt to market my new book?

Sure.

But it’s an article I would have written anyway, book or no book. In many ways, we developers are our own worst enemies. We follow fads, join cults, and adopt practices. We sometimes do this because it looks interesting. More commonly we do it because we hope that one day we’ll find the silver bullet that puts to rest our fear of failing.

Ironically, every time we do that, we add complexity and stress to what we do, which in turn just accelerates the cycle.

So, buy the book if you want. But, regardless, think about how you, personally, can reduce the complexity in your (professional) life. Think about the things you take as a given (exhaustive testing, stand-up meetings, the framework du jour) and ask how you could test to see if they aere actually making things better. How you do things can be refactored in the same way as the things you produce, and the same rules apply: you need an objective, a timebox, and feedback.

It is difficult to make things simple. It takes a conscious process, applied over time.

It’s worth it.

Get the ebook

Subscribe now

Share

I’m currently writing code to import royalty data from our distributor. In comes in the form of 28 or so spreadsheets (because, of course it does). For each, the first step is to upload and ingest it. And, because I’m paranoid about getting royalties correct, I try to do a bunch of validation and reconciliation as I read each sheet in.

The typical flow is:

1. Extract the spreadsheet file from the incoming HTTP request.

2. Extract metadata from the file and use it to start populating an Upload record, where I keep track of each upload, regardless of its content.

3. Parse the spreadsheet, extracting the data I need. Sometimes this is easy; it’s just a table. Other times the data is more like a statement or invoice, with a variable format, so I have an actual parser to understand it.

4. I then associate the data to do with an actual title with the title in our database.

5. Finally, I save all the data, typically across three or four tables, in a single transaction.

These steps are basically a pipeline, where each step feeds data to the next. However, errors can happen at each step, and I need to record them in the Upload record. When an error occurs, I stop processing that spreadsheet. This means I can’t use regular Ruby pipelines.

Each step returns a hash, either

{ status: :ok, data: data_to_pass_to_next_step }

or

{ status: :error, message: error_reason }.

Here are the three alternative designs I played with.

Design 1: Linear Brute Force

Why overthink it? Just write the code:

result = excel_file_attached?(file)
if result[:status] == :error
    record_error(upload, result[:message])
else
    result = add_details_to_upload(result[:data], upload)
    if result[:status] == :error
        record_error(upload, result[:message])
    else
        result = parse_statement(result[:data])
        if result[:status] == :error
            record_error(upload, result[:message])
        else
            result = map_isbns_to_skus(result[:data])
            if result[:status] == :error
                record_error(result[:message], upload)
            else
                result = save_rows(upload, result[:data])
            end
        end
    end
end

Code like this hurts my soul, so at the very least I’d want to flatten the nesting.

result = excel_file_attached?(file)
unless result[:status] == :error
    result = add_details_to_upload(result[:data], upload)
end
    
unless result[:status] == :error
    result = parse_statement(result[:data])
end

unless result[:status] == :error
    result = map_isbns_to_skus(result[:data])
end

unless result[:status] == :error
    result = save_rows(result[:data], upload)
end

if result[:status] == :error
    record_error(upload, result[:message])
end

Better, but the code that does the actual work is buried inside all those unless statements.

Design 2: The Mini State Machine

When I have a sequence of steps like this, I often use a trivial state machine:

    step = 0
    result = nil     # for scoping

    loop do
      result = case step
               when 0 then  excel_file_attached?(file)
               when 1 then  add_details_to_upload(result[:data], upload)
               when 2 then  parse_statement(result[:data])
               when 3 then  map_isbns_to_skus(result[:data])
               when 4 then  save_rows(result[:data], upload)
               when 5 then  break

               when :error
                 record_error(upload, result[:message])
                 break
               end

      if result[:status] == :ok
        step += 1
      else
        step = :error
      end
    end

In general I like this code, but it seems a little over-the-top.

Design 3: Exceptions

The third thing I considered was having each of the processing functions raise an exception rather than return an error status. That cleans things up considerably:

begin
  file
  |> excel_file_attached?
  |> add_details_to_upload(upload)
  |> parse_statement
  |> map_isbns_to_skus
  |> save_rows(upload)
rescue UploadError => e
  record_error(upload, e.message)
end

Clearly, this is the most direct of the three.

Design x: Metaprogram

I briefly considered reimplementing the pipeline operator |> to exit early on seeing an error, but my days of that kind of tomfoolery are long past.

Design y: A Monad Library

I’m aware of the dry-monads library and the Railroad article. I considered using it, but adding an extra dependency for just a couple of lines of code was a nonstarter. Perhaps if I’d written the whole codebase from scratch using dry-monads, it would be an improvement, but this particular chunk of upload code is the first time I’ve felt the need for monadic behavior in this app.

What I Ended Up Doing

In the end it came down to design 2 or design 3. Of the two, I much prefer the exception-based approach: it seems on the surface to be a lot more direct and easy to comprehend.

You might be surprised, therefore, when I oped to use design 2, the state machine.

I admit it was a difficult decision, and as I write this I still have doubts. But I had two nagging thoughts about the exception/pipeline approach that swung it for me.

First, and weakest, is that I have a instinctive aversion to using exceptions as a flow-control mechanism. My rule of thumb is to use exceptions only for exceptional things—things that should never happen. A validation failure when reading external data doesn’t seem to count.

The second reason is what swung it for me. The pipeline is a great construct, but it ends up coupling functions together. When you write a() |> b(), the return value of a must be acceptable as the first parameter of b.

For established code, which is unlikely to change, this seems like an acceptable tradeoff: the types are unlikely to change, and the pipeline is just plain more readable.

However, when I’m writing new code for a new application, I just know that I’ll be changing things around as I learn about the domain.

In the state machine approach, I’m currently emulating a pipeline by passing the result from one function to the next. But that isn’t set in stone. I can add new code before a particular call, or massage a return value. The design is more open.

The pipeline approach would make that trickier, because the fact that I’m using a language feature to chain functions together constrains my choices down the road. And as I just know I’ll end up having to change stuff, I chose to forgo the pipeline’s elegance.

But That’s Just Me

My reason for posting this is that I’m still not sure. I would really like to start some discussions in the comments. What would you do, and why? Are there other, better choices? Am I overthinking this?

Chat with you there.

Leave a comment

Subscribe now

Share

I’d like you do stop and thing about the following question.

Why do you write tests?

Seriously. Stop reading, sit back, and think about it for a couple of minutes. Maybe jot down an item or two on a bullet list. Do it now, and I’ll go make a cup of tea.

What did you come up with? Here are some of the reasons I’ve heard over the years. First, come two sad ones:

• I have to write tests: they are part of our project standards.

• I feel I should write tests because everyone says I should.

And then some perfectly good reasons:

• I write tests to verify that my code works.

• Tests mean that I can more safely change my code in the future.

• Tests help me explore my APIs before I use them elsewhere in my system.

• Tests drive me to write code that is more decoupled.

Each one of these is a credible assertion. Each seems like common sense. Who wouldn’t want to know if their code works? Who doesn’t want to write code in more isolated, decoupled chunks?

The problem is that assertions, particularly ones that seem to be virtuous, have a nasty habit of becoming accepted facts.

Personally, I don’t think something deserves to be a fact unless it can be tested.

Testing Testing

Let’s think up some experiments to try to validate whether the four testing assertions above are actually true. But, before we do, a warning. Truth is almost always contextual. Most people would assert that stabbing someone in the throat is always bad; but it’s good if you’re performing a tracheotomy. In all the discussions that follow, I’m going to have to generalize, because I don’t know your specific context. That’s why I’ll be suggesting ways you could experiment with the testing assertions personally, in your own context. Anyone who tries to sell you on a universal do-this-rule is either stupid or fraudulent.

Assertion: I write tests to verify that my code works

This is an obvious assertion: verifying functionality is pretty much the definition of testing. But, as with all assertions, we really should test to see if it is true, but how?

Here’s what I did a while back. I first made sure that I had a handle on the number of faults that surfaced in the code that I delivered. This code had gone through my normal testing process: it wasn’t 100% coverage (because that’s kinda dumb) but it was decent.

Once I had the baseline, I stopped writing unit tests.

Some of the impacts that I measured I’ll be talking about when we come to the other assertions. Looking just at bugs, however, I discovered two main things:

1. Many of the bugs that would have been found by testing were actually found instead as I was using the buggy code in other code. In every case I encountered, the source of the bug was pretty obvious: I can’t remember triggering a bug that was more that 2 levels of call away from the place where it impacted me.

2. To my surprise, the failure rate of the system once delivered was slightly higher in the first few days, but then fell back to about the same levels I’d seen previously in tested code.

I looked into the failures in the delivered code, both before the experiment and during, trying to work out why there was so little variance.

It turned out that most of these bugs had the same root cause: I had misunderstood something about the application while I was writing it. I hadn’t realized that a value might be missing, or that a key could be duplicated between two data rows. And, because I didn’t know about these things, I wouldn’t have written tests for them in the first place.

Conclusion: for me, on the kinds of projects I’m writing, religiously writing tests seemed to have little impact on my delivered error rate.

Assertion: Tests mean that I can more safely change my code in the future

This assertion seems very plausible: having a good suite of tests reduces the chance of introducing regressions while making future changes.

To test this out, I had a look at what happened when I came to update the code I’d written without tests.

This is anecdotal, but I found the lack of tests made changes considerably more stressful. For every change I made, I’d need to manually explore the future trajectory of any affected data, making sure that my modifications didn’t break any assumptions elsewhere in the code.

In the end, I found myself writing canary tests: functional-level tests that verified the behavior not of the changes I made, but of the code that relied on that changed code.

On the face of it, the assumption that tests are a benefit during code updates would seem to be valid. However, there’s another side to consider. This is not something I’ve explicitly tested, but instead comes from my experience.

Over the years, I’ve learned the hard way that it’s a mistake to simply add new chunks of code at some convenient point in an existing codebase.

When the app was first written, I was constantly refactoring the code; any friction adding new code was a hint of some kind of problem with the existing codebase. One of the joys of decoupled code is that it is easier to restructure, as changes are more likely to be localized.

However, what happens when I come to modify that code later on? Again, adding the new code could prompt a refactoring of the existing code. But if that code has a lot of unit tests (and I’m not proud of this) I just couldn’t bring myself to update both the existing tests and existing code, and so I just carbuncled the change in where it could fit. My laziness meant that the tests would tend to lower the quality of the app over time.

Conclusion: Overall, and for me, on the kinds of projects I do, the idea that having tests makes future changes less fraught seems to be true. As a result, although I may not have a full test suite when entering in to a change, I’m likely to add some high level tests before I actually make that change.

Assertion: Tests help me explore my APIs before I use them elsewhere in my system

The idea behind this assertion is that your test will be the first time you call the code you just wrote (or are about to write for you Test-First folks). This gives you a fresh perspective on that code, and often that perspective gives you hints that things aren’t quite right.

A key indicator (for me) is when the tests are tricky to write because they require setting up a bunch of context: your format_name function takes a User object as a parameter, so each test must go through the pain of constructing a valid User before it can call format_name.

When that kind of thing happens to me, I stop and wonder why my formatting function needs the entire user object: could I just pass in the first and last names, and perhaps a title? Or maybe this friction caused by writing the test is saying that the various name fields shouldn’t just be separate attributes of a user; instead I might need to have Name objects which User could reference.

This assertion and the next were my personal go-to reasons for writing tests. It turns out that what I learned about this API assertion mirrors my discoveries about the decoupling assertion, so I’m going to defer my conclusion until the next section.

Assertion: Tests drive me to write code that is more decoupled

Code that has a lot of coupling is very difficult to work with, because you can never fully know the scope of a change you make.

It turns out that unit tests are a great way of discovering some types of coupling. If you want to test function A, but to do so you need to construct half a dozen new objects, involving calls to twenty other functions, they you’re being forced to live through the pain imposed by A’s degree of coupling.

So, just as with APIs, a test that is difficult to write because the thing under test requires a lot of context is a red flag: you might get the test to run, but underneath that success you know that the code it is testing smells.

The thing I feared most when I started my experiment of not writing unit tests was the damage it might do the the design of the code I was writing. I worried that coupling would sneak in if I had no tests to keep the code simple.

In practice, I discovered something surprising. And, before I go on, I have to confess that, of all my observations in this experiment, this is likely to be the most personal and least applicable generally.

Anyway, to my surprise, I discovered that, although I was not writing unit tests, I was constantly thinking about writing them. “How would I test this” was running in a kind of loop all the time I was coding. It turns out that just considering that question was enough to trigger the doubt that would end up driving the refactoring I needed. In effect I was getting the design benefits of testing without actually writing the tests.

I think part of the reason for this is that I had spent decades before the experiment doing various kinds of testing, and the various facets of creating tests and become part of my tacit brain. (This is also what happens when you learn to drive. Initially, you have to think about everything you do, but after a while it becomes almost automatic: the tacit side of your brain has developed the reflexes and intuitions to handle most of the act of driving for you.)

My experience was that I was refactoring code just as often when I developed with tests as when I developed without. Looking back at code written without tests, I really don’t see too much I’d want to change in terms of structure.

Conclusion: I thing that thinking about testability is a key tool when it comes to improving your APIs and your code structure. In my personal experience, though, I find that I get that benefit even if I never write the tests; just thinking about them is enough.

So…

I am not saying that tests are good or tests are bad. I’m not saying that testing makes your code better or makes worse, easier to maintain or harder.

All I’m saying is that I performed the experiment, and came up with some personal conclusions that inform what I do now.

I still write tests, but I focus in on areas where the cost of the code being wrong are greater than the costs of creating and maintaining the tests. When I’m writing code that deals with money, or with information that has personal value, I write unit tests. I will write higher level functional tests that exercise a whole chunk of the app (does importing this spreadsheet cause the correct royalty adjustments to appear in that table?). And if one of those high level tests fails, I might explore with some lower level tests until I’ve isolated the issue.

I make a point of maintaining the “how would I test this” inner dialog when I’m coding, and I make time to react and refactor when the question has no easy answer.

But that’s me. I have no idea what you should do. Except…

• If you’re just starting out, write tests. Keep doing it until in becomes reflexive. This probably takes two to five years.

• If you’re working in a domain where the cost of failure is high, write tests. It is always going to be more expensive to write this kind of code, and the tests won’t be as costly in relative terms.

• If you’re exploring, write tests. I do this a lot, and I find that the series of tests becomes a great reference; I look back on them to remember what I’ve learned.

More suggestions:

• If you write exploratory tests, delete them before delivering the project.

• If you’re a TDD person, and the strictly follow the no-code-without-a-failing-test mantra, do back and delete all the silly tests which have nothing to do with the application functionality.

Tests are code, and the more code you’re carrying around in a project, the more dependencies you have, and the more that can go wrong.

At the end of the day, I’m simply suggesting that you don’t have to do something because people tell you it’s a good idea, or because you yourself assume that it’s the right thing to do.

Test you assumptions, and refactor what you believe accordingly.

Subscribe now

Share

I’m writing code to import sales information from a variety of sources. The data from each source is first uploaded and normalized, then imported into our main database for reporting and royalty calculations.

Uploading the data involves ingesting a binary file (typically a spreadsheet) and normalizing it depending on the data source. The import into the main system is also source-dependent, as different business rules apply to each.

In the past, my go-to approach was to have a base class, with a subclass for each source.

These days, though, I favor functions over object instance methods, and modules over classes. As I’m integrating this into an existing system, I’m sticking with Ruby.

I experimented with a couple of approaches before settling on one I liked.

I use the following directory and module structure:

lib/
    royalties/
        dispatcher.rb              Royalties::Dispatcher
        source_one/
            import_handler.rb      Royalties::SourceOne::ImportHandler
            upload_handler.rb      Royalties::SourceOne::UploadHandler
        source_two/
            import_handler.rb      Royalties::SourceTwo::ImportHandler
            upload_handler.rb      Royalties::SourceTwo::UploadHandler
        source_three/
            import_handler.rb      Royalties::SourceThree::ImportHandler
            upload_handler.rb      Royalties::SourceThree::UploadHandler

I also have a model object representing an upload (regardless of source). (And, yes, it’s an object, because I’m using ActiveRecord.) Each upload object has an attribute that identifies its source.

Elsewhere in the code, I want to be able to work with these upload objects regardless of their source, so I write something like:

Royalties::Dispatcher::handle_upload(upload)

The dispatcher handles the polymorphic aspects of this by using the upload type to select the module to invoke:

module Royalties::Dispatcher
  extend self

  def handle_import(upload)
    find_handler(upload.source)::ImportHandler.handle(upload)
  end

  def handle_upload(upload)
    find_handler(upload.source)::UploadHandler.handle(upload)
  end

  private

  SOURCE_TO_HANDLER = {
    Upload::SOURCE_ONE   => Royalties::SourceOne,
    Upload::SOURCE_TWO   => Royalties::SourceTwo,
    Upload::SOURCE_THREE => Royalties::SourceThree,
  }

  def find_handler(source)
    SOURCE_TO_HANDLER[source] || 
        fail("No handler found for upload source #{source}")
  end
end

Because the source-specific implementations are just functions, testing them in isolation is easy.

Subscribe now

Share

Summary: using method chaining might look fluent and elegant, but it adds coupling while reducing design flexibility. For the last ten years I have been moving away from chaining, instead using various kinds of function composition. Let me tell you why I find it better.

The Appeal of Method Chains

Here’s a turtle graphics program that draws a square. It’s written in some arbitrary OO language.

turtle
.pen_down()
.move(1)
.turn(90)
.move(1)
.turn(90)
.move(1)
.turn(90)
.move(1)
.pen_up()

It’s a kind of Fluent Interface, a mini-Domain Specific Language. It’s made possible because each of the actions is a method defined in the Turtle class, and each does its business and then returns that same object.

def pen_down
# move the pen…
return self # (or this, or …)
end

There’s no law that says it has to be the same object that is returned by each method:

File # → File class
.open(“data.dat”) # → File object
.read_line() # → string1
.trim() # → string2
.split(/\s+/) # → array of string

This is the method chain, and it has been a staple of OO development pretty much from the start.

I love them, and use them every day.

But the lure of the fluent-style method chain can lead developers down some pretty dangerous paths. Let’s look at a couple of these before considering an alternative.

“I Can Make It Like English”

(or French, German, Spanish…)

Testing is a big deal in the Ruby world. There are two schools: traditional and behavior-driven. Both use assertions to check expected values against the values produced by code. In the traditional camp, we have tools such as minitest:

assert(update_complete)
assert_equal(7, add(2, 5))
refute(count > 5)

In the BDD classrooms, they teach tools such as RSpec:

expect(update_complete).to be(true) # or be_truthy
expect(add(5, 2)).to eql(7)
expect(count).to_not be > 5

Both styles achieve the same thing, but RSpec attempts to look more like a specification, more like natural language. And it is pretty clever.

But, for the same reason there’s an (increasingly shallow) uncanny valley between artificial and real images, RSpec’s attempts at fluency leave me feeling uncomfortable.

Every test starts expect(expression).to (or .to_not). The .to method is there largely as window dressing; it gives us a semblance of English-like syntax. It’s followed by the actual assertion: a verb and possibly the expected value. RSpec calls these parts matchers.

They’re trying to be fluent, but failing. Like AppleScript, it’s fairly intuitive to read, but it is anything but to write. Why the space after to? (Because the matcher is actually a parameter.) Why be(true) but eql(7)? (Because these are two of the three different kinds of Ruby equality). And why the split infinitive in the last case? Who knows?

I believe that part of the reasoning for this was to allow less technical people to read and agree on the tests. It may actually have happened that a manager peeked at this code, at some point, somewhere. But my guess is that this worked about as well as assuming managers could read COBOL because of its English-like syntax. The barrier that needs to be crossed is not one of syntax; it’s one of abstraction, of thinking like a developer.

Programming is all about layering abstractions, and method chaining is one way of doing that. But it can come with a high price.

Homogenous Chains Good,…

For a method chain to work, the methods that form the links must each return an object that implements the subsequent method in the chain: in

a().b().c()

The method a must return an object that implements a method b, and that method in turn must return an object that implements c.

There is a style of method chain that makes this easy. It’s when you start with some initial object that has a bunch of methods that can be used to update its state. Each of these methods returns the initial object, allowing these calls to be chained. Sometimes this is called a builder or a constructor. The turtle graphics example at the top of this article is an example of this:

turtle
.pen_down()
.move(1)
.turn(90)

The turtle object has an initial state, and methods such as pen_down and move update that state. The chain is homogenous; it’s turtles all the way down.

Another example of this is the ActiveRecord query builder:

Book.where("id > 100").limit(20).order("id desc")

Each of these methods adds to the query structure without actually executing any SQL; that SQL is only generated when you try to access data from the query.

But there’s another style of chain, where we’re striving for some kind of fluent API, but we have to bend the code to make it work.

Heterogeneous Chains Bad

Another common use of chains is to traverse a bunch of objects, either to extract some particular piece of data or to execute some particular functionality.

Here we have a line of code in the fulfillment module trying to work our the charge for shipping based on a client’s address:

client.address(“home”).postcode().delivery_zone().shipping_cost(package)

We can get all academic and call this a violation of the supposed Law of Demeter, or we can just be pragmatic and say that this code has just increased the coupling in your application in manifold ways. Let’s see how.

Chains That Bind

Let’s look at dependencies.

We can infer that our client object depends on an Address class, with in turn depends on a PostCode, and so on.

This is a perfectly fine set of dependencies: if the PostCode class implementation changes, then that might affect the Address class, but that’s all (at least in this picture).

But add in our Fulfillment class, and the picture changes:

Our Fulfillment class now depends on all five of the other classes: a change in any of them could potentially cause the need to change Fulfillment. Maybe we decide that the PostCode class has no business knowing about its delivery zone, and we move that functionality somewhere else. Fulfillment breaks

One way to fix this is to say that Fulfillment only depends on Client, and that Client must implement a shipping_cost() method. If we apply that same logic throughout, we’d also have to add a shipping_cost() method to Address, PostCode, and DeliveryZone as well. That’s clearly untenable.

Functions and Composition

One way out of this mess is to stop insisting that we pass state and functions together; that we no longer have to invoke methods only on objects. We’ll call these liberated methods functions, as they transform one or more parameters into a new value. This lets us write something like this:

shipping_cost(delivery_zone(postcode(address("home”, client))), package)    

Of course, this particular cure is worse than the disease; these kinds of nested calls are a nightmare to read and maintain.

This is where method composition and pipelines come into play.

In a functional language such as OCaml, function composition creates a new function that is the result of running the first, then passing the result to the second. We can create a function that does composition; we’ll call it >>.

let ( >> ) f g x = g (f x)

This says that >> is a function that takes three parameters: two functions, f and g, and a value x. It passes x to f, and then passes the result of that to g. We can use this function to build a new function:

let plus1 n = n + 1 
let square n = n*n 
let plus1_and_square = plus1 >> square 
plus1_and_square 10    → 121   

In this example, plus1_and_square is a new function that is the composition of plus1 and square.

We can parameterize these composite functions:

let plusn n = (+) n
let plusn_and_square n = plusn n >> square  
plusn_and_square 3 2     -> 25

Now we can create a new function that creates a function that calculates the shipping for our customer and package:

let shipping client package =
  address "home" client >>
  postcode              >>
  delivery_zone         >>
  shipping_cost package

Using that, I can create a function that calculates the shipping cost of delivering any package to me:

let shipping_to_dave package = shipping_client “pragdave” package

or, more concisely

let shipping_to_dave = shipping_client “pragdave” 

See what we’ve done? We created a chain of functions which can be parameterized, and each is independent of the other.

And There Are Pipelines

Your language du jour may not support function composition, but an increasing number of languages support something similar, the function pipeline. Using the OCaml and Elixir notation, we can say

value |> f |> g

takes value, pipes it into the function f as a parameter, then pipes the value returned by f into g. It is similar to function composition, but it executes immediately. Using pipelines, we could write our shipping code as

client
  |> address "home"
  |> postcode
  |> delivery_zone
  |> shipping_cost package  

Try This…

I’ve found that this kind of decoupling of functions from state pays enormous dividends, to the point where I now write my code in this style even if I’m writing in an OO language. Typically I’ll use objects or read-only structs to hold state, and then I’ll write independent functions to transform that state. If the language doesn’t have a pipeline equivalent, I’ll either use temporary variables or a loop and a trivial counter-based state machine to chain together the function calls.

It takes getting used to, but if your experience is anything like mine, you’ll find yourself writing code that easier to test (because each function is independent), maintain (because state is easy to examine at every step), and update.

Subscribe now

Share

In 1977, the architect Christopher Alexander published the book A Pattern Language, containing over 250 suggestions for folks designing spaces. Each pattern was a context followed by a suggested solution and a discussion. For example, pattern 159 starts

When they have a choice, people will always gravitate to those rooms which have light on two sides, and leave the rooms which are lit only from one side unused and empty.

Therefore:

Locate each room so that it has outdoor space outside it on at least two sides, and then place windows in these outdoor walls so that natural light falls into every room from more than one direction.

This pattern, perhaps more than any other single pattern, determines the success or failure of a room. The arrangement of daylight in a room, and the presence of windows on two sides, is fundamental. If you build a room with light on one side only, …

Ten years later, Ward Cunningham and Kent Beck cowrote the paper Using Pattern Languages for Object-Oriented Programs, which suggested Alexander’s approach might also work for code. They continued to work on this, and in 1994 Ward invented the wiki in order to create a place where people could discuss patterns.

That same year, Gang of Four1 published Design Patterns; Elements of Reusable Object Oriented Software. Their intent was to isolate and name various structures that people used in particular situations; if two or more people had address a particular issue in the same way, they’d name it and describe it. In their case, they were looking at code written in C++, and many of the patterns exist to overcome constraints imposed by that language.

The crowd went wild. For the next 8 or 9 years, Ward’s wiki was the place the cool kids hung out. If a thing moved, someone would write a pattern language describing how. There are thousands of published patterns for software development.

And every single one of them is used incorrectly.2

The Title is to Blame

I don’t know why the GoF gave Design Patterns the subtitle they did, but Elements of Reusable Object Oriented Software sent a subtle but damaging message. The word reusable implies that they are presenting you with some kind of plug-and-play component library. Do you need to create instances of different classes depending on some value? Then you need a Factory! And, for a limited time, we’re throwing in the code and a UML diagram at no extra cost!!

And that’s what happened. People jumped on the idea of patterns being building blocks of design. Suddenly you couldn’t write a class unless it’s name ended with the name of a pattern: UserFactory, ConfigSingleton, PrintDecorator. What’s worse, people then implemented these classes by transcribing the C++ code from Design Patterns into their language du jour, line for line.

And this is really, really dumb.

Patterns are not reusable chunks of code. They are not components.

Patterns are descriptions of techniques that might be useful in a particular context. They are not recipes; Alexander called them hypotheses.

Time for a bad analogy.

How To Paint, A Pattern Language

Through a miracle of engineering, scientists bring the painter Rembrandt van Rijn back to life. Before he dies for a second time, they want to capture his knowledge and expertise as a artist. He decides to document what he knows as a pattern language.

He sets to work, writing patterns such as “Capturing soft light on the forehead”, “The nose of a drunk,” and “The disdainful eyebrow.” When finished, the book is a global phenomenon. Artists everywhere hail it as the new way to construct a painting: apply each of the patterns as needed as you traverse the canvas, and you’re guaranteed a masterpiece.

We end up with pictures like this:

This is what happens when you say “I need a nose here and a forehead there”, go to Rembrandt’s patterns book, and copy his implementations onto your canvas.

Clearly, that’s not how Rembrandt’s patterns should be used. They should be studied for technique and for insight, and then reinterpreted by the hand holding the brush.

For Example

Search for “factory pattern ruby” and you’ll find examples such as this.

This is the programming language equivalent of the Monty Python fish license, but rather than being a dog license with the word “dog” replaced with the word “cat”, it’s C++ (or Java, or …) code with the syntax replaced with Ruby.

On top of that, it doesn’t actually show a factory method: it’s just two unnecessary levels of delegation.

Why pick on this code? I’d rather not, but it’s all like that, and this was the least obnoxious. Rather than ask themselves “what do I need to achieve?”, the implementors of these online exemplars made the decision “I need a factory",” looked it up, and transcribed what they found into Ruby.

Let’s convert this example to use an actual factory method, and write it based on need, and not some pattern.

We need to simulate one of n machines. Which one we chose depends on the value of an external parameter. Every machine is a class, and each implements an operate method.

Still pretty ugly. We can tidy it a little by replacing code with data (my current mantra).

It still has those strange Machine1 and Machine2 classes though. This is where sample code breaks down, because we don’t know what they do in real life. If they are simply stateless wrappers for the functions they contain, then they should be modules, and not classes.

If instead they have behavior in common, and the operate method is the difference between being a machine1 and a machine2, then we have some options.

The option that you probably shouldn’t take is the one that everyone uses: create an abstract base class containing the shared behavior and make Machine1 and Machine2 subclasses. In a strongly typed OO language you are forced to do this to get polymorphic behavior. But in languages such as Ruby and JavaScript, there’s no need, and adding class hierarchies increases coupling.

So if we assume that the external interface to all machines is a do_something method, which in turn calls the machine-specific operate method, we could use mixins:

An alternative is to remove the polymorphism altogether, and instead inject the appropriate method at runtime:

This code assumes that the machine object do_something functions can be called multiple times. If that wasn’t the case, there’s no need for a separate StandardMachine class: we can just pass the operation to the do_something method.

Don’t Design Before You Need To

In last week’s article I talked about the evils of premature design. This article is really just a continuation of that. The only difference is that when you use a design pattern, that design predates your need by at least 30 years.

My personal take: design patterns are a convenient way of describing code that you already wrote. Don’t use them as recipes.

Postscript

Whenever I meet an architect, I make a point of asking what they think of Christopher Alexander and his idea of patterns. I’m well past my tenth, and none of them has heard of him.

There’s more of this kind of old-man-fist-waving-at-clouds in my new book, simplicity…

Subscribe now

Share

In the 1974 paper with the somewhat click-baity title Structured Programming With go to Statements, Donald Knuth wrote:

premature optimization is the root of all evil (or at least most of it) in programming

Knuth’s point was that optimizing code normally makes that code trickier to work with; when you’re still in the exploring phase, trying to work out what your code should do, that’s the last thing you need. So, when you optimize (and you probably should at least think about optimization), you should do it only when (a) you find out what actually needs optimizing, and (b) when you’re unlikely to make major structural changes to the code base.

I’m coming to believe that what’s true for optimization is true pretty much universally: premature anything is likely to be a bad thing.

And you won’t find a better example of this than the way folks design code.

Premature Design is the Devil’s Work

I know this because I’m guilty of doing just that: bringing my vast arsenal of design principles to bear simply because I believed the Devil when they said “if you don’t make it perfect now, you’ll never come back and fix it later.”

That was old, sinful, Dave. Today’s Dave views the world differently. Today’s Dave believes that design is something that is iterated with the code, and is only introduced when needed.

Let’s look at a somewhat simplified example.

RBAC Rabbit Hole

A while back I needed a way to share information among all the people who work on writing and producing books at the Bookshelf. Everyone was already using our giant Rails app in other ways, so I decided to bolt on a kind of wiki-like thing.

Then a little red figure with a pitchfork, horns, and a tail popped into existence on my right shoulder. “How are you going to protect key information from being altered by novices,” asked the miniature Prince of Darkness. “You’re going to need hierarchies of user access rights.”

Old, sinful Dave listened. Old Scratch was right. We had different levels of user: systems admins, managers, editors, book authors, and third parties such as copy editors and layout folks. Managers should be allowed to alter pages they created, as well as pages created by any of the editors who reported to them, and the authors who worked under those editors. Editors could update their pages, along with those of their authors. Poor old authors could only edit their own work, while third parties had no ability to edit anything.

That was my high-level design for the authorization, but before I opened up an editor, I’d need to do some more work on the lower levels.

Clearly I’d need a database table mapping users to roles. But, thinking ahead, I’d likely need to have different kinds of authorization for different resources, so I shouldn’t hard wire in my Wiki. That called for another database table containing the resource types (it would start off with just one row). I’d need to create the code to manage the entries in those tables.

Then I’d need to be able to determine if user A could edit page X. There was clearly a recursive step in there: a manager could change pages owned by their editors, and the pages owned by that editor’s authors. So I’d need to design a recursive SQL query, which sent me off on a day’s worth of exploring CTEs and how I could express them in ActiveRecord.

I started off needing a place to store notes, but with the Devil’s help I was going to do it properly, and implement a fully-fledged Role-Based Access Control system.

That felt righteous, so off I went down the rabbit hole.

Today’s Dave Would Do It Differently

Today, when that little red fellow makes an appearance, I try to ignore what he says.

Instead, I wait until I need something before designing it.

I’d start my wiki with an in-memory hash that mapped page names to page content, accessed via a trivial Page model. I’d get the wiki stuff working using this.

At that point, I’d make a decision. Clearly I need to store the data somewhere more permanent than RAM. And I clearly need to implement some kind of access control. Which should I do first?

On the basis of not doing things until I need to, I’d most likely choose the access control next, simply because it is likely to impact the data I need to store, and that data is easier to mess with while I’m just storing it in an in-memory hash.

I’d write the access control code as a placeholder method in my User model:

class User
  def can_edit_wiki_page?(page)
    true
  end
end                                                                                                                                                                                                                                     

Then I could update the editing code to call this method.

At this point, I wouldn’t be able to write any meaningful access-control tests, so I’d need a little more logic.

class User
  def can_edit_wiki_page?(page)
    page.owner == self
  end
end 

You’re right! This isn’t the recursive SQL query. It doesn’t use a role table or a resource table. But it’s enough design to see me through to the next phase, when I add some persistence.

And, you know what? I deployed it with just one minor change:

class User
  def can_edit_wiki_page?(page)
    self.admin? || page.owner == self
  end
end

Turns out I didn’t need any of that fancy design. Everyone was happy with it being simple.

The Devil Likes DRY

Recently I’ve come across folks who don’t like the Don’t Repeat Yourself principle, which first appeared 25 years ago in The Pragmatic Programmer. These folks say that slavishly removing duplicated code from your apps can make the designs more complex, and can lead to architectures where unrelated things are somehow conflated.

As a response, I updated the DRY section of the 20th anniversary edition extensively. I tried to explain that DRY is not about code duplication; it’s about the representation of knowledge. If two pieces of code that represent different pieces of knowledge happen to be identical, they are not a DRY violation, and it would be a mistake to factor them into a single place. (Assuming, of course, that the two pieces of knowledge are not just two statements of a single fact).

Viva Las Vegas!

It can’t be a coincidence that I most recently saw the Devil at work when I when to the Sin City Ruby conference. Fito von Zastrow and Alan Ridlehoover gave a great talk with a fantastic example of the way the premature design, combined with an incomplete understanding of DRY, can lead to some crappy code.

Their example was a machine that dispenses cups of hot beverages. Initially, it just knew how to make a cup of coffee. (The code that follows is just a sketch of theirs.)

def dispense
  heat_water
  drop_cup
  grind_coffee
  force_water_through_filter
  present_cup_to_customer
end

So far, so good. But then the requirement changes: the machine also has to dispense tea.

def dispense(options)
  case options.drink
  when :coffee
    heat_water
    drop_cup
    grind_coffee
    force_water_through_filter
    present_cup_to_customer

  when :tea
    heat_water
    drop_cup
    measure_tea_on_to_filter
    force_water_through_filter
    present_cup_to_customer

  else
    ...
  end
end

Here’s where premature DRYing kicks in. We look at that code, and think “the two parts of the case statement are identical apart from the one line in the middle of each that adds either coffee or tea to the filter. So the DRY principle says we should refactor.

def dispense(options)
  heat_water
  drop_cup

  case options.drink
  when :coffee
    grind_coffee

  when :tea
    measure_tea_on_to_filter

  else
    ...
  end

  force_water_through_filter
  present_cup_to_customer
end

The a new requirement comes in: we need to be able to add creamer to the coffee and milk to the tea. Obviously, the milk has to go in before the hot water, and the cream goes in only after the coffee has poured. We also need to add sweetener on request. Oh, and there the option for decaf coffee.

def dispense(options)
  heat_water
  drop_cup

  case options.drink
  when :coffee
    if options.decaf?
      grind_decaf_coffee
    else
      grind_coffee
    end

  when :tea
    dispense_milk if options.white?
    measure_tea_on_to_filter

  else
    ...
  end

  force_water_through_filter
  dispense_cream if options.drink == :coffee && options.white?
  present_cup_to_customer
end

Then along comes the request to add hot chocolate to the machine, with the option for whipped cream. Then we get a request for low-fat milk…

We started out with the best of intentions, but our initial refactoring, done to satisfy DRY, is leading us down a decidedly ugly path. We’re left trying to salvage the mess by adding more and more code as we try to split up the dispense method.

DRY is About Knowledge, Not Code

Our process broke down when we looked at this code:

def dispense(options)
  case options.drink
  when :coffee
    heat_water
    drop_cup
    grind_coffee
    force_water_through_filter
    present_cup_to_customer

  when :tea
    heat_water
    drop_cup
    measure_tea_on_to_filter
    force_water_through_filter
    present_cup_to_customer

  else
    ...
  end
end

Our DRY instinct kicked in, and we immediately made the design decision to fix it. But we were rushing things, updating the design prematurely. I’ve come to realize that being hasty like this almost always leads to worse code in the long run. Today Dave would probably look at the code I’d written and add a single line:

# TODO: dry?

Sure, there might be a DRY violation here, but it is currently doing no harm. Remember, all design comes down to “how easy it it to change?” Even though our first thought might be that the duplication in this code might lead to problems, we have no current evidence that this is the case. In fact, if we stop to think about it, there isn’t actually a duplication here at all, because DRY is about knowledge, not code.

all design comes down to “how easy it it to change?”

The body of the first when clause is the recipe for making a cup of coffee; the body of the second is the recipe for tea. Those are separate items of knowledge that just happen to share some steps. Thinking that there’s duplication here is like thinking that making an omelette and baking a cake are the same thing because they both involve cracking some eggs.

The Lazy Path

We feel justified to leave the code untouched, but then along comes the dairy requirement. Again, we don’t try to “do design” until we have a problem to solve, so we make the obvious changes:

def dispense(options)
  case options.drink
  when :coffee
    heat_water
    drop_cup
    grind_coffee
    force_water_through_filter
    dispense_cream if options.white?
    present_cup_to_customer

  when :tea
    heat_water
    drop_cup
    measure_tea_on_to_filter
    dispense_milk if options.white?
    force_water_through_filter
    present_cup_to_customer

  else
    ...
  end
end

At this point, I’m starting to feel that we do have a design problem: this method now has four different paths through it. in it’s initial form it was possible to take it all in at a glance, but it now requires active reading. So let’s split it up.

def dispense_coffee(options)
    heat_water
    drop_cup
    grind_coffee
    force_water_through_filter
    dispense_cream if options.white?
    present_cup_to_customer
end

def dispense_tea(options)  
    heat_water
    drop_cup
    measure_tea_on_to_filter
    dispense_milk if options.white?
    force_water_through_filter
    present_cup_to_customer
end

def dispense(options)
  case options.drink
  when :coffee then dispense_coffee(options)
  when :tea    then dispense_tea(options)
  else
    ...
  end
end

I’m liking this: each method is now a representation of a recipe. It’s easy to see what is going on, and how we’d add new recipes in the future. To me, this is DRY; each piece of knowledge is represented just once.

Could we do more? Of course. Right now each recipe starts with the same two actions and ends with the cup presentation. We could definitely extract that out:

def dispense_tea(options)  
  serve_hot_drink_in_cup do
    dispense_milk if options.white?
    measure_tea_on_to_filter
  end
  force_water_through_filter
end

But, again, I think that’s premature. We only have two recipes in our code, and two of anything is not a pattern.

Similarly, I look at the dispense function. Whenever I see a case statement used to dispatch to different behaviors, I want to replace it with a hash/dictionary lookup. But that’s also premature. Let’s make a # TODO note in the code, just to keep an eye on it as things change in the future, and move on.

Evidence Based Design

One of the keys to keeping things simple is to avoid doing stuff until it actually needs to be done. There are no rules that say you must stop what you’re doing and refactor just because two lines of code are the same.

Instead, good design comes from a discovered need; from evidence that your code is not as easy to change as it could be.

Sometimes we can anticipate that need, but the world doesn’t end if instead we discover it as we go along.

Design is not rules that you follow. It isn’t in charge of what you do.

Instead, design is a tool you use to make your code easier to change.

My latest book, simplicity, is now available at The Pragmatic Bookshelf.

Subscribe now

Share

How do we learn, really learn?

We experiment.

We create mental models, we test them, and we adjust them. We have theories that this new piece of knowledge might relate to some other piece of knowledge, and we test that assumption. Over time, we accumulate a web of things we feel that we know.

But what we really have is a self-consistent set of beliefs. New information is admitted to this web if it doesn't damage too much of the existing beliefs.

This process is what leads to bias. When sufficient numbers of people share sufficiently similar beliefs, they reinforce each other. The beliefs become rules, and believers become less critical; asking questions might damage the group.

I suspect it might be because so much of what we do is novel, but developers seem particularly prone to glorifying shared beliefs. Here are some that have graduated to dogma:

• everyone must do test-driven development

• agile beats waterfall

• agile doesn't work

• web pages should be rendered in the client/server (choose one)

• customers don’t know what they want

• new is better than old

• we know what we're doing

and so on…

As individuals, we need to be cautious of this phenomenon and avoid joining these soft cults as they surge past us in our social feeds. After all, if no rules are universal, acting as if a belief is actually a rule is pretty much guaranteed to make us wrong some percentage of the time.

So my advice is that you hear about something being “the way to do it” or, even worse, a “best practice,” take note, but consider what you're hearing to be nothing more than an opinion; someone else's belief. There are things you can learn from it—things you can integrate into your own belief system—but as with everything you allow into your head it needs to be tested and refined before it becomes yours. Be a skeptic, not an ardent disciple.

This advice applies to the stuff I say; it applies to the stuff everyone tells you.

None are rules.

Instead, view them as beliefs and suggestions. They’re not actions to take or rules to follow, they're just my beliefs that you might want to consider, test, modify, and maybe merge into your beliefs.

Truth is Contextual

No rules are universal.1 At the very least, a rule needs a context in which it applies. It's generally considered bad form to stick a knife into someone's throat, unless they're dying and you're performing a tracheotomy.

Context is a multidimensional concept. At a minimum, it includes:

• timescale
  We will do things differently depending on how long they are likely to take.

• group size
  Are we looking at a personal practice, or something that involves the team, or the company?

• impact
  We might behave differently if the outcome has a significant up or downside.

• moral and legal risk
  Is this another social media site, or are we creating a medical device?

• confidence
  How much uncertainty affects our actions?

And so on.

So, a plea. Stop for a minute and think about the things you feel are obvious, or that you believe because the group believes. Examine the basis for that belief. Is it universal? (I’m personally suspicious of any rule that doesn’t have exceptions.) Can it be tested to see if it is true? Can it be discussed rationally, or do things get heated?

Received wisdom is skin deep. True wisdom comes only from experience.

Subscribe now

Share

One of the problems of our times is that people are convinced that the only way they can win is for someone else to lose. Couple that with the fact that many people care more about winning than about the actual longer-term outcome, and you end up with a dysfunctional and noisy world.

A simple difference of opinion can end up as a confrontation. Some people enjoy that, and others walk away. Neither side actually wins in that situation.

No one wins a zero-sum argument

This approach is called dialectical reasoning. It assumes there are elements of truth in both sides of an opinion, and it emphasizes the benefits of trying to synthesize something new and bigger from them. Just as important, it turns something that threatened to be a confrontation into an interesting discussion.

Try This: Practice dialectical thinking

Here's an exercise. The next time you find yourself disagreeing with someone, try not to flip the bozo bit and assume they're an idiot. Rather than trying to show them they are wrong, instead genuinely try to work out if maybe you are. Be actively interested in what led them to their position, not because you're hoping to trip them up, but rather because it might give you insights. Your mission is not to prove you're correct, or that the other person is wrong. Instead, you are trying to find out which parts of each of your positions have merit, and from that synthesize a new understanding that's better than the two original ones.

This is tricky; it's easy to come across as condescending or maybe sly. You might start by saying "I hear what you are saying, but I need to understand more about Xyz, because it doesn't quite line up with my experience. Could you show me what led to your belief."

Never disagree; you're not trying to win. The exercise is to see things from their point of view. A nice side effect might be that you end up learning something.

Try This: Arguing with knives

There's a fun exercise that helps you practice seeing things from both sides. Next time you're at a lunch or dinner where half the people advocate one thing, and the other half advocate the opposite, suggest the following game.

Everyone who agrees with the position should place their knives (or spoons, or pens, or...) pointing into the center of the table (the way a knife is normally set). Those who disagree should place their knives at 90º to this, parallel to the edge of the table.

Let the debate run for a few minutes. Then pick a pair of people who disagree and flip their knives around; the person who previously advocated for the position must now argue against it, and the person who previously refuted it must now do their best to sell it to everyone else.

Keep swapping a pair of knives every few minutes. Try to give everyone a chance to cycle through both sides of the issue more than once.

It becomes a game: how can you marshal what you know to support something you disagree with. It's like high-school debate.

You're not trying to find winners or losers, or even to find a consensus. It's just an interesting way to spend 30 minutes with friends. But, on the way home, you might just find yourself reevaluating some of your strongly held views.

Absolute is the enemy of simple

Every time you hold an absolute and inflexible opinion, you’re very likely to be at least partially wrong. Even if you believe your idea is justified by science, try to remember that the fundamental tenet of science is to disprove what it believes.1

That means that there will be times when your firmly-held opinion may be contradicted by your direct experience. Now you have a choice. The rational approach is to treat the inconsistency as an opportunity to explore and possibly revise your belief. The most popular approach, though, is to come up with some kind of justification of why your belief is still 100% true and reality is mistaken.

One of the problems with lying about things is that you need to remember your past lies and make sure that everything is consistent with them. Subsequent lies now multiply the complexity of your imaginary world; most people struggle to maintain a consistent story when it’s informed by more than one lie.

Denying reality in favor or a belief is lying to yourself. Do it often, and you are forced to build a world view that is so convoluted that it takes a lot of mental energy just to maintain. Meanwhile, the decisions you make based on this baroque version of reality become less and less effective as you struggle to reconcile their outcomes with what you thought would happen.

Everyone lives in this state to some extent; it’s a consequence of the world being unknowable. But some people embrace opinions which are detached from reality. These are the people who are angry at the world, because the world is wrong. These are the people who make debate difficult, because they are not interested in exploring alternatives. These are the people who feel threatened by “different.”

And it all comes down to the fact that disagreement is not a zero-sum game.

Subscribe now

Share

This is an adaptation of one of the practices in my new book, simplicity, available now in beta from the Pragmatic Bookshelf.

Fear is a rational response to change. Fear is the enemy of change.

I could not find a source for this following graphic, but it quite nicely illustrates how fear is a barrier that keeps us contained within our comfort zones.

We developers share a nasty secret. We all know just how fragile the world of software is; how easy it is for a slight change to create some unintended consequence and take down an application.

So we're (quite rightly) frightened to make changes. We resist making them because we can't be sure that they will work. We put procedures in place—meetings, committees, sign-offs—so that when a change is made, the blame gets shared if something goes awry.

Don't get me wrong. Fear is a valuable feeling; it stops us from being reckless. But too much fear paralyzes us.

So here are some suggestions for exploiting the benefits of fear without having it getting in the way of change.

Good practices mitigate fear

This sounds obvious, but consistency is one of the most important ways to alleviate fear.

Have you ever worked on a project where the build instructions were a three-page document, and where deploying software involved copying a bunch of files into particular places on some servers? I have, I and can tell you that after installing a new release I was in a state of mild anxiety for the next day, waiting for the inevitable phone call from an irate customer or boss.

These days, all my deployments are automated: a single command will update any number of applications and services on any number of hosts. And now I deploy on pretty much every commit I make.

Automate to Reduce Fear

This can be taken further if you implement a Continuous Integration/Continuous Delivery pipeline. To do this, you'll

• need to be using version control,

• have a release strategy (perhaps using tags or branches),

• have decent tests,

• use automated deployment,

• and have a way of rolling a bad deployment back.

This is something that you grow into as a team: you shouldn't just announce one day you're switching to CI/CD, But if you have that as a goal, it will inform the decisions you make while constructing the project.

Having automated testing and deployment takes a lot of the fear out of changing software. What's more, the fact that deployment no longer takes a lot of ceremony or time means that you can deploy smaller changes. You won't be as afraid of a 2-line change as you will of a 200-line change.

Good Practices Include Good Code

Every change you make builds on the existing codebase. And if that code is unreliable, hard to understand, or just plain wrong, then you're quite right to fear any changes you make.

So maintaining good code hygiene is an essential requisite of approaching code changes without fear.

Take Smaller Steps

When you're unsure of the ground ahead, taking small steps is just good sense.

You might not be unfamiliar with the domain, uncertain of the customer's needs, or uncomfortable with the design.

Whatever the reason, the fear brought on by this uncertainty can lead to the infamous writer's block: the fear of the blank page.

Alleviate that fear by reducing the impact of getting it wrong. If every step you take is a small one, backtracking is easy.

Small Steps Are Safer

Sometimes you can drive this process with tests. Pure Test-Driven Design (TDD) asserts that you never write code without a failing test. AT the extreme, that means you can't start coding a new module without first having a test fail trying to reference the (nonexistent) module.

At a more pragmatic level, you can do something similar: write a test of something trivial, and then write the very simplest piece of code to make that test pass.

My advice is to be very cautious with this approach. It is very easy to start following a drunkard's walk (a process where each step is some random distance and/or direction from the end of the previous step) where successive tests add to the codebase but take you no closer to the goal.

Walk With Me

Sometimes you can involve your customer in the small-steps path.

They are asking for a bunch of functionality; do they need it all at once? Or is there some small subset that could be delivered early and that would still give them some business value?

:Step With Your Customer

The benefits to your customer include:

• they get to see their software as it is being developed, given them the opportunity to learn and to make adjustments;

• they get a better feeling for progress and likely timescales;

• they start seeing a return on their investment early;

• they'll start to see other opportunities for development.

You also get many benefits:

You can approach the project in smaller, less scary, chunks. This goes a long way to reducing the fear of starting.

• You get to shake down your build and deployment practices.

• You get feedback, and any changes are easier to make because they're localized to a smaller delivery;

• * You get the confidence to make future changes.

Most importantly, though, doing this builds a relationship between you andyour customer; they'll know that, even if you deliver stuff that's not quite right, you'll make it better. You'll know that your customer will change their minds, but will work with you to find ways to make the corresponding software changes easier.

In this environment, handling change turns from being a confrontation to a more enjoyable "you bet!"

Cheap Code Is Easier to Throw Away

When I'm writing something in a new domain or using new tools or technologies, I worry that I'll spend a month creating something only to discover that, finally, I understand, and that what I've just done is basically junk.

I fear that because I've done it—many times.

Nowadays I overcome that fear by prototyping; writing code to try out ideas that I know I'll be throwing away. This code will be ugly. It won't have any error handling (unless I need to handle errors to explore the system).

It might not be in a repository, and will only have tests if tests are the cheapest way to learn what I want to learn.

Prototype To Resolve Fear

I recently used GraphQL in a project. Having never used it before, I started with a piece of code that sent a simple query off to an API. To get that working, I had to work out which GraphQL library to use, how authentication worked, how to express queries and create fragments, and what error responses looked like.

The resulting code would have made Dr Frankenstein proud: snippets of code cobbled together from a bunch of web sources, GitHub archaeology, and random experimentation. Half-way through the second day, something flipped in my brain and I felt like I got it. So I created the repo for the actual project and started coding. Maybe an hour later I had what I felt was well-structured and resilient code chatting away with the service API. I broke my own rule and kept the prototype code lying around in a temporary directory: it was a useful playground when I wanted to try out something new and didn't want to mess up the main project.

“We Have Nothing To Fear But Fear Itself”

Churchill’s aphorism still sounds good (until you stop to parse it). The problems we anticipate are not the enemy; they are simply the challenges. It’s the fear of those problems that we have to overcome. And hygiene, small steps, and experimenting are just a few of the many charms we can use to banish fear whence it came.

Subscribe now

Share

Apparently, a bunch of larger companies have decided that, as AI makes their more senior developers more productive, they can thin the ranks of more junior staff.

I believe this is a short-sighted move for all the usual reasons, and also for a reason I haven’t heard expressed before.

The Usual Reasons It’s a Bad Idea

• With no junior developers, where are tomorrow’s senior developers coming from?

• If the end result of this move is that all code is AI generated, then where do those AI train on new paradigms, frameworks, languages, and so on?

• I’m anticipating a massive and public failure some time soon: a large company going out of business because of code that no-one understood did something catastrophic. When that happens, the rest of the world will retrench somewhat, and all those junior devs you let go today are suddenly going to be in demand.

Another Reason It’s a Bad Idea

Junior developers (really, any junior members of staff) have a real and unique benefit.

No, it’s not that they’re cheap.

It’s that they are both naive and enthusiastic. They are the toddlers of the industry, untouched by the world-weary cynicism of the rest of us. Unlike us, they don’t know that “you do it this way because we’ve always done it this way.” They don’t know that an idea is ridiculous, or that something can’t be done. Like toddlers, they’re happy to try stuff and learn, and they almost expect to fail often—it’s no big deal.

This has always been an untapped resource. We rarely listen to these tyros when it comes to making decisions; they are basically cannon fodder, thrown in to do the grunt work.

But what if, when the AIs make your senior devs 5x more productive, you pause before laying off the rest. You take 20% of that savings, and use it to keep your junior devs at work.

But change the focus. No longer view them as code generators. Instead, treat them as what they are: the next generation of senior devs. Give them space. Let them make mistakes. Try to listen to their crazy ideas. I’m betting that some of those ideas, once tamed, could fundamentally improve the way you do things.

I also bet that these junior devs, who have grown up with AI, will find new, and better, ways of exploiting these coding assistants, ways that might not even occur to people who are adapting to AI after a career spent manually coding.

It requires a little faith, and a little courage. But it might just be the factor that lets you succeed while the more short-term companies decline.

Subscribe now

Share

I can’t help it. Just like the lab rat pushing the button to get the next food pellet, I love delivering features; creating and delivering something new. In the past, I was known for delivering features the customer didn’t even know they wanted. That’s a superhero play.

It’s also ridiculously stupid.

Every feature we add to our software represents a future liability, because all that code will need to be supported, maintained, extended, and understood by new developers.

Does that stop everyone adding feature after feature? No.

Should it?

Yes, sometimes.

Need Driven Development

There’s a basic rule for keeping software simple: don’t write code unless someone needs the functionality it provides. And “wouldn’t it be cool if…” and “it’d be really easy to add…” don’t count as need.

Obviously, need depends on your context. If your company is creating a mass-market product, then the marketing folks will likely be keeping an eye on competitors and creating feature requests to keep up and surpass them. That’s genuine need.

But if you’re creating software for a single client or internal use, then the concept of need becomes more interesting. It can also be tainted by politics.

I’ve spoken to managers who have to budget for software development. Every year, they prepare a report saying what they want to build, how much it will cost, and what value it will produce. They’ll receive back an approval to spend some amount, normally lower than their request. So what do they do? They add all the features they think they might need to the project before they submit it. This both bumps up the price and also acts as a form of insurance; they don’t want to ask for a baseline project this year only to get turned down for the extensions they need next year. This applies to both internal and external clients.

But we can be more agile than that. Back when I was delivering software to paying customers, I’d flip the equation around. Rather than talking about need, I’d ask the customer where they’d extract value from the project, and what their priorities were for doing so.

Then the team and I would go away and assign a level of effort to delivering these values. This is purely internal to the team: we just needed a sense of how big the task was. The result of the guess would be as simple as “easy,” “medium,” “difficult,” and “unknown.”

We’d take these rough estimates and compare them to the value they’d deliver. Assuming the dependencies would allow it, we could then work out which features we could deliver early that would also give the customer a decent amount of business value.

Then we’d go back to the customer and say something like:

We’ve been looking at your requirements, and we think we can give you some options. This first is a conventional project, where we deliver the software to you at the end.

With the second option, we’ll deliver the software incrementally, feature by feature. We looked through your list of values, and combined them with our project planning, and we feel that we can deliver feature X well before the final project is complete. You should be able to start using it then, getting business value early. We can talk nearer the time about subsequent deliveries, but right now we think we might look at features Y and Z next.

This gives the customer two big wins. First, they can start earning some value from the software earlier; value now is always better than the same value later. Second, your customer gets to experience using this first deliverable. In my experience, when the customer first uses a partial delivery, they suddenly realize things about the system that weren’t apparent when it was just some words in a document. They’ll be able to come back to you with ideas and suggestions for next steps that might be radically different from their original request. In fact, a while back I had a customer who took the first two of five feature deliveries and then said the system did just what they needed; the extra three features were there for insurance, and they weren’t required.

It also gives you some massive wins. First, you get feedback a lot earlier than you would. Ideally you’ll be talking with the customer every week, but often that just doesn’t happen. But you’ll definitely get their attention when you deliver something. Second, you start to build a trusting relationship with them. When the project started they were taking a risk on you. When you deliver something, even if it’s not the complete project, they’ll start feeling a lot more comfortable, and you’ll find that you’ll get more open and useful interactions with them. Finally, this approach catches potential problems early: if you have a misunderstanding that would propagate throughout the code, finding out early stops a whole lot of rework later.

Simplicity

You can’t ship software that does nothing. But you can work to reduce the number of features, delivering only what’s needed, free from unnecessary embellishments or complications, and delivering it earlier. Features can always be added later.

Your turn

Look back at some projects you’ve delivered. What percentage of the code in each gets used on a daily basis? What percentage would you guess never gets used?

This was an extract from my new book, simplicity, now in beta at pragprog.com.

Subscribe now

Share

• I work in the smallest increments I can, using feedback at all levels to assess the value being produced.

• I constantly adapt what I do, but only after I’ve decided how to tell if the impact of the change was good or bad.

• I deliver value incrementally, as soon as I have something new to show my users. That way I can use their feedback to guide my next steps.

I’m not always successful. In 2023/4 I wasted nine months because I went heads down on a complex project. I got caught up in the technicalities and forgot to deliver incrementally. I’m still learning. But the vast majority of the time, I stick to these principles, and it works well.

Break It Down

Here’s how I apply these principles. There are just three activities.

1: Orient

• I find out where I am.

2: Step

• I take the smallest possible step towards where I want to be, doing the least amount of work possible to generate something that gives me some feedback.

3: Learn

• I assess whether what I did produced the expected value.

• I adjust my understanding based on the feedback.

• And I go back to the first activity, and reorient myself.

Honestly, there’s nothing special about this; it’s really just common sense. But it only works if you consciously follow the three steps.

All Of agility Comes Down to Orient, Step, Learn

Here’s an example. This afternoon I had a call with Susannah, my wonderful editor. She mentioned that she could no longer build my book; she got a “file not found” error. Every other book built just fine for her. She sent me the console log. (I think it’s so cool that my editor uses the command line…)

I went through the three activities in a cycle. (And, yes, I’m laboring the point a little. Bear with me.)

Orient #1

I tried building my book on my machine and had no problems, so the issue was likely a difference between her environment and mine. One thing she mentioned was that she stored all her book projects in a directory with a space in the path.

Step #1

We’ve had problems with paths like this in the past, but we fixed them and haven’t seen any issues for years. But, in lieu of any better idea, the simplest thing would be to check the book out into a path that contained spaces on my machine and see if it worked. It failed just as hers had done.

Learn #1

Despite my strong belief going in to this, it was the space in the path causing the problem. Time to reorient my thinking. Back to step 1.

Orient #2

I knew the book built successfully on her machine last week, but not this week, so I was probably looking for something I’d messed up over the weekend. But I wasn’t aware of changing anything to do with paths.

Step #2

The simplest thing was to scan the change logs. I saw no modifications to any of the actual tools, but I did see that I’d bumped the version of JRuby that we use.

Learn #2

Given that this had to be related to code, and the only code that changed was the third party JRuby compiler, I had a suspect.

Orient #3

Over the years I’ve taught myself that when something breaks, 99.999% of the time it’s my fault, and not a bug in some third party code. So the orientation here was to convince myself that a quick test was needed.

Step #3

I tried running jruby -v and it also failed.

and so on…

In the end, it was a bug in the jruby script in the newly updated package. I rolled back to the previous version and reported it to the JRuby team on GitHub. Charles Nutter acknowledged it within a couple of hours, and a patch was available a few hours later (which is seriously agile).

But Dave…

I know what you’re thinking: “That’s nothing special. That how everyone would attack an issue like this.”

Maybe—but not in my experience. I’ve given many classes to all levels of developer, and most people do not take a methodical approach to this kind of problem. I have sympathy, because neither did I.

Also, remember that these three steps apply to everything I do, from naming functions though picking a technology for our new website: Orient, Step, Learn, repeat.

Subscribe now

A Larger Example

The publishing industry is mired in the Victorian times. All information is exchanged in spreadsheets, with people at one end manually creating them and people at the other end transcribing information into some 1990s mainframe system. We receive monthly distributor statements this way, but we process them in code.

A few years back, our international distributor started creating a different monthly spreadsheet. We adjusted our ingestion code to match. A couple of months later, it failed: they’d renamed some columns, and added a few new ones. This triggered an Orient, Step, Learn cycle.

During orientation, we determined that this was likely to be an ongoing problem, and that we’d likely have to adjust our import now and then going forward. We also knew that we had to retain the ability to reimport all historical data, so our code would have to add new CSV formats while still accepting the old. This mean that we’d need to decouple to reading of the raw data from the processing of that data. A pipeline was born.

Step one was to make that split in the code explicit and run regression tests to make sure nothing had changed. Not much to learn, so we went back to the orientation step.

We had to deal with two types of change: the order (and number) of columns could be different, and the column names would often change (sometimes with typos, so it was clearly created by hand every month). We needed to decouple the finding of the data we needed from the layout of the spreadsheet.

Step two was to replace the code that analyzed the spreadsheet with a table that mapped column names to internal variables so that the internal value total_sales could be extracted from a column called Sales one month and Sale total the next. Again, we ran regressions to verify the step.

Orienting again, we needed to make sure that we could read all historical imports while adding new layouts.

The next step was to use the tables we’d defined in the previous cycle to pattern match against the incoming spreadsheet. If the column names and locations matched a table, we’d use that for the extraction. If no table matched, we’d abandon the import and someone would have to create a new mapping table.

Again, we ran regression tests, this time against the accumulated months of data. At this point we’d learned all we could until new data arrived.

Over the following months, the distributor managed to throw a few more surprises at us: columns with the same names as before but with different values, different formats for currency values, and so on. We used each of these to trigger a new Orient, Step, Learn cycle. Often we’d find that we’d spend more time on the Orient and Learn steps with these changes, as we tried to find patterns in the changes, and tried to build in more safeguards to ensure we weren’t misinterpreting the data.

Over the last 36 months, there have been 15 different formats sent to us. Sometimes we’ll get the same format for three months in a row, and sometimes a format from the past will suddenly reappear.

I draw two lessons from this. First, the initial orientation was critical; working out that we needed backwards compatibility informed all the decisions we subsequently made. In particular, splitting the code into separate parsing and processing steps turned out to be the right step.

The second lesson was the benefits of the small steps. I am absolutely paranoid about royalty calculations: getting authors and editors paid accurately for their hard work is critical to me. When dealing with something as uncertain as this feed, taking lots of small steps, and running full regression tests at each, gave me the confidence to deploy these changes.

The Benefits

It’s a simple plan, but Orient-Step-Learn is the fundamental essence of agility. It makes me conscious of the changes I make and the feedback I need when making them.

I find there are so many benefits to following this ritual:

• Pausing to orient myself before attacking my keyboard saves my butt many times a day. I realize I was about to solve the wrong problem, or that I’d already solved the same problem elsewhere, or that I didn’t actually understand what I was about to do.

• Being deliberate about feedback focusses what I do in each step; I only do work that’s contributing to generating a successful result.

• I give myself permission to experiment and fail, knowing that the cost is minimal because I’ve taken just a small step.

Pause, Then Act

It can also be a difficult and sometimes painful plan to implement. It means slowing down and giving up the illusion of control. (Hint: we never really had control in the first place.)

It means constantly experimenting, and constantly using those experiments to improve what you do and how you do it. It means getting used to the idea that failing is an integral part of succeeding.

Your Turn

Please try organizing your work into these three practices. It’s a purely personal thing: you don’t need permission to do it. It’s simply a way of organizing how you think about what you do

In practice, it’s probably less work that you’re doing now, and it’s more productive. But it also requires that you build a set of habits, so that you naturally think about feedback when you both start and finish anything. And, as with any habit, this means that initially you’ll need to do it consciously and deliberately. Maybe write on an index card and tape it to your screen.

And, obviously, use your daybook to record all of this. (And if you aren’t using a daybook, perhaps have a look at this article before continuing.) At first, record every decision you make, and go back when you finished that particular step and record the feedback; a tick if it worked and an angry X if not.

And, just to keep you on the straight and narrow, never start something until you’ve established what feedback you’ll use to decide whether it was successful.

The rest of this chapter is a deeper dive into Orient, Step, and Learn.

Try this

Write “Orient, Step, Learn” on something and stick it in your line of sight while working. Or, stick it in a file and make that the default buffer your editor shows. Or decorate your desktop. Maybe tattoo it on the inside of your eyelids.

Do whatever it takes to remind yourself to be deliberate as you work, consciously taking the three steps as you move forward.

(This is going to sound silly, but you might try saying “Orient” aloud as you start an activity, then “Step” when you’ve decided what to to, and then “Learn” when it’s done. I do this all the time when I’m trying to adjust my habits; I find that vocalizing seems to engage a different part of my brain, and doing that speeds my adoption.)

Think about

Do you suppose this idea might apply to things other than software development?

This was an extract from my new book, simplicity, now in beta at pragprog.com.

Subscribe now

Share