Deciding What to Build Before You Build It

If you’ve been following along in the series so far you’ve probably got the basic setup of your analytics system under control. Everything is in source control, has appropriate names, and you’re using a reliable staging area. You feel like you have things under control. Then a new request comes in.

Your boss asks, “How long will it take to make all the reports for this new ERP system?”

“The one I haven’t seen yet?” you reply. “How could I possibly give you an estimate? I don’t even know if it has all the data we need. I don’t know how it’s stored or what level of complexity we’re dealing with. Any estimate I give would be completely made up.”

“Okay, but if you had to? What would it be? Three weeks?”

“Tell them whatever you want. Just don’t hold me to it.”

Three weeks later….

“Hey, why aren’t those reports done yet? We promised them to management by now. What’s taking so long?”

Planning to fail

Somewhere in the world right now there’s almost certainly a team conducting their daily stand-up meeting. Everyone is sitting down, taking turns recounting in mind numbing specificity what they did yesterday, what they are going to do today, and completing their report with the benediction, “No blockers.” Any outsider not versed in tenets of Agile would call this a status report meeting. There’s no talk of priority or requirements. Requests for assistance or stories about difficult portions of the work are met with a shrug and knowing nods from the rest of the group. There’s talk of Jira. And the sprint is probably over, or just started. Nobody is sure and someone might need to check the calendar. Retrospectives are either non-existent or are pro-forma, producing no change.

Most projects are late or canceled. Or even worse, the ones that have forward momentum are blocked halfway through by outside forces. But no project ever fails, it’s just de-prioritized.

If you watch this strategy play out in enough places, you’ll see why the industry reputation for on-time delivery is so bad. There’s better ways to plan our projects. Lots of better ways, in fact. But before we look at at least one way to build a better project, let’s take a detour to talk about how management got this way in the first place.

It starts at the top

Executive leadership treats software like construction. Come up with a plan and execute it. That’s how most of their other departments work. When HR does payroll they can tell you how long it will take and what it will cost. If they’re wrong it’s probably not by much. Executives aren’t programmers, so they don’t see why the web app or data pipeline you’re building should be any different. They’re also under pressure from their bosses, the shareholders, or owners. The shareholders are pretty easy to understand. They want profits. They want to be presented with an easy to understand spreadsheet of how revenues will exceed profits.

CTO’s, VPs of engineering, and even some of the technology middle management work primarily as middleware between the executive leadership and the developer teams. Their primary challenge is fending off executive needs for guarantees against dev teams complete inability to provide them.

Your haggard-looking boss probably deserves a little sympathy. Being constantly asked why things are behind schedule only to be given shrugged shoulders by your team is a drag. If they live this way long enough they’ll start demanding timelines from you, even if they are complete fantasy. Anything to get through this quarter. When the next quarter comes along, it’s a repeat of this cycle. You do that for a few years, then move to a new job. The unluckiest among us think this is normal. It’s not. Let’s look at some better options.

Fixing the executive team

Come on, we can’t do that.

Deciding what to build

If I were less scrupulous, this is where I’d try to sell you a new SAAS project planning service. It would be very expensive and require consultants to implement and have a twelve week official training course. If you’ve been reading the series so far you know the real answer is some up-front work and attention to fundamentals. We need to change both the method of communication we use with leadership, and our planning method. If you’re an Agile acolyte, be forewarned, this is gonna sound a bit like “waterfall”.

The strategy we’re going to use I like to call “Decide what to build before you build it”. Sure, most teams have a basic idea of what they’re going to build. It’s a web app. It takes in data. It has user accounts. There are images. How complex can it be?

In most organizations the default planning method is “we’ll just keep writing code until it kinda works.” There’s lip service given to sprints, kanban boards, or Agile, but structure is usually abandoned once scheduling pressure is applied. The end result is a series of code rewrites based on whatever is the most on fire at the moment. Whether this is ‘real’ Agile or not, this is how most teams operate.

Instead of this, let’s write a functional spec!

The idea of requiring a spec doc will probably be seen as heretical to most engineers. If you’ve used one it was probably six pages of extensive checklists that no one actually used. Or, if they did, half of it was blank, and no one knew where it was in Confluence because no one knows where anything is in Confluence. Instead, let’s write a better spec doc with just three mandatory fields, and put it where everyone can find it.

1. Who is this for? Be as specific as you’re able. This could be “users of our web app” or as direct as “Janice and Armid in accounting”. These are the people who get to approve this spec. For projects with large audiences, pick someone to be their representative.

2. What will the software do? This should be a list of bullet points. Use as much detail as possible. The points should be entirely about outcomes. For example, “the service will take this document and transform it into a parquet format and drop the file in this cloud storage account.” Include bullet points about things this won’t do as well. If the accounting team was fine with parquet, but wanted excel and csv, include a bullet point “we discussed csv and excel output as options, but aren’t supporting that in this version.” When you’re two days away from shipping and someone interrupts you and says “this thing supports csv, right?” you’ll be glad you wrote this in your spec.

3. What are the risks of this project? Remember all those ways that your last ten projects were late? Do they still apply? Will this work under high data volumes? Can the current infrastructure support it? Are we using new frameworks with different feature sets than we’re used to? Did we check to make sure the lead developer isn’t going on vacation for three weeks? Are other priorities for the business likely to interrupt this and make it late? Are we being too optimistic about what we can do? Include everything you can think of here.

Resist the urge to include any other sections. Don’t include technical designs either. Those are for the engineering team to hash out during implementation. Getting bogged down in technical implementation is the surest way to create an engineering war and never actually start the project. Technical design happens after functional requirements are defined.

Don’t include the extra seventeen mandatory fields from your other docs. Don’t require checklists of all the technology you plan to use. Don’t include timeline proposals. Don’t include version numbers.

Every addition to the spec makes it more likely you’ll skip the whole thing. Keep it simple.

Get buy-in

Once you’ve done this exercise and your spec is complete, what do you do next? Well, now you have to schedule a meeting. It hurts me to suggest it, but sometimes it needs to be done. But this is the good kind of meeting, where we make decisions. The sole purpose of this is for spec approval. Invite everyone from bullet #1. Don’t let anyone skip it. You can’t fill them in later. That one person who couldn’t make it but promises to read the spec and get back to will wait two weeks only to demand it support excel format after it’s too late and development is already underway. Get everyone. If they really can’t make the meeting no matter how hard you try, they don’t get a say in the spec approval. Sorry.

If anyone not in bullet #1 wants to attend the meeting, let them know it’s for spec approval only and you’d prefer any “helpful suggestions” to be made before the meeting. This is not the time to discuss how this affects Q2 profits or how much faster the code would run in Rust.

Before the meeting, distribute your functional spec. Ask everyone to read it and come ready to agree or disagree. During the meeting, go over each of the do/won’t do from bullet #2. For each point in the spec, ask everyone, in these words:

“If I make it do this, is that okay?”

Sit there with an awkward pause as long as you need. This is the important part. The users need to agree on what it will do. No work should start until you have these answers. You don’t know what you’re building until you get them.

A few things are likely to happen. First, there will be objections. If they’re small and easy to work around, update your spec. You just saved lots of development time.

Second, users will argue with each other about what it really needs to do.This is also great, because now you can get them to agree on something. Make sure to write it down in the spec and put their names on it. You may need the evidence later.

Third, users will swear up and down that something you put in the “won’t do” category is essential and they can’t survive without it. Maybe they’re right, but try a few of these questions:

“How would that affect your workflow with this feature?”

“Why is this essential?”

“What about my proposed solution to this problem won’t work and why?”

“Is this the only way that result can be achieved?”

You can either agree with them and add it to the spec requirements or get them to stand down. Frequently these requests become fodder for the next version. If your intuition is that it’s a silly request, a few months of waiting will prove you right or wrong. Features that are “‘do or die”’ end up forgotten about surprisingly often.

The other important point of this meeting is to ask for what might be missing from this spec. If you’re dealing with workflows that you are less familiar with, there may be things you’ve entirely neglected. You may have assumed everything needs to be a data pipeline, when a locally run script could do all the work instead. Or the data you’ve been asked to import to the data lake could easily be accessed from the source instead. Be receptive and willing to make changes. This is an important method of getting the users to believe you are working in their best interests. (You are, right? You want them to like your feature, don’t you?) Engineers prefer to shy away from these kinds of interactions, but the value of getting people on your side is enormous. Get out of your comfort zone and hang out with the normies. You might like them.

At the conclusion of all this you’ll either have an approved spec or lingering questions. If you’ve got an approved spec, you can move on to the next phase. Most likely you’ll have a few questions that can be answered with a little bit of extra research. If it requires another meeting to discuss, set one up. More often you’ll get a tentative approval, pending the answer to a few outstanding questions. Answer them and document them. Keep your receipts.

In rare instances I’ve seen these spec docs get completely blown up. If the premise of the feature just doesn’t work, or the approach is entirely wrong, you might be unable to get approval. That’s okay. If halfway through your approval you’re still nowhere close to agreement, call off the remainder of the meeting and rethink your spec. A senior engineer with good domain knowledge is unlikely to need to start from scratch, but don’t discount the possibility. And don’t beat yourself up. It’s still better than shipping the wrong thing. Remember that project that took an extra year and never really worked? It probably could have been avoided entirely with a better spec.

The building part

There’s a classic mistake that a lot of engineers make. You start by looking at all the things you can currently work on, and do those first. It seems reasonable. You set up the repo and the dev environment ,and you start cranking out some boilerplate for the things you know you’ll need later. When the boss asks how it’s going you have nothing but good news. Progress is flying!

Unfortunately, now all your roadblocks are at the end of the project. Whoops! You’ll get 60% of the way done only to realize your language of choice doesn’t support what you’re trying to do. Now you have the awkward job of explaining why you’ll be shipping much later than you said, or not at all because it’s actually impossible to do the thing in the first place.

What do you do instead?

Brush off that spec you just wrote, and go directly to the ‘risks’ section. One by one, do everything you can possibly think of to eliminate those risks. If you expect a certain level of performance and you’re not sure if the system will support it, write some dummy code as fast as possible and test that assumption. If you need something from the infrastructure team, request it first and be loud about it. Systematically go through all the risks to the project and minimize or eliminate all those potential problems. They are disruptors lying in wait and your goal should be to eliminate them all. Only then is it safe to do the easy stuff.

Engineers are usually uncomfortable with testing out ideas instead of diving right into coding. There is an implicit fear that we are too slow. This a mistaken assumption. The real issue is a lack of trust between users and the dev teams, and you can’t fix that by looking fast. Don’t be afraid to make statements like “I need to verify that ‘x’ is possible otherwise the whole project doesn’t work, so I’m going to spend a few days on that first and make sure it works.” Doing the hard parts first makes estimating at the end of the project much easier. If you’re mostly done and all that’s left is boilerplate connector code then when you say “I’m almost done” it might even be true! And when you start providing accurate estimates you build trust.

Frequent objections to this method

Objection #1: “I can’t talk to the customer.They ask for ridiculous features that’ll never work.”

There is an idea floating around in the back of most engineer’s minds that goes something like this: “Nobody understands software and they all have unreasonable demands and we’re better off just keeping our distance because there’s no way they’d have anything useful to add.”

Banish the idea from your mind.

Sure, users don’t understand software, but they understand the work they’re doing, and would probably make similar comments about you. “These software guys have no idea what we do all day and the stuff they build never works right for what we’re trying to do and I wish they’d just yadda yadda.”

The way to get past this is with a little humility. Ask them how they are doing their job and what they’re trying to accomplish. Then bridge the gap by using your software expertise to design something that solves their problems. The worst designs are when the user tries to “design software” instead of just telling you the root objective and letting you design it, or when software teams design based on their intuitions of the user expectations instead of actually working with them. Take their feedback, but don’t make them do your job for you.

Objection #2: “We already use a five page PRD document for everything, and it’s just a bunch of wasted effort. Why should I bother with this?”

Most teams’ use of the PRD (product requirements doc) is an artifact of “Let’s write lots of documents and somehow that’ll make it better.” The PRD is a mostly unread document that doesn’t make any demands on the business users or have any real requirements. Documentation only works when the parties involved agree to use it as their contract for behavior. There’s a reason I include only three elements on the functional spec, because those are the only ones that matter. Engineers don’t like writing documentation in the first place, so don’t ask them to do more than they need to. If you’re going to require documentation, be consistent. No team likes having tons of required documentation that everyone ignores. It just wastes time.

Objection #3: “The design is constantly changing. What’s the point of a spec doc if it just goes out of date?”

This approach works even better for designs that iterate. By keeping the functional requirements written down, it’s easy to get in touch with the users when things change. And if the design change is something not included in the requirements you know you can change it without another twelve meetings to discuss. This is not a document written in stone. It’s an agreement about what you’re building. If you need to propose a change, get the users together and agree on the change. Update your document and keep working. We don’t live in the fictional world where technology or requirements never change. Incorporate it into the process.

Why doesn’t everyone do this?

Because it requires some planning and forethought. We’d all prefer the fiction that we can just have a daily meeting and it’ll all work out. Plan in the beginning to avoid failure in the end. As an industry we’ve spent decades dealing with project planning mired by eight different bosses on three different levels asking where your TPS reports are. We’ve tried adding bean bag chairs and work from home, but the problems remain. Creature comforts do nothing to fix a poorly planned project. If your days are spent staring at tickets with empty descriptions, uncertain of whether what you’re doing has any value, this is the work that will get you out of it.

Being good at project management is a super power well worth learning. Especially if you like shipping code.