Specification by Example for Reluctant Product Managers

Soon after I started E-accent back in 2005, the engineer I hired passed me a first iteration of a new application and asked, “Hey, can you test this?”

I was happy to help, but wasn’t sure how. “What exactly do you want me to do?” I asked.

“Just see if it works,” he replied.

I started to test, and found that the application worked very differently from what I had imagined. So I made a list of changes.

I thought these helpful suggestions would go down a storm. After all, we all want to improve the outcome, right?

The engineer didn’t quite see it that way. “These changes come with a lot of rework. Why didn’t you tell me earlier?”

Right there and then, I learned about one of the most important problems in software development: the illusion of agreement.

The illusion of agreement, and why it matters

Before development begins, you have to reach a shared understanding of what the end product needs to do.

The problem is that users (and product managers) are often tempted to talk about solutions before the underlying problem is fully understood. For example, they will tell you that they need a report or a notification or a calendar, which isn’t necessarily the appropriate solution. And even if it is, your mental image of the solution isn’t necessarily the same as theirs. The map is not the territory.

As a result, you wind up building what you think is required, only to discover later on that the solution isn’t really fit for purpose, and may end up redoing the whole thing.

Functional specifications

The obvious solution is to write a brief that everyone can agree on before work starts. And traditionally, a business analyst does this by writing a functional specification, usually a Word document describing, in great detail, how users should be able interact with the software.

Having all the details worked out beforehand sounds good in theory. But in practice, there are several problems with it:

• The document can get extremely long, so it becomes nearly impossible for a developer to absorb and remember all the relevant details from it. In one particular project, a business analyst handed me a specification of over 600 pages. As the project progressed, it became fiendishly difficult to keep track of how all the details in the document corresponded with functions hidden away within 25,000 lines of code.
• Despite its colossal length, the spec still leaves too much room for interpretation. Human language is ambiguous, so it’s too easy for a developer to “joins the dots” in the wrong way.
• Over time, the source code grows and changes as new features are added and the brief evolves. But the original specification is never updated, so it quickly gets out of sync with the code as it stands.

As a result, a functional specification that starts out with good intentions winds up being a white elephant that nobody uses or cares for.

User interface as specification

One approach to this problem – first proposed by 37signals – is to do without the functional specification altogether and let the user interface (UI) be the documentation.

At first, we loved the idea of having a single source of truth for everything. But when we actually put it into practice, we found that relying on how things work in a UI can be quite overwhelming – especially when you’re dealing with elaborate or complex workflows. Sometimes there are tons of steps you need to go through before you get the part of the workflow you need to understand to work at all. And you have to replicate that sequence of steps each and every time you want to troubleshoot or make changes.

Basically, we needed some sort of specification – the question was, what?

Our first experience with Specification by Example (SBE)

When researching how other teams were dealing with specifications, I found an approach called Specification by Example, which seemed to solve all problems at once.

Specification by Example is:

• a formal specification
• in one place
• accessible to the whole team
• using concrete example scenarios
• written in formal but human-readable language
• that can be executed as an automated acceptance test.

We decided to use it for our next project.

That project was an application for a care organisation in the Netherlands. In this country, patients who need long-term care (typically elderly people or those with a chronic disease or a disability) can opt to manage their own care (nursing, cleaning, physical therapy etc) based on a budget they receive from the government.

Governments love to complicate things, and in the Netherlands that means that part of your care budget comes from the national government, and the rest from the municipality. These two budgets are linked, and surpluses and deficits can sometimes be carried forward.

What’s more, the national government asks for reporting per calendar month, while a municipality might specify four weeks. By the end of the year, these two periods are hopelessly out of sync, leading to some crunchy calculations.

So we were looking at some pretty complex scenarios, and testing them manually would have been a time-consuming nightmare. This was a good opportunity to try SBE.

We started off with setting up a documentation system that synced with our automated testing suite, so that everybody was looking at the same thing. Then we worked with the business analyst on a set of examples which we turned into functional tests.

The tests ran automatically whenever we made a change in the source code, and as long as the code didn’t conform to the spec, the test failed and we knew there still was work to do. We knew the test was covering our backs, and that gave us confidence. We didn’t have to do cumbersome manual tests every time, let alone troubleshoot issues after the fact (which is extremely hard and painful in a complex application).

The crowning glory came when the client introduced a new idea with a calculation example. When we converted it into an acceptance test and started working on the code, a pre-existing test failed, because the calculation example had a hidden error. The testing suite was nipping potentially serious problems in the bud.

Meanwhile the client, who had access to the executable specification, quickly figured out what had to be changed in order to make the new concept work.

It was clear that we had saved ourselves a lot of trouble by having one single source of truth that we could all read and access.

How SBE works

Essentially, SBE works by expressing software requirements as real-world scenarios rather than abstract statements.
To illustrate, here’s a simplified example of an executable specification:

It’s just a list of requirements and a couple of tables with sample data. But when you look a little closer, these examples provide plenty of important insights to talk about.

Firstly, what’s the deal with these accounting periods? Why do we have one covering a calendar month, and another covering a four-week period that spills across two calendar months? Are there any other types of period? What about the last four-week period of a calendar year – does it end on December 31, or does it stretch into the new year?

Secondly, that nursing visit on March 4 starts at 11:52 pm, and ends in the morning of March 5. What if that appointment starts in one accounting period and ends in another? Does that mean that the amount needs to be split pro rata into two accounting periods?

Then there’s the surname Kılıç, which has non-Western characters. We’ve had issues in the past where those got printed incorrectly, as the database management system was configured to use a Western European character set. We need to use a Unicode character set instead.

As these issues show, as a product manager you can use your insight into the business behind the software, as well as your own creativity and intuition, to sharpen up your examples and make sure they focus on all the important things.

Benefits and drawbacks

SBE has some major advantages:

• It’s precise. Using concrete examples, and jointly working through the thought exercise to find the ones that matter most, lets you reach a much more specific agreement on what the software needs to do.
• It’s fast. Your developers don’t have to hang around waiting for you, as product manager, to test the software. They just run the test and a few minutes later they’ve got the confirmation they need.
• It’s responsive. The documentation and code don’t go out of sync, because they can’t. The documentation is a test that fails as soon as the underlying code is changed, so no software can be delivered and pass the test unless both are in sync.

However, SBE can backfire when it’s not implemented correctly.

Above all, you need to make sure you get the right examples. The crux of SBE is finding examples that are representative for the workflow you want to support – ideally, based on real-world data. Making assumptions is a dangerous game.

Another common pitfall is to test through the user interface instead of underneath it (we fell for that mistake). Let me expand on that a bit.

The problem of testing through the user interface

Some testing suites run automated tests through a browser, interacting with the user interface to test the application behind it.

Intuitively, it sounds ideal – like having a “robot user” who uses the software just like a human would. But in practice, it yields a lot of false positives, and tests also run very slowly this way.

That’s why it’s much better to test functionality underneath the user interface, rather than through it.

For example, let’s say we have a functional test for a form that a user needs to submit. If we went through the user interface, using a browser, our test would read along these lines (in pseudocode):

• When all fields have been filled in,
• Click the “OK” button,
• Validate if the next screen has a message saying “Your data was submitted successfully”

OK great, the test passes. But what if the designer makes a quick text change in the UI, like this?

Shoot, the test fails. Whereas it really shouldn’t, because the designer didn’t touch the actual functionality. They just made a change in the copy.

Now contrast that with how the test would work underneath the UI. In pseudocode terms, it would be something like this:

• When the form is submitted,
• Validate that a new record was created in the database,
• … and that it contains the first and last names as entered by the user

This way, the text of the submit button is irrelevant. Which is good, because we don’t care if the button reads “OK” or “Save”. In fact, you want designers to be able to make improvements in the UI when necessary (after all, the UI is their domain) without having to worry about breaking tests. In an acceptance test, all we need to know is that the application is capable of taking data from this form, creating a new database record, and correctly storing first and last names in it.

The only time you should test the user interface with an automated test is when there’s a specific concern there. For instance, a complex design with lots of moving parts that’s simply too cumbersome to test manually. Or, an error message that a user really needs to see in case of a serious problem.

Recommendations

If you’re familiar with any of the problems I’ve been talking about – frequent misunderstandings with your development team, exhausting and repetitive manual testing procedures, outdated documentation – you may want to start implementing Specification by Example.

A benefit of SBE is that it doesn’t just work for new projects – it can be retrofitted into existing ones too.

Implementing SBE takes investment from both managers and developers in money and time, and they might not be convinced of the benefits right away. If that’s the case, start small. Document and automate one workflow, and start reaping the benefits. Once management and developers see the return, you’ll have less trouble getting them on board to go all the way. After fully implementing SBE, you’ll find that the development process is back under control, instead of controlling you. Your work days will look a great deal brighter.

Further reading

• Adzic, Gojko (2011). Specification by example: How successful teams deliver the right software. Manning. ISBN 9781617290084.