Specification by example and Model based testing

· October 31, 2012

My current client, Spotify, is really on the forefront when it comes to good, solid engineering practices. There are inventors, leaders and all-around great people wherever you look. So this gig has rendered me a lot of learnings, even in the areas that I don’t touch much (being hired as an agile coach that reads coding people…).

So the first week or so heard the term Model based testing for the first time in my life. Investigating further I realized that it had strong similarities, technology-wise, with Specification by example. And there’s some fundamental differences in the thinking and reason behind each approach too.

In this post, I’ll take a brief look to clear out the differences and similarities. I think that you can have great use for both approaches but you should probably know why or you’ll find yourself fighting the tools and process for a better fit.

Model based testing

Model based testing overview

Model based testing is a great idea and described in a great way on the wiki-page about it. I especially like the picture, that tells it all.

Basically you create a model of the system that shows the aspect that you want to test. You know, all models are just showing part of the system and are hence a simplification. These model can be drawn as diagrams or maybe in a high-level language like for example Cucumber (just sayin’ :))

From this model abstract tests can be derived and maybe even generated for you. Finally you can then supply the concrete implementation for these tests and have them executed against the system under test, what’s often called instrumenting the application.

So why on earth would you go through this problem? There’s a number of reasons but one big one is that a model is much easier to reason about than the nitty-gritty details of the implementation of the automation. So it’s improved communication for starters.

Then the details change. A lot. The model behind the system more seldom so. Say for example that you’re testing the login procedure for your site. The model stays the same even though you might change the implementation, looks, and other features of the actual implementation, right. The model is more robust than the implementation.

Finally, with a model in place, we can start to reason about the system. For example, as the excellent tool GraphWalker does. In GraphWalker, you draw the model of the system as a finite state machine (hey, look at me, mr Fancy Words). This graph is run by GraphWalker. Here’s an example from their site:

GraphWalker example

So firstly this is nice since you only have to supply the implementation for the “edges” (or transitions between states for you and me lay-men) but also GraphWalker can make sure that:

  • we run all edges a certain number of times
  • distribute the “load” between different edges so that we get a nice realistic load on our system
  • make sure that we have taken all the roads through the graph (what’s known as the travelling salesman problem)

So in short; Model Based Testing is a way to test software systems, from an abstract model. This model is great for reasoning about and gives you the possibility to make sure that all paths are covered etc.

Keyword around this: test.

Specification by example

Specification by example, from the site; “is a set of process patterns that facilitate change in software products to ensure that the right product is delivered efficiently.”

Although automation is common pattern that many teams use I would say that the main thing is communication. Not only that but communicating in the form of concrete examples written in the language of the business. By doing just that we bring questions, hidden complexity, and misunderstanding to the forefront of our process instead of discovering them later in the process, when we test the application for example.

So the main thing is communicating with each other in an unambiguous language that everybody in the team understand. As a nice side effect of that, you can use that written communication as tests for your system. There are a lot of tools in this area of which I’ve used Cucumber the most.

When you use the specifications with example to automate the code, architecture

Twitter, Facebook