Specification by example and Model based testing

Posted by Marcus Hammarberg on October 31, 2012
My current client, Spotify, is really on forefront when it comes to good, solid engineering practices. There are inventors, leaders and all around great people where ever you look. So this gig has rendered me a lot of learnings, even in the areas that I don't touch much (being hired as an agile coach that reads coding people...).

So the first week or so heard the term "Model based testing" for the first time in my life. Investigating further I realised that it had strong similarities, technology-wise, with Specification by example. And there's some fundamental differences in the thinking and reason behind each approach too.

In this post I'll take a brief look to clear out the differences and similarities  I think that you can have great use for both approaches but you should probably know why or you'll find yourself fighting the tools and process for a better fit.

Model based testing

Model based testing is a great idea and described in a great way on the wiki-page about it. I especially like the picture, that tells it all.

Basically you create a model of the system that shows the aspect that you want to test. You know, all models are just showing part of the system and are hence a simplification.  These model can be drawn as diagrams or maybe in a high level language like for example Cucumber (just sayin' :))

From this model abstract tests can be derived and maybe even generated for you. Finally you can then supply the concrete implementation for these tests and have them executed against the system under test, what's often called instrumenting the application.

So why on earth would you go through this problem. There's a number of reason but one big one is that a model is much easier to reason about than the nitty-gritty details of the implementation of the automation. So it's improved communication for starters.

Then the details change. A lot. The model behind the system more seldom so. Say for example that you're testing the login-in procedure for your site. The model stays the same even though you might change the implementation, looks and other features of the actual implementation, right. The model is more robust than the implementation.

Finally with a model in place we can start to reason about the system. For example, as the excellent tool GraphWalker does. In graph walker you draw the model of the system as a finite state machine (hey, look at me, mr Fancy Words). This graph is run by GraphWalker. Here's an example from their site:


So firstly this is nice since you only have to supply the implementation for the "edges" (or transitions between states for you and me lay-men) but also GraphWalker can make sure that:
  • we run all edges a certain number of times
  • distribute the "load" between different edges so that we get a nice realistic load on our system
  • make sure that we have taken all the roads through the graph (what's known as the travelling salesman problem)
So in short; Model Based Testing is a way to test software systems, from an abstract model. This model is great for reasoning about and gives you the possibility to make sure that all paths are covered etc.

Keyword around this: test.

Specification by example

Specification by example, from the site; "is a set of process patterns that facilitate change in software products to ensure that the right product is delivered efficiently."

Although automation is common pattern that many team uses I would say that the main thing is communication. Not only that but communicating in the form of concrete examples written in the language of the business. By doing just that we bring questions, hidden complexity and misunderstanding to the forefront of our process instead of discovering them later in the process, when we test the application for example.

So the main thing is communicating with each other in an unambiguous language that everybody in the team understand. As a nice side effect of that you can use that written communication as tests for your system. There are a lot of tools in this area of which I've used Cucumber the most.

When you use the specifications with example to automate the code, architecture and structure resembles the one that Model Based Testing end up with. You have steps in a Cucumber .feature-file while a MBT graph have edges. These are then automated with code behind the scenes in both cases. And you probably want some sort of DSL or abstraction over your system in order to keep that implementation maintainable. 

Keyword: communication.

The difference

Only by looking at the names of the approaches we can get a clue on the big difference in approach and usages; model based testing or specification by example. 

Specification by example is a communication tool. It makes sure that we have understood the same thing. That we are building the right thing. By coming up with concrete examples we are forced to think about how the system should work, before we implement it. 
It's implemented using a model in the form of plain english and as a side effect you can automate tests to verify that your system behaves as specified. 

Model based testing focus more on testing and getting a robust abstraction over the details that probably change. It's great for regression testing your system and can, if your model is complex enough, be used to distribute your "test load" when it comes to variants over path, running through all the paths in the model or just run the tests against different environments, such as browsers for example. 

With MBT, you could draw the model before you have the implementation in place and hence get an interesting discussion going on if the model is describing the thing we are going to build. And then get verification that it's like that by executing the model. But that's not where MBT shines. 

And with Specification by example, you could write all your tests and make sure that every path is executed in Cucumber (or another way). But it would be cumbersome and akward, you'll probably have to write your own cucumber runners to get it to execute serveral times. It could be done. But it's not where Specification by example shines. 

Summary

In this post I've talked about two great tools in your software development tool-belt; specification by example and model based testing. 

They are very similar in implementation but fundamentally difference in the reasoning behind them; what they are for. 

Model based testing is used for testing. To, after the system is built, make sure that we have built the thing right

Specification by example has as main focus to make sure that we understand each other. That we are actually building the right thing.  


Published by Marcus Hammarberg on Last updated