My current client, Spotify, is really on forefront when it comes to good, solid engineering practices. There are inventors, leaders and all around great people where ever you look. So this gig has rendered me a lot of learnings, even in the areas that I don’t touch much (being hired as an agile coach that reads coding people…).
So the first week or so heard the term “Model based testing” for the first time in my life. Investigating further I realised that it had strong similarities, technology-wise, with Specification by example. And there’s some fundamental differences in the thinking and reason behind each approach too.
In this post I’ll take a brief look to clear out the differences and similarities I think that you can have great use for both approaches but you should probably know why or you’ll find yourself fighting the tools and process for a better fit.
Model based testing
Model based testing is a great idea and described in a great way on the
wiki-page about it. I especially like the picture,
that tells it all.
Basically you create a model of the system that shows the aspect that
you want to test. You know, all models are just showing part of the
system and are hence a simplification. These model can be drawn as
diagrams or maybe in a high level language like for example
Cucumber (just sayin'
:))
From this model abstract tests can be derived and maybe even generated
for you. Finally you can then supply the concrete implementation for
these tests and have them executed against the system under test, what's
often called instrumenting the application.
So why on earth would you go through this problem. There's a number of
reason but one big one is that a model is much easier to reason about
than the nitty-gritty details of the implementation of the automation.
So it's **improved communication for starters**.
Then the details change. A lot. The model behind the system more seldom
so. Say for example that you're testing the login-in procedure for your
site. The model stays the same even though you might change the
implementation, looks and other features of the actual implementation,
right. **The model is more robust than the implementation.**
Finally with a model in place we can start to reason about the system.
For example, as the excellent tool
GraphWalker does.
In graph walker you draw the model of the system as a
finite state machine (hey, look at me, mr Fancy
Words). This graph is run by GraphWalker. Here's an example from their
site:
So firstly this is nice since you only have to supply the implementation
for the "edges" (or transitions between states for you and me lay-men)
but also GraphWalker can make sure that:
- we run all edges a certain number of times
- distribute the "load" between different edges so that we get a nice
realistic load on our system
- make sure that we have taken all the roads through the graph (what's
known as the
travelling salesman problem)
So in short; Model Based Testing is a way to test software systems, from
an abstract model. This model is great for reasoning about and gives you
the possibility to make sure that all paths are covered etc.
Keyword around this: **test.**
### Specification by example
Specification by example, from the site; "is a set
of process patterns that facilitate change in software products to
ensure that the right product is delivered efficiently."
Although automation is common pattern that many team uses I would say
that the main thing is **communication**. Not only that but
communicating **in the form of concrete examples written in
the language of the business**. By doing just that we bring questions,
hidden complexity and misunderstanding to the forefront of our process
instead of discovering them later in the process, when we test the
application for example.
So the main thing is communicating with each other in
an unambiguous language that everybody in the team understand. As a nice
side effect of that you can use that written communication as tests for
your system. There are a lot of tools in this area of which I've used
Cucumber the most.
When you use the specifications with example to automate the code,
architecture and structure resembles the one that Model Based Testing
end up with. You have steps in a Cucumber .feature-file while a MBT
graph have edges. These are then automated with code behind the scenes
in both cases. And you probably want some sort of DSL or abstraction over your system in order to keep
that implementation maintainable.
Keyword: **communication.**
### The difference
Only by looking at the names of the approaches we can get a clue on the
big difference in approach and usages; model based **testing** or
**specification** by example.
Specification by example is a communication tool. It makes sure that we
have understood the same thing. That we are building the right thing. By
coming up with concrete examples we are forced to think about how the
system should work, before we implement it.
It's implemented using a model in the form of plain english and as a
side effect you can automate tests to verify that your system behaves as
specified.
Model based testing focus more on testing and getting a robust
abstraction over the details that probably change. It's great for
regression testing your system and can, if your model is complex enough,
be used to distribute your "test load" when it comes to variants over
path, running through all the paths in the model or just run the tests
against different environments, such as browsers for example.
With MBT, you could draw the model before you have the implementation in
place and hence get an interesting discussion going on if the model is
describing the thing we are going to build. And then get verification
that it's like that by executing the model. But that's not where MBT
shines.
And with Specification by example, you could write all your tests and
make sure that every path is executed in Cucumber (or another way). But
it would be cumbersome and akward, you'll probably have to write your
own cucumber runners to get it to execute serveral times. It could be
done. But it's not where Specification by example shines.
### Summary
In this post I've talked about two great tools in your software
development tool-belt; specification by example and model based
testing.
They are very similar in implementation but fundamentally difference in
the reasoning behind them; what they are for.
Model based testing is used for testing. To, after the system is built,
make sure that we have *built the thing right*.
Specification by example has as main focus to make sure that we
understand each other. That we are actually *building the right
thing*.