Right now I’m very impressed with the way that two my colleagues (Hugo Häggmark and Tobias Karlsson) has introduced BDD or rather Specification by example at their client. They have in a very short time gone from people not speaking to each other (analysts and developers) to collaboration around the scenarios of their specifications and demos running from those specs. I’m officially impressed! Another colleague even threw together a speech engine integration that speak the SpecFlow features as they are run. Silly but …Coolness! I got a very interesting and I think common question from Hugo that I thought I’ll share here, since the answer might be useful to others.
Hugo asked me if I update my specifications with any bugs found later on. Or if not – where do I keep them?
The reason for adding bugs in the same .feature file as the specification itself is of course compelling; over time you would get a long list of every change ever made to the specification. Every bug we’ve fixed and it’s always up-to-date and automatically verified against the system. Nice…. but are there some drawbacks?
Bug scenarios – yes please
Firstly let’s agree on that creating Cucumber style scenarios for bugs that occurs is a very good thing indeed. It’s a nice little work item and it automatically sorts the “I cannot reproduce”-problem. Also when the scenario is complete you have nice regression for bugs that you have fixed. I also recommend this a good starting point for people who haven’t done test automation before. Start by automating a bug or two making sure that they don’t come back once you have fixed them. But when it comes to adding them to the specification…
Specification with key scenarios
In the excellent Specification by example book, Gojko Adzic comes back again and again to the concept of key examples. And that a specification (as thought of in Specification by example) in essence is made up with a user story and key examples that helps us understand the story.
It’s not every possible way that the user story can be played out… It’s key examples to help us collaboratively understand the story better.
So back to Hugo’s questions; should I add scenarios for bugs under the specification? No probably not – is my answer. The specification will, over time, be cluttered with a lot of scenarios and the nice idea of a user story with examples that gives clarifying examples is lost.
So no bug scenarios then?
Aha – I didn’t say that! I still think that creating bug scenarios is a great idea, but it can be done in different .feature files. Then they might cut the system in a different way as well to better suit the description of the bug.
They can be tucked away in a different folder and tagged with some tags (@bugfix or @regressionTest) that you only run on daily builds, keeping your continuous build quick.
Maybe a folder structure for you feature-file could look something like this:
Specification (a separate Visual studio project probably)
- Specifications
- Fixes
- Effective User Interface Testing
- …
- …
I’m sure your creativity is better than mine when it comes to making up different groups of features.
How many key examples should a feature have then?
So one could wonder; how many scenarios / key examples is correct for a specification then? Well … being a consultant you know I have to say; It depends.
But I’m sure that 1 is to few and 10 is to many. Even 8 is stretching it and probably showing us that we should have a new feature instead. You should at least show one successful and one failure example of using the story (logging in successfully and logging in with failures for example).
Enough for everybody in the team to be sure that we all mean the same thing and agree on how it should work. It really comes down to what you are specifying and the complexity of the business logic.