With your SpecFlow installation comes SpecFlow.exe that is a program that can be used to generate tests from the scenarios AND to create nicely formatted reports from the generated test results. There’s been a lot written on how to generate these reports when your using NUnit (see this and this for example), but when it comes to managing this for MsTest there’s been almost silent. And facing this problem I can sure see why… It’s a bit trickier.
In this blog post I want to show you two things; how to generate MsTest’s from your .feature-files and how to create a report from the generated results. Finally I’ll show you how to put the two together and link it up to a nice “External tool”-button in Visual Studio. Here we go:
Generate MsTest’s from your .feature-file
With this step you can generate the test from you scenarios. SpecFlow.exe picks up your configuration and generates the test in your test framework of choice. From the look of it, it seems quite straight forward. Using the “help command” of SpecFlow.exe, “specflow help generateall” produces this help:
Generate tests from all feature files in a project usage: specflow generateall projectFile [/force] [/verbose] projectFile Visual Studio Project File containing features
OK. I put together a .bat file that does that. (Note that I’m on a 64-bit machine and had to use some funky DOS-shortcut to get there. No simple way to be architecture agnostic I’m afraid). Here is my file:
"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe"
generateAll Specs\Specs.csproj
/force /verbose
pause
I’ve added some line breaks for readability.
Well – that was easy.
Create a report from the generated results
To get this step to work we have to run the test and get hold of the location of the test report file (.trx). When you do this from within Visual Studio the test reporting is done in a TestResult folder and the file get a name with a timestamp. That is not very script-friendly sadly and we’re forced into writing a bat-file that also run the tests.
I used the MsTest Command line reference to put together this .bat-file:
if Exist TestResult.trx del TestResult.trx
"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe"
/testcontainer:Specs\bin\Debug\Specs.dll
/resultsfile:TestResult.trx
pause
Again, watch out for me using a 64-bit Windows and use %programfiles% if you’re not.
Some strangeness with MsTest made me delete the testResults.trx before each run. Also MsTest create some folders (username_testrun…) but that doesn’t bother me now.
When we now have a created .trx file we can run the command that creates the report for us. According to the “documentation” (specflow help mstestexecutionreport”):
usage: specflow mstestexecutionreport projectFile [/testResult:value] [/xsltFile:value] [/out:value]
- The projectFile is the Visual Studio Project File containing features and specifications.
- Test Result refers to the .trx file generated by MsTest And it defaults to TestResult.trx
- Xslt file to use, defaults to built-in stylesheet if not provided
- out is the generated output file. Defaults to TestResult.html
I’m happy with the defaults of the last three since I choose my .trx-file name … wisely. :) So my whole .bat-file becomes this:
"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport Specs\Specs.csproj
pause
A low and behold; it actually produces the nice report we wanted:
### Putting it all together
That’s neat – we now have three different .bat files that we need to
click in consecutive order ;)
No really – the first one (generate test from features can most
certainly be handled by Visual Studio in most cases. Or in any case will
probably not run in conjunction with the other two steps.
But to run the tests and produce a report would be nice with a single
file. Here it is:
``` brush:
if Exist TestResult.trx del TestResult.trx
"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:Specs\bin\Debug\Specs.dll /resultsfile:TestResult.trx
"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport Specs\Specs.csproj /testResult:TestResult.trx
pause
```
And looking to this blog post I’ve also created a parameterized version
of it that I can hook up to a “external command” that does that with a
single click. That changes the bat-file into this:
``` brush:
if Exist TestResult.trx del TestResult.trx
"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:%2 /resultsfile:TestResult.trx
"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport %1 /testResult:TestResult.trx /out:TestResult.html
echo Created file TestResult.html
```
So you need to send the name of the test container (the .dll) and the
project file. Save that file to a known location so that you can point
your external command to it.
Finally you can create a external tool button in Visual Studio and set
the parameters as follows:
The arguments to the external command are:
- $(ProjectDir)$(ProjectFileName)
- $(TargetName)$(TargetExt)
Note that the project with specifications has to be selected before the
External command can be run.
### Conclusion
In this blog post I’ve learned how hook up SpecFlow to generate tests
and reports for me.
You can
download my code here