In recent years, user scenario testing based on customer stories is very popular in Open Source world, and inside Microsoft. There is a mind shift for testing which is that testing should focus more on what customer need, but not finding functional bugs. Acceptance Test-Driven Development (ATDD) is one of the wonderful testing practice, one that is easy to learn, fun to practice and that speeds up testers’ productivity. It also puts testing more in the driver’s seat, so that quality is driven upstream, rather than taking a back-seat to development.
In the paper, I will use SpecFlow which is an open source .Net tool for ATDD and show a real use case in the DAC Logical Import and Export improvement. The user scenario tests can be run on any test framework and testing different product interface without change a single line of code. This talk is very good for PMs. You can learn how you can write runnable and measurable and tracking user stories instead of hard track functional spec. Come and learn how you can establish a shared definition of Done among all disciplines, and make sure the product we make meet customer’s requirement by using the real and up-to-date user stories in the whole improvement life cycle.
Introduction to ATDD
“Behavior driven development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project. It was originally named in 2003 by Dan North as a response to Test Driven Development, including Acceptance Test or Customer Test Driven Development practices as found in Extreme Programming. It has evolved over the last few years. BDD focuses on obtaining a clear understanding of desired software behaviour through discussion with stakeholders. It extends TDD by writing test cases in a natural language that non-programmers can read. Behavior-driven developers use their native language in combination with the ubiquitous language of domain driven design to describe the purpose and benefit of their code. This allows the developers to focus on why the code should be created, rather than the technical details, and minimizes translation between the technical language in which the code is written and the domain language spoken by the business, users, stakeholders, project management, etc.”
Acceptance Test Driven Development (ATDD) is a practice in which the whole team collaboratively discusses acceptance criteria, with examples, and then distills them into a set of concrete acceptance tests before development begins. It’s the best way I know to ensure that we all have the same shared understanding of what it is we’re actually building. It’s also the best way I know to ensure we have a shared definition of Done. In this paper, I did not distinct between BDD and ATDD since I believe the ideas behind are the same.
Two excellent books related to how to write good user storied and how to do ATTD can be found at
Acceptance Test Driven Development (ATDD) is a practice in which the whole team collaboratively discusses acceptance criteria, with examples, and then distills them into a set of concrete acceptance tests before development begins. It’s the best way I know to ensure that we all have the same shared understanding of what it is we’re actually building. It’s also the best way I know to ensure we have a shared definition of Done. In this Wiki, I did not distinct between BDD and ATDD since I believe the ideas behind are the same.
You can look at the attached document for detail instructions. SpecFlow is hosted at github, please start with this Wiki page, you can easily to write your first ATDD tests in 10 minutes.
There are a couple of good books related to ATDD. The books are not only for testing, it also talks about why acceptance testing is important, and how to manage a project using ATDD.
Introduction to SpecFlow
In this section, I will introduce SpecFlow, an open-source .NET tool that lets you write acceptance specifications using plain text. You can install SpecFlow at http://specflow.org/downloads/installer.aspx. It requires Visual Studio 2008 or Visual Studio 2010. After you install the software, you can star the journal of acceptance testing driven development in Visual Studio. In following section, I will describe several steps to write the first tests.
Describe the Scenario
Like TDD, ATDD use the same ourside-in approach, which is that we describe what we want to build before doing any code, and we use the acceptance tests to guide the writing of code.
To describe the scenario, SpecFlow use a natural-language way to specify behaviors of a software system in terms of “Given, When, Then” (GWT) sentence fragments. This language is called Gherkin which was original created for another popular ATDD tool called Cucumber, which is in Ruby language. Because you can describe scenarios in the language of your domain and user interface, your resulting feature files should make perfect sense to business experts. So, unlike traditional unit test driven development, this methodology can finally deliver on the promise of executable specifications that are actually useful as documentation.
To specifying the behavior, you first create a plain-text file (called a “.feature” file) containing a set of scenario definitions written in Gherkin. To start to write a feature file, you can first create a Test Project in Visual Studio, which will create necessary references to MsTest framework. Then, you can add a SpecFLow Feature file using the [Add New Item] dialog shown below
For example, here is the template features SpecFlow generated:
In order to avoid silly mistakes
As a math idiot
I want to be told the sum of two numbers
Scenario: Add two numbers
Given I have entered 50 into the calculator
And I have entered 70 into the calculator
When I press add
Then the result should be 120 on the screen
The feature section describes a “feature” or an improvement which we are developing. In here, it is the Addition function. We use the three simple sentences to describe what is the feature, what is the persona, and why we need to feature. These three simple sentences clearly described the goal of this feature.The feature was supported by a couple of user scenarios, while each scenario is encoded as a scenario section. A .feature file can have a single feature section and many scenario sections. The syntax is extremely flexible: SpecFlow only cares about the Gherkin keywords “Given”, “When”, “Then”, “And” (plus a few others); everything else is simply matched against your regular expressions.
You should easily be able to understand in high-level terms what the application does by reading the Gherkin specifications. Just imagine these specifications are the written result of a design meeting where you’ve got your client or boss to tell you how the user interactions should work.
In order to build run SpecFlow based MsTest, you need to do following configuration. First, you need to add TechTalk.SpecFlow.dll which installed with SpecFlow as the reference, and adding following app.config:
<unitTestProvider name=”MsTest” />
Another simple way to configure your project is using NuGet Package Mangament tool at http://nuget.codeplex.com/ to install SpecFlow Library which is very user friendly to developer.
After you finish the initial configure step, you are ready to build and execute your first ATDD tests. First, build your project; you will see there is a .feature.cs file generated. This feature is a SpecFlow system generated test file. We will discuss how the test generation works.
Right click the .feature file you created in step 1, and click [Run Tests] button, you will see that there is a test named [AddTwoNumbers] is running, and the result is [Inconclusive], which means that the tests haven’t be implemented yet. In next, we will show how we implement the tests.
Implement Your Tests
The way to implement SpecFlow test is to define a SpecFlow Step definition step. A step definition step contains a small piece of code which implements a Gherkin clause, i.e., Given/When/Then clause. Here is the default step definition step template if you add a new [SpecFlow Step definition step] in Visual Studio (SpecFlow is very user friendly, If you double-click the test result for previous step, you will see that it actually generated the missing step definition step for you):
public class StepDefinitions
[Given(@”I have entered (.*d) into the calculator”)]
public void GivenIHaveEntered50IntoTheCalculator(int number)
[Given(@”I have entered 70 into the calculator”)]
public void GivenIHaveEntered70IntoTheCalculator()
[Then(@”the result should be 120 on the screen”)]
public void ThenTheResultShouldBe120OnTheScreen()
[When(@”I press add”)]
public void WhenIPressAdd()
The Binding keyword tells SpecFlow that it is a Step Definition File. As you can see in the example, we have several methods, and each method has some attributes which matches the text defined in the .feature file. In this way, SpecFlow will know that how to match a Gherkin clause with a method.
By default, we don’t have any implementation, so SpecFlow adds ScenarioContext.Current.Pending(); to indicate that the feature is not implemented yet. Then, as a developer, you need to fill in the implementation and make the acceptance tests passes.
How we use SpecFlow
We start to use SpecFlow testing tool in SQL Server Manageability Dac Testing Team four month ago. The first improvement we used is the logical export and import improvement. Here is a real example which I written for the improvement:
In order to port data from different version of SQL Server and SQL Azure
As a DBA,
I want to import and export a database to/from a file or stream without data lose
Scenario Outline: Export and Import Round-trip
Given a predefined <database> database
When I export the database to a file
And I import the file I exported to a new database
Then the newly created database has the same content as the old database
| database |
| pubs |
| northwind |
And here is a piece of code which implement one scenario step:
[Given(“a predefined database (.*)”)]
public void GivenISpecifiedInputDatabaseName(string scenarioName)
ScenarioDatabaseType scenarioDatabaseType =
(ScenarioDatabaseType)StringEnum.Parse(typeof(ScenarioDatabaseType), scenarioName, true);
ScenarioContext.Current.Add(Tag_ScenarioDatabase, new ScenarioDatabase(scenarioDatabaseType));
Reuse code with Parameterized Step Definition
Another important feature is that the attribute below
[Given(“a predefined database (.*)”)]
uses Regular Expression to match the Gherkin clause, and transform the correctly text into the parameter of the method, which will us to reuse the same method for many Gherkin clauses.
Using Scenario Outline for Parameterized Tests
As you can see from the above example, we can use Scenario OutLine and Examples, which allow you to specify one or more examples to support a scenario (outline). This enables us to write data driven tests much easily. Under the hook, SpecFlow will generate a test case (or a parameterized test if the test framework support it) for each line in each example.
Writing Test Framework Independent Tests
The beauty of SpecFlow is that it can support any kind of Test Framework. Just change the config to
<unitTestProvider name=”NUnit” />
It will generate NUnit Test Cases.
We implemented two Test Generators: LTMSpecFlowTestGenerator and TestShellSpecFlowTestGenerator, which allow people to use existing test framework with SpecFlow.
In our case, we generate MsTest based tests when we run test locally. Then, we build LTM test case in the build lab, and run them under WTT.
Writing Implementation Independent Tests
Someone, one feature might expose UI, API or PowerShell. In this case, how can I write once, and test both? SpecFlow allow you to write multiple implementations (the Feature definition files), and decide which implementation you want to run against it by using config:
<stepAssembly assembly=”MySharedBindings” />
Generate Feature Coverage Report
There are quite a lot of articles talking about ATDD and BDD. In here, I list several relevent articles for specflow, You can following Web pages to understand how SpecFlow used in real cases.
> The SpecFlow Google Group is a good place to search and ask for answers
> Search The SpecFlow tag on StackOverflow for answers.