IntelliTest – One Test to rule them all
July 5, 2015
In traditional unit test suites, each test case represents an exemplary usage scenario, and the assertions embody the relationship between the input and output. Verifying a few such scenarios might well be enough, but experienced developers know that bugs lurk even in well-tested code, when correct but untested inputs provoke wrong responses.
Generating traditional unit tests
IntelliTest generates traditional unit test suites. When run on a piece of code-under-test, the IntelliTest engine tries to generate a test suite with high code coverage (in particular, it focuses on branch coverage by default). It does so by iteratively synthesizing precise concrete inputs that let it execute deeper and deeper into the code-under-test. The generated traditional test suite contains an individual test case for each such concrete input and assertions on the corresponding observed output. The inputs include edge cases that the developer might have missed.
‘How is that testing whether the code is correct or not? How useful is a test suite that provides high code coverage for an add method that might actually be doing a multiplication?’ one might think.
Even without knowing about the intended and correct behaviour, the IntelliTest generated test suite describes the actual observed behaviour of the code under test. By itself this is useful as characterization tests, protecting existing behaviour against unintended changes.
Testing for correctness
Testing for correctness can be introduced naturally into this approach using assertions. Assertion statements (for e.g. Debug.Assert) are all compiled down to branches – an if statement with a then branch and an else branch representing the outcome of the predicate being asserted. Since the engine computes inputs to exercise branches, it ends up exercising such assertions as well. Thus, if the code-under-test contains assertions representing its correct behaviour, then IntelliTest ends up generating a test suite that validates such correctness. Assertions connect code coverage and correctness.
One Test to rule them all
As a developer our understanding of the code-under-test will eventually grow beyond what can be represented by a few individual example test cases. By understanding the code-under-test and the individual unit tests, we might arrive at a more general representation of its behaviour. Instead of saying ‘for this input, this is the expected result’ we might be able to say ‘this method performs this kind of a computation on its inputs’ – instead of saying ‘codeUnderTest(2) should return 4’ one might be able to say ‘codeUnderTest takes an integer and doubles it’. We have hoisted exemplary usage scenarios to a level where they capture a general relationship between inputs and outputs. Such a relationship can be encapsulated in a method that looks like a test method but is universally quantified – any assertions we make must hold for all possible input values – universally quantified assertions or “for all” assertions, if you will. For example, the following is one such universally quantified method that asserts that after adding an element to a non-null list, the element is indeed contained in the list:
Assumptions work well in such cases, placing restrictions on the input data and exposing developer intent along the way, and IntelliTest provides a rich Assumptions API. The example shows the use of the special assertion API that IntelliTest provides, but we can as well use the ones that come with the test framework. IntelliTest emits a stub for such a universally quantified method – this is the parameterized unit test method (PUT or “the” IntelliTest) we discussed in the previous post. It serves as the partial specification for the code-under-test. Elaborating it does not require or introduce any new language or artifact. It’s written at the level of the actual APIs implemented by the software product, and in the programming language of the software product. IntelliTest exploration can be launched directly on the code-under-test or on this parameterized unit test to generate and update the suite of traditional unit tests. This unlocks the full potential of IntelliTest.
We started this post saying that IntelliTest tries to generate a suite with high code coverage. The organization and complexity of the code-under-test present IntelliTest several barriers to this, and in a future post we will show how they may be overcome.

More code, please. 😛
Great feature. When will this be available for users?
Will this work for SharePoint SOM and CSOM code? Would love to see a post showing that.
Is there a HOL that I can download that shows how to use this for code examples?
@John, do note that SharePoint related code/dependencies will need to be mocked.
@Adivishnu, this is already shipping in VS2015. You can download the Enterprise RC bits here: http://www.visualstudio.com/…/visual-studio-2015-downloads-vs.aspx
@MichaelD, @Tad, will publish that soon.
Thank you for your interest. Keep the feedback coming.
How can I see a code base that uses this today so I can learn more.
@Danish, please see here: twitter.com/…/616667640048783361.
Hello Mr. Lasham,
I have been working with Intellitest for a few weeks now and I cannot seem to find some answers. I was wondering if you could tell me what the 'allow' option does. Let me give you some context. If I run intellitest in a function and it creates a unit test that fails due to some sort of exception (in this case a ThreadStateException), when I right click over the row in the "Intellitest Exploration Results" I see 2 options (4 total but 2 are greyed out) which are 'allow' and 'fix'. I understand the 'fix' option but I cannot seem to notice any difference with anything when I click on the 'allow' option. I was just wondering if you could explain this to me or point me to a more comprehensive intellitest instruction supplement.
Thank you very much!
Adam Ash
@Adam,
IntelliTest “Allows” you to triage exceptions into “expected” exceptions and “unexpected” exceptions.
Try the following:
(1) Run IntelliTest on your method, and save all of the generated tests. Note that a .cs (with the parameterized unit test) and a .g.cs file (with the generated unit tests) are emitted. See this post blogs.msdn.com/…/smart-unit-tests-test-to-code-binding-test-case-management.aspx for context.
(2) Keep both these files open side by side and do as follows:
(3) Select the row in the exploration results window corresponding to the (failing) test that raised the unexpected exception, and click on “Allow”.
(4) Notice that a [PexAllowedException(typeof(<xyz>))] is added to the parameterized unit test (the one with the [PexMethod] attribute).
(5) Now Run IntelliTest again (by clicking on the Run icon in the exploration results window).
(6) In the exploration results window, notice that this now shows up as a passing test.
(7) In the .g.cs file, notice that the old failing test case has been deleted and a new (passing) test case generated with the [ExpectedException(typeof(<xyz>))].
Expected exceptions generate negative test cases with the appropriate annotation (i.e. ExpectedException(typeof(<xyz>))), while unexpected exceptions generate failing unit test cases.
Hello Mr Lakshman,
Thank you so much for your response! I apologize for the egregious misspelling of your name in my first question. I appreciate the clarification and now it all makes sense to me! I am a software test engineer in charge of setting up automation and unit testing. I have been experimenting with intellitest for a few weeks. So far I am impressed with the power and potential this tool has to aiding me in creating salient unit tests. Our code base is huge and I am the only software test engineer on our team so this tool seems like it will be very useful. I am sure I will have more questions in the future. I appreciate your help!
Warm regards,
Adam Ash
Yo, you've got some bugs – github.com/…/621
This doesn't work period . Yes I'm in the correct edition of Visual Studio.
Yes I tried as simple class library
stackoverflow.com/…/how-do-i-enable-intellitest-in-visualstudio-2015-rc
"Run IntelliTest" option doesn't appear in the context menu when I right-click a method.
I've Visual Studio 2015 Enterprise.
What is wrong here?
@ChristianProgrammer, @Anil1986,
Can you please let me the project type on which you tried IntelliTest?
Me too. No sight of IntelliTest. I'm working on Windows 10 Universal App. There is an option called "Run Test" and "Debug Test", click on these, the project build, and then nothing happen
@Tuan,
IntelliTest is presently not supported for Universal Applications. We are tracking asks for it here: visualstudio.uservoice.com/…/6902587-enable-intellitest-to-generate-tests-for-universal. Please consider voting.
Hello Mr Lakshman,
How are you doing? I just wanted to ask another question with regards to the Automatically generated C# sharp Unit test files that have *.g.cs as an extension. All of my code is being created with Visual Studio 2015 Enterprise Edition.
I have made much progress since my last question which you were so helpful with. I am currently designing a Unit Test Automation /Build plan that requires me to work with the version control software Git. IT has come to my attention that files with a *.g.cs are not handled smoothly by Git when it comes to viewing diffs and merge conflicts. As these files are not to be touched by human engineers, it is not such a big issue. However I have read that storing these types of files in the repository is not recommended. My questions to you are as follows:
1) what is best practice for Unit Test versioning? Are the Unit Test Projects to be stored in the repository alongside the code? If so is it recommended that only the Unit Tests that are engineer modifiable be stored in the repository or should the Unit Test files that end in extension *.g.cs be stored in the repository as well?
From my perspective the *.g.cs files can stay in the repository as they are not to be touched by humans so there should not be any merge conflicts spawned from one of these files.
2) Basically what I have set up is all unit tests and test projects are stored in the code repository just like any other project and so far everything has gone smoothly. We just noticed that we can't view *.g.cs files from the git application and were concerned as to how this might effect the Testing Framework and wanted to know how this should be handled. We are not using Team Foundation Server. We are using Atlassian Stash as the repository host, Bamboo as the build server, and Git as the version control system. So some of these issues may have been addressed if we went with TFS.
I look forward to your response!
Warm regards,
Adam Ash
When comparing differences in vs 2015 releases I noticed that the intillitest is missing from vs pro and present only in vs enetrprize. Why is that ?
I have VS 2015 Enterprise, I can create "Hello World" type of application and IntelliTest works fine but for my product which is WPF application with 50+ libraries dependency, IntelliTest Run starts – IntelliTest Exploration Result shows pending and then stop. No idea what is going on, nothing what so ever appear in the project or result window. Should IntelliTest shows some error or log or error dialog why is didn't work.
If try with MVC website, I can see IntelliTest in context menu on controllers, but in Forms Web Site, neither in *.aspx.cs or classes in app_code get this IntelliTest.
Am I missing anything ?
Hi Mr Lakshman,
I’ve VS 2015, and I’m wondering if the IntelliTest could generate unit tests with statements, assertions, and all operations that can throw exceptions automatically. And how to do that. It’s bothered me for a long time. I’m looking forward to you reply!
Thank you so much!
Grace