Testing VC++ Compiler and Intellisense

Hello,

My name is Rashid Sarwar and I am on Visual C++ Compiler Front End team. I have been on this team for several months and have learned many things on how to test a complier. In this post I will be sharing with you how we test our Visual C++ compiler and it’s Intellisense.

Testing features in a compiler is not easy as each feature/construct can interact with the already existing features/constructs in many different ways. Coming up with all the possible cases in which the new construct can be used is a big task as the Visual C++ compiler has many features, starting from standard C++ language features to our own Microsoft extensions to compiler. Every time when we have to add a new feature to our compiler like the new C++0x features coming in Visual Studio 2010, we try to come up with the good number of possible cases in which the feature can be used as a standalone feature .We first try to exhaust the new grammar of the newly added language constructs and try to make sure that we are able to compile the newly added language constructs as a standalone construct.

For example:

Lambda expressions in new C++0x can be used in many different ways. A lambda can have an empty capture clause.

A simple compiler test to test empty capture clause would be something like this in which we will come up a source code related to empty lambda capture clause. The automated test will take these sources and compile them with Visual C++ compiler. If let’s say any compilation on a valid usage of lambdas on empty capture clause fails than it is an indication that something is there which either compiler currently not supports or is broken.

 

//Source1: testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 return 0;

}

 

The above test verifies that we are able to compile correctly the empty capture clause of lambda expressions. But how can we verify that we are generating the correct compiled output? How can we verify that code generated by compiler is correct? We take a further step in the test by looking at the output of the lambda function object. We do run time verification by changing the test in following manner:

//Source2 – testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 if(even)

 {

   return 0;

 }

 else

  {

   return 1;

 }

}

Our testing automation framework will look for the return value of the main method. If the return value from main method is 0 than the test is assumed to pass and assumed that we are generating correct code from compilation. If the return value is non zero than the testing framework assumes that the test failed as the result of main method can only be non zero if let’s say we generated wrong machine code.

This is very simple source code to test a big feature like lambda expressions which can be used in many different ways. So we then try to test each feature from the perspective of how it can be used in combination with other different features. Example can be like testing lambda expressions in a class member functions.

Now, I will discuss testing Intellisense for Visual C++. For Intellisense testing we divide our automated tests into two categories. Engine/Component level tests and End to End User scenario tests.

In the End to End User scenario test our automated test will try to simulate user actions in IDE example test can be like:

1.       Creating a new Visual Studio instance.

2.       Creating a new Visual C++ project.

3.       Adding a new cpp file in the project and set the cursor focus in the new file.

4.       Writing some C++ code in the opened file in the editor.  Simple code can be like:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

5.       Trying to invoke some intellisense operations like QuickInfo on the code in the file by hovering the mouse cursor over the newly created instance of class person in the main method.

6.       Finally verifying the output of tooltip with the expected output the test expects.

7.       If tooltip output differs from the test expected output than QuickInfo operation is assumed to not working and thus making test fail. The test passes if both expected and actual outputs by hovering mouse cursor over the construct match.

 

The EngineComponent level tests directly talk to Visual Studio to invoke the intellisense operations on the code constructs. We let the test know that at which positions in the code it has to invoke the intellisense positions.

For example on sample code:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

Simple test scenario can be like testing QI functionality on variables of user defined types. We will let the test know to invoke QI operation on the position where class person p instance is declared in main function. We will define in the test the (row/col) position and the test when it runs it will look for the specified QI operation on the specified position. It will get the result of tooltip that appears and match it with the expected output.

Our Engine level tests are easy to write and they have less execution time, as the test itself does not need to do all the extra work. But they don’t exactly mimic user scenarios. We write IDE tests to get better coverage of user end to end scenarios .The drawback of IDE tests is that they take a long time to execute. So we try to strike a balance in engine level and end to end scenario testing when we have to test a new or existing feature in our Visual C++ intellisense.

Hopefully this post will give you an idea of how we test our Visual C++ compiler and also its intellisense.