Hello,
My name is Rashid Sarwar and I am on Visual C++ Compiler Front End team. I have been on this team for several months and have learned many things on how to test a complier. In this post I will be sharing with you how we test our Visual C++ compiler and it’s Intellisense.
Testing features in a compiler is not easy as each feature/construct can interact with the already existing features/constructs in many different ways. Coming up with all the possible cases in which the new construct can be used is a big task as the Visual C++ compiler has many features, starting from standard C++ language features to our own Microsoft extensions to compiler. Every time when we have to add a new feature to our compiler like the new C++0x features coming in Visual Studio 2010, we try to come up with the good number of possible cases in which the feature can be used as a standalone feature .We first try to exhaust the new grammar of the newly added language constructs and try to make sure that we are able to compile the newly added language constructs as a standalone construct.
For example:
Lambda expressions in new C++0x can be used in many different ways. A lambda can have an empty capture clause.
A simple compiler test to test empty capture clause would be something like this in which we will come up a source code related to empty lambda capture clause. The automated test will take these sources and compile them with Visual C++ compiler. If let’s say any compilation on a valid usage of lambdas on empty capture clause fails than it is an indication that something is there which either compiler currently not supports or is broken.
//Source1: testing empty lambda capture clause
int main()
{
bool even = [](int k){ return k%2 == 0; }(2);
return 0;
}
The above test verifies that we are able to compile correctly the empty capture clause of lambda expressions. But how can we verify that we are generating the correct compiled output? How can we verify that code generated by compiler is correct? We take a further step in the test by looking at the output of the lambda function object. We do run time verification by changing the test in following manner:
//Source2 – testing empty lambda capture clause
int main()
{
bool even = [](int k){ return k%2 == 0; }(2);
if(even)
{
return 0;
}
else
{
return 1;
}
}
Our testing automation framework will look for the return value of the main method. If the return value from main method is 0 than the test is assumed to pass and assumed that we are generating correct code from compilation. If the return value is non zero than the testing framework assumes that the test failed as the result of main method can only be non zero if let’s say we generated wrong machine code.
This is very simple source code to test a big feature like lambda expressions which can be used in many different ways. So we then try to test each feature from the perspective of how it can be used in combination with other different features. Example can be like testing lambda expressions in a class member functions.
Now, I will discuss testing Intellisense for Visual C++. For Intellisense testing we divide our automated tests into two categories. Engine/Component level tests and End to End User scenario tests.
In the End to End User scenario test our automated test will try to simulate user actions in IDE example test can be like:
1. Creating a new Visual Studio instance.
2. Creating a new Visual C++ project.
3. Adding a new cpp file in the project and set the cursor focus in the new file.
4. Writing some C++ code in the opened file in the editor. Simple code can be like:
class person{
public :
char * name();
};
int main()
{
person p;
return 0;
}
5. Trying to invoke some intellisense operations like QuickInfo on the code in the file by hovering the mouse cursor over the newly created instance of class person in the main method.
6. Finally verifying the output of tooltip with the expected output the test expects.
7. If tooltip output differs from the test expected output than QuickInfo operation is assumed to not working and thus making test fail. The test passes if both expected and actual outputs by hovering mouse cursor over the construct match.
The EngineComponent level tests directly talk to Visual Studio to invoke the intellisense operations on the code constructs. We let the test know that at which positions in the code it has to invoke the intellisense positions.
For example on sample code:
class person{
public :
char * name();
};
int main()
{
person p;
return 0;
}
Simple test scenario can be like testing QI functionality on variables of user defined types. We will let the test know to invoke QI operation on the position where class person p instance is declared in main function. We will define in the test the (row/col) position and the test when it runs it will look for the specified QI operation on the specified position. It will get the result of tooltip that appears and match it with the expected output.
Our Engine level tests are easy to write and they have less execution time, as the test itself does not need to do all the extra work. But they don’t exactly mimic user scenarios. We write IDE tests to get better coverage of user end to end scenarios .The drawback of IDE tests is that they take a long time to execute. So we try to strike a balance in engine level and end to end scenario testing when we have to test a new or existing feature in our Visual C++ intellisense.
Hopefully this post will give you an idea of how we test our Visual C++ compiler and also its intellisense.
how do you test intellisense performance?
Hi, Could you guys please explain the Rich header generated between DOS & PE header?
yeah, Intellisense performance
And, resource leak in vs…
Hi, this appears to be a bug (VS2005 SP1), it prints garbage.
class String
{
public:
char string[8192]; //keep array less than 4096 for correct results
String(char *buffer="")
{
strcpy(string,buffer);
}
};
void func(String str1,_bstr_t str2="")
{
printf(str1.string);
}
int _tmain(int argc, _TCHAR* argv[])
{
func("asd","dfgdfs");
return 0;
}
You should be glad that your code even compiles. Run cl at /W4 and see what I mean.
You are converting const char* to char* through that user defined conversion which is a non-standard extension. No wonder it prints garbage.
Secondly, you should never omit a format string in one of (print/scan)f functions.
Third, use cout and cin.
Fourth, redefine func as void func(const String& str1,_bstr_t str2="").
Fifth, add cctor, dtor and assignment operator to that class.
//compile with /CLR
class A
{
public:
A(int *i)
{
printf("this: %xn",this);
}
};
void func(A a)
{
printf("func: %x n",&a);
}
int _tmain(int argc, _TCHAR* argv[])
{
func(0);
return 0;
}
why do I get different results printed out from my constructor and from ‘func’, (compile using clr, without clr, it is correct).
//compile with /CLR
class A
{
public:
A(int *i)
{
printf("this: %xn",this);
}
};
void func(A a)
{
printf("func: %x n",&a);
}
int _tmain(int argc, _TCHAR* argv[])
{
func(0);
return 0;
}
Why do i get seperate results from the constructor and ‘func’ (without /clr, I get same values)
//compile with /CLR
class A
{
public:
A(int *i)
{
printf("this: %xn",this);
}
};
void func(A a)
{
printf("func: %x n",&a);
}
int _tmain(int argc, _TCHAR* argv[])
{
func(0);
return 0;
}
Why do i get seperate results from the constructor and ‘func’ (without /clr, I get same values)
//Correction
//compile with /CLR
class A
{
public:
A(int i)
{
printf("this: %xn",this);
}
};
void func(A a)
{
printf("func: %x n",&a);
}
int _tmain(int argc, _TCHAR* argv[])
{
func(0);
return 0;
}
The code works fine with my install of VS 2005 SP1. Several of Badar’s comments are silly:
<i>Secondly, you should never omit a format string in one of (print/scan)f functions.</i>
What does that mean? The string is the format string.
<i>Third, use cout and cin.</i>
Bullocks. cout and cin are bloated functions that don’t have the control and simplicity printf does.
<i>Fifth, add cctor, dtor and assignment operator to that class.</i>
What are you some sort of academic? If you don’t need that crap, why add it? In this case, the compiler does it for you and that’s perfectly fine.
Yes, Asif should use const correctly and guard against buffer overruns, but I figured he was simplifying the code to show a compiler error. (He should also be asking these questions elsewhere and needs to brush up on what the /CLR switch does.)
Asif,
I compiled and executed your first code in VS2005SP1 pro(v50727.762) in both release and debug config and using both unicode and multibyte for each of the configs. The result was same for all configurations: "asd" that I believe is correct result.
Asif,
Your second example is interesting. When I compile it without /clr I get same result for constructor and ‘func’ but with /clr I get different results too. Seems some magic happens there.
Anybody any explanation?
The only way i’m going to be impressed by Intellisense is when it works properly with Boost.
> You are converting const char* to char* through that user defined conversion which is a non-standard extension.
It is a perfectly standard conversion, just a deprecated one. ISO/IEC 14882:2003, 4.2[conv.array]/2:
"A string literal (2.13.4) that is not a wide string literal can be converted to an rvalue of type “pointer to char”; a wide string literal can be converted to an rvalue of type “pointer to wchar_t”. In either case, the result is a pointer to the first element of the array. This conversion is considered only when there is an explicit appropriate pointer target type, and not when there is a general need to convert from an lvalue to an rvalue. [Note: this conversion is deprecated. See Annex D. ]"
vote+1 on boost.
Save yourself the trouble load your test engine up and parse the boost (you have the code already). If it cannot fetch the names and types, you have a bug.
Thanks a lot to everbody for putting down some interesting questions and also some interesting piece of code to look at.I will try to answer all of them.
Jk & yuguang : Intellisense performance is really a good question.We have a lot of test specifically for Intellisense perfromance testing. In our performance tests we try to simulate user actions as i described earlier in the post.We try to perform some operation that is to be measured for some defined no of times continuously in an iteration and get the average amount of time for that operation. That is the most simple explanation. The peformance test try to take into account that they are measuring the time of only the opeartion that is to be measured.Some tests measure time in isolation of all other scenarios and some measure in which a normal user would have invoked the operation.
For testing memory leaks in VS we normally test our products both retail and debug builds .If there are any memory leaks they appear up in form of assertions in case of running our tests on debug build.We aks
Joel : Thanks for pointing out Boost.We do provide intellisense for Boost library and we have tests that use Boost. We have done intellisense testing against Boost for our Intellisense parser and I hope our customers will have good experience with Intellisense in Boost library.Thanks a lot for your interest in Boost.
Majkara : Nice idea oftesting the Boost Library.Thanks for it.
Asif: Thanks for putting some interesting stuff here.I will try to explain the behaviour of the second question u had.
Why do i get seperate results from the constructor and ‘func’ (without /clr, I get same values)
When the code is compiled as native, the compiler does not creates a copy instance first. It constructed an A directly as a parameter to ‘func’ and this way just one instance gets created so you see both addresses same in case of native.
When compiled with /clr, it first creates a temporary A instance (at one address), invoked the trivial copy constructor and then copy it as an argument to func .This way func prints the address of this second object.The compiler can choose to construct an A instance directly as an argument to func, or it can construct a temporary, then copy it as an argument.
I am looking forward to some more interesting questions.Do let me know if you guys have any regarding the Intellisense and our VC++ compiler.
http://social.msdn.microsoft.com/Forums/en-US/vclanguage/thread/3ba4afd7-9109-4426-8dc9-8b16374cc5b5
go through this discussion.
Hi, I stuck because of this bug in the VC++ compiler (or so it seems). This code is simpilfied for the sake of this discussion, and I cannot implement copy constructor etc on classes/structs that do not belong to me…
What happened with the Hotfix for the "VC9 SP1 Hotfix For The vector<function<FT>> Crash" Hotfix?
http://blogs.msdn.com/vcblog/archive/2008/12/17/vc9-sp1-hotfix-for-the-vector-function-ft-crash.aspx
Hi Anders,
We are currently working on the Hotfix for the "VC9 SP1 Hotfix For The vector<function<FT>> Crash" and it should be posted shortly. We will also update the original blog with the new link.
Can vs add a menu or button which is "Rebuild intelisense" so that I can manually full rebuild it? Even it needs 600 seconds would be helpful.
Another question:
Why when I open multiple VC IDE(3 or more), and use for a long time(more than 1 day),the CTRL+F show the dialog very slow? If I close all VC IDE and reopen, everything seem OK.
I really doubt VS has GDI leak or handle leak.
This is a bug I found (and reported) in VC 6.0 and is still in 2003. I was hoping it would be gone im my next generation of compiler>
#include <ctype.h>
#include <math.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <stddef.h>
static double dmx1,dmx2;
#define NRC_DMAX(a,b) (dmx1=(a),dmx2=(b),((dmx1)>(dmx2))?(dmx1):(dmx2))
/* ————————————————————————–*/
int main( void );
/* ————————————————————————–*/
/* ————————————————————————–*/
int main( void )
/* ————————————————————————–*/
{
double test1;
double test2;
double test3;
test1 = NRC_DMAX(7.,NRC_DMAX(4.,3.));
test2 = NRC_DMAX(NRC_DMAX(4.,3.),7.);
test3 = NRC_DMAX(NRC_DMAX(4.,7.),3.);
printf("n NRC_DMAX(7.,NRC_DMAX(4.,3.)) = %le n", test1 );
printf("n NRC_DMAX(NRC_DMAX(4.,3.),7.) = %le n", test2 );
printf("n NRC_DMAX(NRC_DMAX(4.,7.),3.) = %le n", test3 );
return 0;
}
Test1 , -2 ,-3 should be the same but are not in the versions I have tested.
As someone said. Please give users some control over intellisense.
eg some settings like
"Run Intellisense in background", "Don’t run Intellisense in background, just do it when I tell you – even if it takes 5mins", "Kill Intellisense, its eating my memory"
Intellisense certainly shouldn’t be run during a compile!
We’ve managed to mitigate some of the probs by not letting it write to disk.
I am curious about code style Listing 2, where there is so much verbosity! I hope this is only seen here and is not in compiler code :)
if (even)
return 0;
else
return 1;
== return !even;
Unfortunately this intellisense bug https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=339575 still happens in VS 2008 and is a show stopper for us. :( I hate it that we can’t reopen bugs in connect