Testing the IntelliSense engine

Hi, I am James Wang, an SDET on the VC++ Compiler Front End team. Currently, I am working on designing the test architecture for the new IntelliSense engine. I am responsible to design tests that make sure the IntelliSense engine gives correct answer for quick info, member list, parameter help, and etc.

Currently we have a test suite that directs testing the IntelliSense feature. But one of the drawbacks is that it only targets a very limited amount of scenarios. Testing converge becomes a major issue. But due to vast varieties of C/C++ constructs, testing every scenario becomes an impossible or at least a very time consuming task. In the meantime, we do have ~100,000 of tests that targeting compiler / libraries that covers a majority of C/C++ constructs. Would it be better to adapt these tests for the purpose of IntelliSense engine testing?

To accomplish this, I wrote a tool that will analysis any arbitrary C/C++ code and generate IntelliSense test site. The tool will also generate the expected results for each test site. So for any arbitrary C/C++ code, we can generate a set of self verifying IntelliSense tests on quick info, member list, parameter help, and etc. As the result, we can reuse the ~100,000 tests for the purpose of IntelliSense testing.

Two things make this possible:

1. Better intermediate language (IL) representation. After parsing the source code, the new IntelliSense engine generate a better IL and expose a wealth of APIs that allows traversing and querying various compiler artifacts. This allows me to figure out where to create IntelliSense test site and what the expected results should be. For example: while traversing through the IL, if I see a field operator (->), I may want to generate a member list tests. For the expected results, I can query the type for the left operand of the field operator and retrieved the expected member list.

2. Componentized the IntelliSense engine. The new IntelliSense engine exposes API that allows engine level testing. So I can do IntelliSense testing without the IDE. Otherwise, running a large amount of IntelliSense tests become impractical due to the time needed to start / shut down the IDE for each tests.

Hopefully, this new approach will improve the test coverage for the new IntelliSense engine and drive up its quality.

Thanks,

James