Visual Studio 2005 – Extensibility Test Specifications


Test Specification for VS Extensibility


 


Have you ever wondered what we test on VSIP & Extensibility features in the VS QA team when we release Community Tech Previews?  We run a small number of automated tests on most of our features against each new build of VS & VSIP. If the results from tests are good, we declare the build usable, Self-host, for further testing/usage.  We call this subset of tests our Acceptance tests or Nightlies.  They basically verify build quality.  This is the same set of tests we run when we release Community Tech Previews.  We have many more automated and manual tests that are run on an ongoing basis during the product cycle.  We would like to share this set of our Acceptance tests with our customers and would like to hear your thoughts.  Is this too much testing, is it good test coverage, do you see a big hole in a most commonly used feature area?  Any other feedback you would like us to evaluate and incorporate?  Is this useful information if you also had the results of these included when we post the Community Tech Previews?


 


Extensibility consists of 2 major parts, features that ships in the VS Box(namely add-ins, macros, policy, object model) and features/tools that ships in VSIP.  Note, there are many other QA teams doing testing in other feature areas, what we present below is what is applicable to the Extensibility user community. 


 


VS


 



Add-Ins


Walk through the VS Add-in wizard for all lanugages supported, verify add-in builds


Verify add-in loads successfully


Verify unload of add-in from the Add-in manager works


Removing of .add-in file (new xml registration for add-ins) updates the add-in manager and does not show the add-in.


Verify the add-in manager respects the Tools options setting for add-in “Allow Add-in Components to load”


Verify add-in with wizard option to create a Tools menu item works.


Walk through the Shared Add-in wizard for all languages supported, verify add-in builds.


 


 


Macros


View the Macro Explorer, verify default Macro project created


Verify Can add a new module and a new macro to the MyMacros project, can run the macro


Verify creation of a new Macro project and unloading of the same.


Recording of a set of actions in the VS Editor, Stop recording, verify the code emitted,


Playback of macro above results in expected behavior


Record of new VB project creation, stop recording, verify code emitted.  Record a new macro to  unload the VB project.  Verify Temporary macro code previously emitted is overwritten.  Verify new temporary macro runs as expected.


Verify a new macro behaves like a VS menu command that can be invoked from the Command Window


Add a breakpoint to the macro in the Macros IDE, verify running the macro from Macro Explorer hits the breakpoint and then run to execution in debug mode


Macros with event handlers does disable events when user clicks on Disable event handlers.  Verify no event handlers from the Macro project are executed.


Macros with event handler, enabled for running event handlers does work correctly when a macro project with event handlers is loaded.  Verify events do fire.


Verify Tools options setting for Enabling/Disabling macros works as expected from the VS IDE macro explorer.


 



Policy


Verify a policy action file can be applied to different language projects


Verify can apply and remove policy action files to Solution folders and projects within the solution folder


Verify ElementSet Exclude works when System.Windows.Forms.dll reference is excluded. Apply this policy file to a Winform application, verify a warning is shown in the Errorlist window.  Apply the same policy file to a Console Application project, verify no warning is shown


Verify a policy file having Context links when applied shows the Contexts in the Dynamic Help window and Dynamic help window is updated when the policy file is removed.


verify can apply policy with reference for System.Windows.Forms.dll defined and excluded. Verify for a windows application a warning is shown in the error list,  verify with a console app you can add the reference and a warning shows up


Verify an errorneous policy file results in an error message and does not apply policy


Verify Property constraints, ToolboxItem constraints and Command constraints work


Verify adding and applying code type policy rules work for a Winform application, where policy file has code type for winform defined and excluded.  Verify for a winform application it shows a warning and for a console application it does not


Verify with solution folders and projects, Policy conflicts and propogation are handled correctly in a solution folder hieararchy


Enum constraints work correctly


Verify the correct warnings show up in the error list window


 


Object Model


·         Events


DTE startup/shutdown event fires


TaskList,Documents, Window events fire


VC Project and ProjectItem events fire


VB,C#,J# Project and ProjectItem events fire


Build events fire


 


·         Solution, Project & Build Configuration


On the Project object – Verify you can get to CofigurationManager, CodeModel, ProjectItems, Properties, Project.Object returns VSProject object.


On the ProjectItem object – verify you can get to ContainingProject, Document object, FileCodeModel, VSProjectItem, Properties, SubProject, Save


On the ProjectItems collection – you can get the ContainingProject, can iterate through the items and AddFromTemplate, AddFromFileCopy, AddFromFile, AddFromDirectory work


On the Solution Object – can get to the Projects, TemplatePath, AddFromTemplate, FindProjectItem, Close, Remove, Saved work


Solution Build object – can get the ActiveConfiguration, iterate through solution configurations, BuildDependencies, BuildState and Solution building works


VSProject object on VB.C#, J# projects – Verify we get can get to the BuildManager, VSProjectEvents, References, Imports and  DTE Project object, PublishManager and VSProjectEvents2


Can get to the VSProjectItem from the DTE’s ProjectItem object.


 


·         DTE, Tasklist, Outputwindow, Document, Text Editor, Toolbox, Window


Verify objects returned with DTE – ActiveWindow, ActiveDocument, ExecuteCommand


Get the tasklist object, iterate taskitems, can get to the taskitem description and add a new taskitem, set the priority on the new taskitem


Get the outputwindow object, add a new pane, can get to the TextDocument object on a outputwindowpane, add an item to the errorlist window.


Can get to the Document object, can iterate through documents collection, can get to the TextDocument object, selection returned is correct


create each object in the text editor object model and verifies creation – TextPoint, TextRange, TextSelection, VirtualPoint, EditPoint, TextWidnow, TextPane, UndoContext, HTMLWindow


Can iterate through Windows collection, can get to Document, Project, Projectitem objects on a Window object, returns the toolwindow object where supported


Get the Toolbox object, add a new toolboxtabs, new toolboxitem, can delete toolboxtab


 


·         Globals


Verify can get to DTE globals, can add a new global, verify name-value pair returned.


 


·         Code Model


Test CodeNameSpace methods and properties on C++, C#,  Projects – AddClass, AddDelegate,AddEnum,AddStruct,AddInterface,AddNameSpace, DocComment


Test FileCodemodel & FileCodeModel2 methods and properties for VB project – AddClass, AddNamespace, AddStruct, AddEnum, AddVariable, AddFunction


 


 


 


VSIP


 


Setup:


Verify VSIP SDK can be setup successfully after installing VS. 


      Verify the Experimental hive is correctly setup after installing VSIP


 


 


Interop Assemblies


A Primary Interop Assembly is a unique assembly that contains type definitions (as metadata) of types implemented with COM. VSIP Primary Interop Assemblies refer to the managed definitions for the Visual Studio Integration Partner (VSIP) interfaces that clients can use to write packages using managed code (C#, VB.NET and Managed C++) targeting Whidbey IDE.


 


·         Type Verification. Verify number of types and signatures.


Where a type can be an interface, struct or enum. The comparisons are first made between the source (a type library) and the corresponding target (an interop assembly) and then they are compared against last known good interop assembly. The second comparison is a conservative test to track, any changes or new types added, to the IA. There are some types in type libraries that are purposely omitted from the IAs.


 


PIA nightly test verifies following scenario for each IA that shipped in VSIP Everett Extra’s (as we call them baseline IAs)  plus the new IAs that are part of VSWhidbey VSIP


1. interface number in IA matches the number in the corresponding type library.


2. enum number in IA matches the number in the corresponding type library.


3. struct number in IA matches the number in the corresponding type library.


4. interface signatures in IA matches the signature in the corresponding type library.


 


Besides, test stores type information of the last known good  (LKG) IA build and verifies 4 scenarios above against LKG IA. Any unexpected change from LKG is logged as a bug.


 


·         Round-trip testing between Managed and Unmanaged code


Calling Unmanaged code from a Managed package and vice-versa for 4-5 different Interfaces


 


Wizards


·         Package Wizard:  For both the languages supported for the package wizard (C# and C++):


Create new Integration package with ToolWindow, MenuItem and Editor options checked.


Verify that the project gets built without errors


Start Exp shell and verify package loads


 


·         Language Wizard:  Test that the wizard operates correctly on the default settings (using MyC) with symbols loaded from the default lexer & parser files


Verify that service.cpp & service.h were created


Verify that the g_languageProperties table was generated according to the specifications given in the wizard


Verify that the tokenInfoTable was generated according to the specifications given in the wizard


Verify that the lexer & parser files were added to the project


Verify that the project compiles


Verify that the language exists in the registry & has the correct settings in all registry keys


Verify that the default features work in the Exp environment


 


Samples  For each Managed and Unmanaged sample:


Verify that the sample builds without errors.


Verify that the sample registers without errors.


Verify that the Environment loads the sample without errors under both the product and experimental hive.


Verify some very basic functionality of each sample.


 


Managed Package Framework


Verify correct operation of RegPkg on a Whidbey package


Verified basic functionality of the OleMenuCommandService class


Verify OleMenuCommand class


Verify the GetService method in the Package class


Verify that all attributes are registering properly with RegPkg


Verify that the Package class functionality is available to derived packages


 


Package Loading:  This is Visual Studio’s package load behavior testing with various switches.


Verify that internal package loads correctly while in safemode


verify that the external package is not skipped after having failed to load and being selected not to skip.


verify that the external package is skipped after having failed to load and being selected to skip.


verify that one previously skipped package is reset when using the /resetskippkg flag


Verify that /novsip causes invalid PLK packages to fail silently to load


Verify that /novsip does not cause an internal package to fail to load


Verify that rerunning devenv w/o the /novsip flag puts the invalid PLK pkgs back the way they were


Verify that /novsip does not cause a package with a valid PLK to fail to load


 


 


You can send us feedback at vsextend@microsoft.com


 


 

Comments (2)

  1. chan says:

    What about Babel, what kind of tests you applied on Babel?

  2. Aaron Marten says:

    Hi Chan,

    Remember that these are only nightly level tests. We are getting testing on Babel through several means. Firstly, on a nightly level, the output from the language service wizard is written to use Babel. Secondly, we are verifying that the LService.sln solution is compiling and registering properly.

    On a non-nightly level, we are doing more complete coverage on Babel using samples from several languages that we have developed.

    If there is a bug that has been bothering you with Babel that you’d like to see fixed, please report it to us at the MSDN Product Feedback center. (http://lab.msdn.microsoft.com/productfeedback/)

    Thanks!

    Aaron Marten