Assessing extension reliability and safety

Harysh Menon

With 3^rd party extensions now being available for Team Foundation Server as well on the Marketplace, there have been a number of queries around evaluating extension reliability and safety. With this post we aim to provide a general set of guidelines for users, as well as publishers, regarding plugin safety and reliability. We also want to get your feedback on some of the changes we are bringing in this space.

We currently verify all publishers before they can publish an extension publicly for VSTS/Team Foundation Server. We do so by looking at the publisher site, GitHub Repo and associated links. In addition, all our extensions are sandboxed and, as part of installation, we also show you all the permissions required by the extension. Following are some of the ways we can look at extension, reliability and safety.

As a user

As a user one of the primary ways you can judge the quality/safety of an extension is by relying on the community. We have recently introduced the Ratings and Reviews feature for Marketplace which can be used to provide feedback for an extension to the publisher and also to inform other users regarding the extension. We will also be introducing additional features to Rating and Review, like publisher responses, marking a review as helpful and sorting reviews based on rating to further enhance the feature and act as a mechanism to judge the quality of an extension.

In addition, other factors that can help in deciding if an extension is reliable are links to support pages, documentation and screenshots of the extension in use. Information on whether an extension is using an external service in order to function or if it relies on a companion service backend can also prove to be useful when judging whether to include the extension in On Premises Instances.

As a publisher

From a publisher perspective having a detailed extension description page that includes the information described above can help users make the choice on whether to install the extension. Also, many of the extensions submitted to the marketplace are open source. In such cases the following information presented as a part of the extensions description can prove useful:

• Links to public repositories
• Build Status
• Standard Static Analysis tool status (SonarQube, Checkmarx, Fortify)
• Links to StackOverflow Profiles

These indicators within your extensions’ detailed description can help users evaluate how much work is being done within the extension and also show the kind of testing done for an extension.

Future Improvements

In addition to enhancements in Rating and Review we are also planning on changes to the extension manifest and item details pages. These changes are aimed at standardizing the look and feel of the extension details page so that users can easily find many of the details mentioned above when browsing different extensions.

Some of the enhancements being planned are:

• Publisher Profile improvements: View more details about a publisher and see other extensions by the same publisher
• Release Notes: View what changes are brought in different versions of the extension
• Standard Indicator Support in manifest: Ability to provide GitHub repository link, Build status link, Static analysis tool links within the manifest to display the same as part of Item description
• Publisher Certifications: Indicators to callout different types of publishers, like Microsoft Most Valuable Professional (MVP), Visual Studio Partner program members, Application Lifecycle Management Rangers and other Microsoft affiliated publishers
• Data Storage Information: Require publishers to indicate if data is being stored outside of the Visual Studio Team Service account and the region of the data store
• Extension Install trends: Number of installs for the extension in the past day/week/month to judge the popularity of an extension

We would love to hear your thoughts on what more we can do or if there are any other enhancements that you would like to see. Do reach out to us at VSMarketplace@microsoft.com.

Harysh Menon (@haryshm) Program Manager Visual Studio Marketplace

Harysh Menon

Follow