Expression SFINAE improvements in VS 2017 RC

This post written by Xiang Fan, Andrew Pardoe, and Gabriel Dos Reis

This post was updated to reflect progress we’ve made through VS 2017 RC since VS 2015 Update 3.

Throughout the VS 2015 cycle we’ve been focusing on the quality of our expression SFINAE implementation. Because expression SFINAE issues can be subtle and complex we’ve been using popular libraries such as Boost and Microsoft’s fork of Range-v3 to validate our implementation and find remaining bugs. As we shift the compiler team’s focus to Visual Studio 2017 release we’re excited to tell you about the improvements we’ve made in correctly parsing expression SFINAE.

With VS 2015 Update 3 we told you that Boost compiles correctly with MSVC without defining the macro BOOST_NO_SFINAE_EXPR. A few libraries in Boost still didn’t compile and Range-v3 support was incomplete. With Visual Studio 2017 Release Candidate Visual C++ can now compile Boost Spirit X3, Sprout, and Range-v3. We’ve made fantastic progress so far and will soon have a complete and correct implementation of expression SFINAE.

Our next focus is Boost Hana where over 70% of the tests pass so far with some source workarounds and some compiler fixes.We’re at the point where many of the bugs that we’re discovering aren’t actually expression SFINAE bugs. We’ve uncovered issues with constexpr, generic lambdas, pack expansions, variadic macros, special member functions, parsing problems, and other issues. Some of these issues look like expression SFINAE issues but turn out to be SFINAE dependencies on other areas. We’ve fixed about 20 issues, three of which were SFINAE issues, and have about 35 left to fix in Hana. We look forward to Hana compiling cleanly without workarounds during the VS 2017 cycle.

What is expression SFINAE?

SFINAE is an acronym for “substitution failure is not an error.” It is derived from an arcane process used by C++ compilers during overload resolution.  At its core, the idea is quite simple: if a candidate function template’s specialization would lead to an ill-formed (compiler-internal) declaration of that specialization, just silently ignore that template as is if the compiler has never seen it. In another words, the compiler will pretend that wasn’t the template it was looking for.  It is an old notion that has been part of C++ since it’s the C++98 release. In that version of C++, the condition for “ill-formed compiler-internal declaration of a function template specialization” was specified for types only.

With the advent of type query features such as decltype and auto, the validity of a function type now entails the validity of expressions, since whether decltype(expr) is a valid type depends on whether the operand expr is well-formed.  Hence the term “Expression SFINAE”.  It is a little bit more involved because now the compiler has to perform overload resolution including potentially unbounded evaluation of constant expressions while it is checking whether a type makes sense.

Improvements since Visual Studio 2015 Update 3

We now correctly compile code that constructs temporary objects as Range-v3 does extensively:

		#include <type_traits>
		
		template<typename T, std::enable_if_t<std::is_integral<T>{}> * = nullptr>
		char f(T *);
		
		template<typename T>
		short f(...);
		
		int main()
		{
			static_assert(sizeof(f<int>(nullptr)) == sizeof(char), "fail");
			static_assert(sizeof(f<int *>(nullptr)) == sizeof(short), "fail");
		}

We’ve also improved access checks for SFINAE which are illustrated in this code sample:

		template <typename T> class S {
		private:
			typedef T type;
		};
		
		template <typename T> class S<T *> {
		public:
			typedef T type;
		};
		
		template <typename T, typename S<T>::type * = nullptr>
		char f(T);
		
		template<typename T>
		short f(...);
		
		int main()
		{
			static_assert(sizeof(f<int>(0)) == 2, "fail"); // fails in VS2015
			static_assert(sizeof(f<int *>(nullptr)) == 1, "fail");
		}

Lastly, we’ve improved support for void_t when used inside of a typename as found in Boost Hana:

		template<typename T, typename U>
		struct std_common_type {};
		
		template<typename T>
		struct std_common_type<T, T> { using type = T; };
		
		template<typename T, typename U>
		struct is_same { static const bool value = false; };
		
		template<typename T>
		struct is_same<T, T> { static const bool value = true; };
		
		template<bool, typename T>
		struct enable_if {};
		
		template<typename T>
		struct enable_if<true, T> { using type = T; };
		
		template<typename...> using void_t = void;
		
		template <typename T, typename U = T, typename = void>
		struct EqualityComparable1 { static const bool value = false; };
		
		template <typename T, typename U>
		struct EqualityComparable1<T, U, typename enable_if<!is_same<T, U>::value, void_t<typename std_common_type<T, U>::type>>::type>
		{
			static const bool value = true;
		};
		
		template <typename T, typename U = T, typename = void>
		struct EqualityComparable2 { static const bool value = false; };
		
		template <typename T, typename U>
		struct EqualityComparable2<T, U, void_t<typename std_common_type<T, U>::type>>
		{
			static const bool value = true;
		};
		
		void f()
		{
			struct S1 {};
			struct S2 {};
			static_assert(!EqualityComparable1<S1, S2>::value, "fail"); // fails in VS2015
			static_assert(!EqualityComparable2<S1, S2>::value, "fail");
		}

Improvements since Visual Studio 2015 Update 2

Continued improvements in the quality of our expression SFINAE implementation enabled our Standard Template Library to begin using it in VS 2015 Update 2. Expression SFINAE is used in our implementations of std::function and result_of.

Improvements since Visual Studio 2015 Update 1

Because we’re now generating parse trees for decltype expressions a number of patterns work correctly in Update 3.

  • We’ve implemented checking for dependent expression using the new parse tree in the compiler. That fixes this Connect issue reported for a failure compiling Chromium.
  • We’ve implemented ability to distinguish different expressions inside decltype using parse tree. Here’s an example simplified from the Boost thread library:
    template<class T>
    struct remove_reference
    {
        typedef T type;
    };
    
    template<class T>
    inline T&& forward(typename remove_reference<T>::type& t)
    {
    	return static_cast<T&&>(t);
    }
    
    template<class T> 
    inline T&& forward(typename remove_reference<T>::type&& t)
    {
    	return static_cast<T&&>(t);
    }
    
    template <class Fp, class A0, class ...Args>
    inline auto invoke(Fp && f, A0 && a0, Args && ...args)
    -> decltype((forward<A0>(a0).*f)(forward<Args>(args)...))
    {
    	return (forward<A0>(a0).*f)(forward<Args>(args)...);
    }
    
    template <class Fp, class A0, class ...Args>
    inline auto invoke(Fp && f, A0 && a0, Args && ...args)
    -> decltype(((*forward<A0>(a0)).*f)(forward<Args>(args)...))
    {
    	return ((*forward(a0)).*f)(forward(args)...);
    }
    
  • A couple of test cases simplified from Range-v3 now work.
    int f(int *);
    		
    namespace N {
    	template<typename T> T val();
    
    	template<typename T> using void_t = void;
    		
    	template<typename T, typename = void> struct trait {};
    	template<typename T> struct trait<T, void_t<decltype(f(val<T>()))>> {
    		typedef decltype(f(val<T>())) type;
    	};
    }
    		
    N::trait<int *>::type t1;
    		
    struct S {
    	template<typename T> static T val();
    
    	template<typename T> using void_t = void;
    
    	template<typename T, typename = void> struct trait {};
    	template<typename T> struct trait<T, void_t<decltype(f(val<T>()))>> {
    		typedef decltype(f(val<T>())) type;
    	};
    };
    		
    S::trait<int *>::type t2;
    
  • Also, this example:
    int g;
    		
    template<typename T>
    using void_t = void;
    		
    template<typename T, typename = void>
    struct S1 {};
    		
    template<typename T>
    struct S1<T, void_t<decltype(g + T{}) >> {};
    		
    struct S2 {
    	int *g;
    	auto f() -> decltype(S1<int>());
    };
    

Moving away from the token stream parser

A lot of the improvements you’re seeing in expression SFINAE support and other areas comes from work we’re doing to rejuvenate our old compiler. The Visual C++ compiler has been around for over thirty years–long before C++ had templates. This means that we’re now working around design decisions that once made sense.

Visual C++ traditionally took a token stream-based approach to parsing templates. When we encounter a template in your code we capture its body as a sequence of tokens without any attempt to understand what the tokens mean. Storing the body as a stream of tokens makes analysis of trailing return types containing decltype-specifiers imperfect, especially in SFINAE contexts.

We have now implemented a recursive-descent parser that generates high level unbound trees for expressions and employed this to analyze the expression argument of decltype in a much more precise way, allowing a better implementation of expression SFINAE. The recursive descent parser is a work in progress; currently, it can parse only C++ expressions but we’re going to soon expand it to parse the entire C++ syntax and make it the basis for implementing features such as two-phase name lookup. These features have been almost impossible to implement with the token stream-based parser. As work proceeds, the remaining gaps in expression SFINAE will also be filled.

If you’d like to read more about the changes we’re making to the parser you can find more in this blog post: Rejuvenating the Microsoft C/C++ Compiler.

Known issues as of VS 2017 Release Candidate

You may encounter the following known issues when using expression SFINAE in the Visual C++ compiler as of VS 2017 Release Candidate.

  • A couple of issues impact input to SFINAE:
    • Some uses of constexpr lead to incorrect specializations. The parser does semantic analysis aggressively even when the template argument is dependent. Thus it will try to specialize f(T{}) in the below example and will fail. This leaves a wrong specialization with a nullptr expression (or a dummy expression) as the template non-type argument. Any further usage of the template will fail.The new parser only does semantic analysis on non-dependent expressions. We are progressively moving the parsing of template arguments to the new parser.Compiling this code:
      		
      		template<bool> struct S {};
      		
      		template<typename T> constexpr bool f(T) { return true; }
      		
      		template<typename T> void g(S<f(T{})>) {}
      		template<typename T> void g(S<f(T{1})>) {}
      

      Currently produces this error message:

      error C2995: 'void g(S)': function template has already been defined

      One possible workaround is to use a variable template:

      		template<bool> struct S {};
      		
      		template<typename T> constexpr bool f(T) { return true; }
      		
      		template<typename T> constexpr auto g_value1 = f(T{});
      		template<typename T> constexpr auto g_value2 = f(T{1});
      		
      		template<typename T> void g(S<g_value1<T>>) {}
      		template<typename T> void g(S<g_value2<T>>) {}
      
    • Some uses of expressions inside decltype cannot be properly distinguished. In VS2015 RTM, we store expressions inside decltype as tokens and we can’t distinguish expression in it, so any decltype is considered the same.We have started to move parsing of expressions inside decltype to the new parser since VS 2015 Update 1. With the new parser we’re able to distinguish some kinds of expressions. However, symbols are not bound yet so the compiler can’t distinguish between T and U. This means you are not able to define the two overloads in the following code sample. When we start to bind symbols in the AST tree generated by the new parser the compiler will be able to compare them.
      template<typename T, typename U> void f(decltype(T{})) {}
      template<typename T, typename U> void f(decltype(U{})) {}
      

      Currently produces this error message:

      error C2995: 'void f(unknown-type)': function template has already been defined

      One possible workaround is to use a helper class to create a unique type, as shown in this code:

      		template<typename T, typename Unique> struct helper { using type = T; };
      		
      		struct Unique1 {};
      		struct Unique2 {};
      		
      		template<typename T, typename U> void f(typename helper<decltype(T{}), Unique1>::type) {}
                      template<typename T, typename U> void f(typename helper<decltype(U{}), Unique2>::type) {}
      
  • A couple of issues impact type replacement during SFINAE.
    • Pack expansion: If the parameter pack is used in dependent expression or decltype, pack expansion may fail. Our current implementation of variadic template is based on tokens from the old parser so isn’t always able to handle arbitrary expressions. One example can be seen in the following code. If you use such expressions as part of the function declaration, then SFINAE won’t work correctly because pack expansion doesn’t happen. Identifying parameter packs and doing pack expansion will be much more robust once we move variadic templates to use the new parser’s parse tree.Compiling this code:
      		template<bool...>
      		struct S1 {
      			static const bool value = true;
      		};
      		
      		template<typename T>
      		constexpr T value() { return{}; }
      		
      		template <typename Ys>
      		struct S2 {
      			Ys ys;
      			template <typename ...X>
      			constexpr auto operator()(X const& ...x) const {
      				return S1<value<decltype(ys + x)>()...>::value;
      			}
      		};
      		
      		void f() {
      			S2<int> s;
      			s(0, 1);
      		}
      
      

      Currently produces this error message:

      error C3520: 'x': parameter pack must be expanded in this context
    • Alias templates: Type replacement may fail if an alias template has dependent expression or decltype in it. Our current implementation of alias templates uses type replacement and reparsing of the token stream from the old parser. The latter is used for dependent expression and decltype, but the context in which the reparsing is done isn’t always correct.If you use this kind of alias templates in a SFINAE context the result are currently unpredictable :-) Once we move alias template parsing to use the new parser we will no longer need to reparse the token stream, an operation that is sensitive to context and error-prone.Compiling this code:
      		template<typename> struct S {
      		 using type = int;
      		};
      		
      		template<typename T> using type1 = decltype(S<T>{});
      		
      		template<typename T> using type2 = typename type1<T>::type;
      		type2<int> i;
      

      Currently produces this error message:

      error C2938: 'type2' : Failed to specialize alias template

      A workaround to make this kind of alias templates work reliably in SFINAE context is to provide a helper class and use partial specialization for SFINAE purposes. This following code illustrates this workaround.

      		template<typename> struct S {
      		 using type = int;
      		};
      		
      		template<typename>
      		using type1_void_t = void;
      		template<typename, typename = void> struct type1_helper {};
      		template<typename T> struct type1_helper<T, type1_void_t<decltype(S<T>{}) >> {
      			using type = decltype(S<T>{});
      		};
      		
      		template<typename T> using type1 = typename type1_helper<T>::type;
      			
      		template<typename T> using type2 = typename type1<T>::type;
      		type2<int> i;
      

Send us feedback!

As always, we welcome your feedback. For problems, let us know via the Report a Problem option, either from the installer or the Visual Studio IDE itself. For suggestions, let us know through UserVoice. And you can always reach us through e-mail at visualcpp@microsoft.com.