Not as easy as it looks


My colleague Kevin works on (among many other things) the refactoring engine in the C# IDE. He and I were at the end of last year discussing the possible cases for a hypothetical “eliminate variable” refactoring. I thought that it might be of interest to you guys to get a glimpse of the sorts of issues that the IDE team faces on a daily basis.

First off, what is the “eliminate variable” refactoring? It is seemingly straightforward, but as we’ll see, appearances can be deceiving. Suppose you have a local variable that is initialized in its declaration, never written to otherwise, and you have only one place where that variable is used. (A generalization of this refactoring is to replace every usage; for simplicity, let’s consider only the single usage case.) For example:

void M()
{
    int x = 2 + R();
    Q();
    N(x);
}

You can eliminate the variable by refactoring the code to:

void M()
{
    Q();
    N(2 + R());
}

Now, obviously this changes the meaning of the code. If Q() throws an exception then now R() is never called, whereas before it was always called. If Q() observed a side effect of R() in the first version, it does not after the refactoring. And so on; there are numerous ways in which this code is different. We assume that the user is all right with these changes; that’s why they’re using the refactoring.

How, at a high level, would you implement this? It seems straightforward:

  • Find the text of the initializer
  • Find the single usage (which the IDE already knows how to do somehow)
  • Textually replace the single usage with the initializer text
  • Erase the declaration and initialization of the variable

That last step might be slightly tricky if you have something like

    int y, x = 2 + R();

because we have to make sure to erase the comma and the declaration and initialization of x, but not erase the “int y;“. But that is not hard to deal with.

You also have to think a bit about comments. What should happen here?

    // x is blah blah blah
    int x = 2 /* blahblah */ + R(); // Blah!
    Q();
    N(x);

Should that become

    // x is blah blah blah
    /* blahblah */  // Blah!
    Q();
    N(2 + R());

Or should any of the three comments before, inside or after the declaration be inserted where x used to be used? Notice that if the one that comes after is inserted, it’s going to have to be changed to an inline comment, or a newline is going to have to be inserted in the inside of the call. Also, a comment refers to a local variable that has been eliminated, which seems bogus.

Assume that we can solve these problems. Are we done? Let’s apply our refactoring to a few test cases. In every case, we eliminate “x”.

    const short s = 123;
    int x = s;
    N(x);

becomes

    const short s = 123;
    N(s);

Is that right? Not necessarily. What if N has two overloads, one for int and one for short? Before we called N(int); now we are calling N(short)! This seems qualitatively different than merely reordering a possible side effect; now our refactoring is changing overload resolution, which seems wrong. Really we should be generating

    const short s = 123;
    N((int)s);

OK, is that the right refactoring? Let’s keep on trying test cases.

    int x = 123;
    N(ref x);

becomes… what? We can’t say N(ref 123) or even N(ref (int)123).

In this case, the refactoring engine has to fail. There is no way to eliminate the variable when what you’re passing is a reference to the variable itself.

How about this?

    int x = R() + S();
    N(Q() * x);

becomes

    N(Q() * R() + S());

Argh, we have now introduced a semantic change because of operator precedence. That should be

    N(Q() * (R() + S()));

How about this?

    Func<int, int> x = z=>z+1;
    object n = x;

becomes

    object n = z=>z+1;

Nope; you can’t implicitly convert a lambda to anything other than a delegate or expression type. That’s got to be

    object n = (Func<int, int>)(z=>z+1);

Whew. What have we learned so far? To eliminate T x = expr, there are scenarios where you can use:

  • nothing at all; give an error
  • expr
  • (expr)
  • (T)expr
  • (T)(expr)

An exercise for the reader: find scenarios in which none of the above are good enough either. Can you find a scenario in which you have to say ((T)expr)?  How about ((T)(expr))? The latter was the worst I could find easily; are there any more that I missed?

Understandably, we would not want to implement this refactoring in a hypothetical future version of Visual Studio if every time you did an elimination of a variable, the compiler conservatively inserted four parens plus a cast; what we really need are heuristics which can tell us when each of the above forms is necessary. That is a surprisingly tricky problem in language analysis!

And these are just the issues that Kevin and I came up with in a few minutes of whiteboard doodling; it is entirely possible that on deeper analysis, we’ll find more interesting problems.

Next time you use the refactoring tools in Visual Studio, think about everything that is happening behind the scenes to make those tools work seamlessly. Like playing the piano, the unseen effort required to make it look easy is enormous.

Comments (41)

  1. Tim Robinson says:

    Has Kevin seen ReSharper's equivalent of "eliminate variable"?

  2. Joe White says:

    All of which makes me even more impressed by tools like ReSharper that have dozens of refactorings like this. I'm always a little bit surprised when I run across another one of these edge cases and they get it right!

  3. Khoth says:

    How about:

    class C
    {
       int y = 2;
       void M()
       {
           int x = y;
           int y = 4; // shadows field y
           Foo(x);
       }
    }

    Then, you need to change Foo(x) to Foo(C.x)

    Interesting idea, but the analysis is not quite right. Remember, y is in scope throughout the block, not just after the declaration, so this is a "variable used before declaration" error; the first 'y' refers to the local, not the field. You're thinking of C++. (Also, don't forget that C# makes it illegal for the same simple name to have two meanings in one block.) – Eric

  4. SSLaks says:

    How about

    static void Main()
    {
      Funny a = null;
      Func<int> b = () => 1;
      Console.WriteLine(a + b);
      Console.WriteLine(a + ((Func<int>)(() => 1)));
    }
    class Funny {
      public static int operator +(Funny a, Func<int> b) { return 1 + b(); }
    }

    Yep, that's the same case that I came up with. Great minds think alike! – Eric

  5. Random832 says:

    What about a generalized refactoring function to insert* or remove unnecessary parentheses and casts in a selected expression? This is something that could be useful on its own, and it could simply hand off to that after it's made its own version with ((T)(expr)) – though this could result in the undesirable effect of altering the text of expr.

    Are there any situations where this could result in needing to cast to an anonymous type?

    * yes, insert.

  6. Stuart says:

    Seems to me that you'd start with ((T)(expr)) and then look to see which parts of it were redundant, and eliminate them.

    Having "eliminate redundant parentheses" and "eliminate redundant casts" as primitive operations within the refactoring engine seems like something that would be needed, or at least useful, for all kinds of refactoring operations (and possibly it'd be useful to expose them directly to end users, too).

  7. Ilya Ryzhenkov says:

    Sometimes you need to rename variables, e.g. inline "f" here:

           var f = (Func<int, int>) (x => x);

           using (null)

           {

             var x = "sdfsdf";

             var y = f(1);

           }

  8. Ilya Ryzhenkov says:

    @Khoth you code has error, because you cannot use "y" in M() before it is declared.

  9. Ilya Ryzhenkov says:

    Another example, inline "numbers":

         int[] numbers = {1,2,3};

         Action<int[]> a = ints =>{};

         a(numbers);

  10. Ilya Ryzhenkov says:

    @Knoth giving it another thought, we can change your sample as follows, inline "x" here:

    class C
    {
     private int y = 2;
     private void M()
     {
       var x = (Func<int>) (() => y); 
       {
         int y = 4; // shadows field y
         int p = x();
       }
     }
    }

    It should change y in lamda to "this.y"

    Nice. Because the two simple names 'y' are not first used in overlapping declaration spaces, they are permitted to have different meanings. The refactoring causes the declaration spaces to overlap.

  11. Serjic says:

    var x = 123;

    var y = new {x};

  12. Simon Buchan says:

    The problem description seems strange – are refactorings even thought of in text replacement terms?

    It seems natural to me that since refactorings can only operate over a connected graph of recovered structure from the text (it has to be able to parse largely everything from the start to the end braces of a method body, for the posited refactoring, for example), refactorings proper should not be rewriting source *text* at all. It (again, to me) seems more natural that a refactoring engine would be generating AST transforms which it passes to a source code rewriter. In this case, the initial parsing would decide how comments attach to source and validation passes would catch missing parens and invalid comment positioning in the AST – all before the rewriter applies the changes.

    The AST is largely already there for intellesense purporses; a lot of the logic you have to do the way you are describing is pretty much the same, just in different places; and it generally seems to be it's aligned better with a lot of the directions Visual Studio seems to be taking (extensibility of the editor and the compiler teams compiler-as-a-service work, for example).

    Is this a "that's what we are doing", a "yeah that's nice, but bad cost/benefit" or a "that doesn't work"?

    The relationships amongst the "top-level-type tree", the fully-analyzed "expression" tree, the syntax tree, the token stream and the hunk of characters managed by the editor are complex, to say the least. In practice, many refactorings are done by manipulating the syntax tree as you describe, re-running the analysis pass to make sure the expressions still are correct, and then turning the syntax tree changes into a set of text diffs that can be applied to the editor. This can be an expensive and difficult process with many different levels interacting with each other. I'm glossing over most of those details here. — Eric

  13. jader3rd says:

    I tend to have the opposite problem of the first one stated. I find myself with

    void M()

    {

    Q();

    N(2 + R());

    }

    and I'll want to pull out 2+R(). Since I'm assigning it to a variable, I'd like it if the Visual Studio IDE atuomatically put the variable I just created as the parameter in the call to N(). Sure, that would create more work if I was trying to change the meaning of the code and wanted to call a parameterless N(), but I can't never recall a time where I wanted that, and I cut and pasted the old parameter. If I cut and paste a parameter into an assignment of a variable, that should give the IDE enough of a hint to put the variable in the spot of the parameter for me.

  14. Jon Skeet says:

    I don't know whether you'd count this, but we may need to include "checked" or "unchecked". For example, take:

       unchecked
       {
           int z = X() + Y();
           checked
           {
               M(z + 1);
           }
           // Other code (just for simplicity so we can't completely remove the unchecked block)
       }

    If we want to extract z, I think we'd need:

       unchecked
       {
           checked
           {
               M(unchecked(X() + Y()) + 1);
           }
           // Other code
       }

    That could probably be combined with the example from SSLaks.

    Nice one! – Eric

  15. Shuggy says:

    You can come up with some quite nasty ones that won't work with checked contexts. though you could fairly easily refuse to move any arithmetic operation from outside a checked, into a checked.

    int a = 0;

    int one = 1;

    int bs = int.MaxValue + one;

    Console.WriteLine(checked(a + bs)); // works

    Console.WriteLine(checked(a + (int.MaxValue + one))); // throws

  16. ShuggyCoUk says:

    curses, skeet shot.

    anything involving collection initialisers is going to add to the complexity:

    int[] foo = {0,1};

    Console.WriteLine(foo);

    Console.WriteLine({0,1}); // illegal

    Console.WriteLine(new int[] {0,1});

  17. Shuggy says:

    curses, skeet shot.

    anything involving collection initialisers is going to add to the complexity:

    int[] foo = {0,1};

    Console.WriteLine(foo);

    Console.WriteLine({0,1}); // illegal

    Console.WriteLine(new int[] {0,1});

  18. Shuggy says:

    sorry for the dupe:

    here's another example where you can't do it (ignoring that the operator itself is likely to be error prone wrt side effect)

    int x = 1;

    int a = ++x;

    Console.WriteLine(++a);

    int y = 1;

    Console.WriteLine(++(++y)); // illegal

    I'm sure someone can do worse with implicit and explicit casts but here's a really unpleasant case if you were only dealing with text rather than the abstract syntax tree:

    void Main()

    {

    short y = 0;

    Frober x = y;

    Console.WriteLine(x);

    Console.WriteLine((Frober)y); // wrong

    Console.WriteLine((Frober)((int)y)); // right – but needs the compiler telling it that this happened

    }

    // Define other methods and classes here

    class Frober

    {

    int X = 1;

    public Frober(int i) { this.X = i; }

    public static implicit operator Frober(int i) { return new Frober(i); }

    public static explicit operator Frober(short i) { return new Frober(i + 1); }

    public override string ToString() { return this.X.ToString(); }

    }

  19. df says:

    A somewhat coerced example, but replacing x here would fail since C as a dynamic can only be implicitly cast to int.  The variable cannot be eliminated.

       class C : System.Dynamic.DynamicObject

       {

           public override bool TryConvert(ConvertBinder binder, out object result)

           {

               if (!binder.Explicit && binder.ReturnType == typeof(int))

                   {

                       result = 3;

                       return true;

                   }

               return base.TryConvert(binder, out result);

           }

       }

       class Program

       {

           static void Main(string[] args)

           {

               Action<int> A = i => Console.WriteLine("A(int) called.");

               int x = (dynamic)(new C());

               A(x); // works

               // A ((dynamic)(new C())) and A((int)((dynamic)(new C()))) throw.

           }

       }

  20. Pete says:

    Eric,

    I question if this type of refactoring is something that you want to encourage.  Let's assume that your first two examples don't have any gotchas like side effects or exceptions thrown in Q.  I would still say that the functions are different from a readability/maintainability standpoint.  For example, I would write the first first function if 2 + R() has some meaning in and of itself.  I would write the 2nd function if 2+R() only means something to N() but doesn't have any meaning in another context.

    I'd be concerned that having Visual Studio do this automatically will encourage collapsing the first function into the second function all the time at the expense of readability.

  21. configurator says:

    Shuggy, I think the ++ case is pretty much the same as a ref case – you can think of ++ as an operator accepting a ref parameter.

    Eric, on my current project I could give you hundreds of examples, because the client insisted that we use the "tool" StyleCop, with most errors enabled. (the word tool is in quotes here because a tool is something that's supposed to help development and StyleCop does the exact opposite…)

    var route = new

    {

     controller = "A",

     method = "B",

     parameter = "C"

    };

    routes.MapRoute(

     "Name",

     "Path",

     route

    );

    In this example,.you'd get an error for the seemingly fine

    routes.MapRoute(

     "Name",

     "Path",

     new

     {

       controller = "A",

       method = "B",

       parameter = "C"

     }

    );

    SA1118: The parameter spans multiple lines. If the parameter is short, place the entire parameter on a single line. Otherwise, save the contents of the parameter in a temporary variable and pass the temporary variable as a parameter.

    We've learned to avoid having more than one or two parameters in an MVC route because of that… Which I find very sad.

  22. configurator says:

    Why do you need the extra parentheses around ((Func<int>)(() => 1)) in SSLaks's example? They seem redundant to me.

    Another example where you would need extra parentheses with functions is when they're applied:

    Func<int> x = () => 1;

    int y = x();

    would turn into

    int y = ((Func<int>)(() => 1))();

  23. configurator says:

    Another example for the extra parentheses: implicit cast and call member

    class Bd {

       public int Value;

       public static implicit operator Bd(int value) { return new Bd { Value = value }; }

    }

    Bd x = 1 + 2;

    Console.WriteLine("The value is " + x.Value);

    would be turned into

    Console.WriteLine("The value is " + ((Bd)(1 + 2)).Value);

  24. configurator says:

    Here's another interesting case.

    int x = 42;

    var properties = new { x };

    would need to be converted to

    var properties = new { x = 42 };

  25. configurator says:

    And last one for today: Expressions go more wrong than anything else.

    int x = 1;

    Expression<Func<int>> e1 = () => x;

    Expression<Func<int>> e2 = () => 1;

    These two are completely different.

  26. josheinstein says:

    I hope you compensate Kevin well. I'd have been fired by now for making too many tests fail.

  27. Gabe says:

    I'd like to point out that Khoth's example is perfectly valid. The code is syntactically correct C# and I would expect refactorings to work on code that I haven't finished writing yet. Of course that could be a situation in which it couldn't perform this refactoring, but it's certainly a valid input to the refactoring engine.

    Would it give warnings when moving expressions across scopes?

       var x = MightThrow();

       try {

           Foo(x); // can we eliminate x here?

       } catch {

           // swallow all exceptions

       }

    Obviously that could change the semantics of the code more than one might expect. What about this:

       int LockX()

       {

           lock (X) { return Bar(); }

       }

       int LockY()

       {

           int bar = LockX();

           lock (Y) { return Foo(bar); } // can we eliminate bar here?

       }

    Now you've changed the logic from locking then unlocking X and Y in sequence to acquiring X while holding lock Y. If the proper protocol is to acquire X and then Y, you've created a potential deadlock.

       int limit = ExpensiveOperation();

       for (int i = 0; i < limit; i++) // can we eliminate limit here?

       { }

    Eliminating limit there would change the code from doing a single expensive operation to untold numbers of them. A variant of this is capturing variables:

       var obj = new Obj();

       D del = () => Foo(obj); // can we eliminate obj here?

    If we eliminate obj then instead of sharing it among all invocations of del, a new Obj instance will be created for each invocation. One might even consider disallowing a substitution like this:

       foreach (var x in L) {

           var V = x; // create new variable to hold loop iteration variable

           Foo(() => Bar(V)); // can we eliminate V here?

       }

    Whenever I write code like that I put in a comment saying why V is there so that nobody comes along and tries to refactor V away and change what variable is closed over.

    One interesting situation where you would need the parens is here:

       bool J = x<y, K = w>(z);

       F(J, K); // can we eliminate J and K here?

    You can safely replace either J or K without parens. However if you try to replace both J *and* K then you will get either an error or change the overload resolution for F without adding parens around one or both of them..

  28. Wilfried says:

    Designing a language explicitly for perfect refactorability (instead of using a syntax compatible with existing mindsets) may be quite interesting.

    How far could the code editing process be shifted from traditional text editing towards snippet selection + refactoring?

  29. Shuggy says:

    continuing configurator's puash to find yet more cases requiring ((TYPE)(X)):

    DayOfWeek d = default(int);

    Console.WriteLine(d.ToString());

    Console.WriteLine(((DayOfWeek)(default(int))).ToString());

    msdn.microsoft.com/…/2bxt6kc4.aspx

    and I believe anything with higher precedence than typecast which is Unary and postfix (if prefix as with ~ you can get away without the outer brackets)  which you can make applicable.

  30. JBSnorro says:

    I also found an interesting case you probably didn't expect: the refactoring can leave an out-parameter unassigned where it previously wasn't. All it takes is that the out-parameter is set in the declaration of the variable to eliminate and a conditional returning branch before the usage of the variable to eliminate. For example:

    void Foo(out int b)

    {

       int x = b = R();//x is the variable to eliminate

       if (Q()) return;//b is assigned

       doSomething(x);

    }

    would after refactoring become

    void Foo(out int b)

    {

       if (Q()) return;//error: b is unassigned

       doSomething(b = R());

    }

  31. Tanveer Badar says:

    You cannot replace a variable's all occurences with its initializers either, but only if the initializer is pure. Case in point:

    int f( )

    {

        ++x;

        return 1;

    }

    int y = f( );

    Console.Write( y );

    Console.Write( y );

    Calling f two times will increment x twice instead of only once.

    Blind text replacement (definitely a bad idea) is going to break things like this:

    int x = k( );

    f( x: x );

  32. masklinn says:

    Eric, why do you have to make up new names for well-understood patterns? This refactoring is called "inline temp(orary)" and it's been called that for more than a decade now (Fowler's Refactoring was published in 1999, and the Refactoring Browser's seminal paper was released in 1997). *Eliminate* aliases to *remove*, and "Remove Variable" is a very different refactoring: it completely removes a binding and its initialization from the source code if the binding is never used.

    First: a quick web search will show that other people have called this refactoring "eliminate variable". This is not a made-up-by-me new name. Second, so what if it was? We're having an informal conversation here about the difficulties of implementing simple refactorings; the name we choose to informally speak about the hypothetical feature is irrelevant to the discussion. Surely there are more constructive ways to participate in this discussion than sniping about nomenclature. Third, "inline temp" is a horrid name compared to "eliminate variable" for numerous reasons.(Just a few: inlining is primarily associated with JIT code generation, the variable in question might not be thought of as logically a "temporary", one can imagine extending the refactoring to other kinds of variables other than "temporary" locals, and the fact that the name emphasizes the elimination of a varible is a strong and helpful hint that you'd better not be using the thing as a variable, but rather, as a value.) Fourth, it is far more important that a name be clearly understood by customers who are not already familiar with the concept than that it conform to some alleged standard name understood only by persons already steeped in the literature of refactoring. — Eric

  33. To get the obvious out of the way, consider:

       struct X { public int Y; }

       static void Main()

       {

           X x = new X();

           x.Y = 123;

       }

    "x" cannot be substituted with "new X()", nor with anything, really – "new X().Y" is not an lvalue (I don't know the proper concise C# term for this, if it has one).

  34. Harold D says:

    @Pavel Minaev

      struct X { public int Y; }

      static void Main()

      {

         new X().Y = 123;

      }

    This should be perfectly valid…

  35. @Harold: go ahead and try it!

  36. Daniel Ziegler says:

    Another case where four parentheses and a cast are necessary:        

    Func<int, int> x = z => z + 1; //eliminate x

    int h = x.GetHashCode();

    This becomes:

    int h = ((Func<int, int>)(z => z + 1)).GetHashCode();

    More interestingly, in a similar vein as Ilya Ryzhenkov's shadowing example, here's a contrived case where the elimination needs to be blocked.

    Action<int> f = n => { int x = n; };

    {

    int x = 5;

    f(3);

    }

    This is the result, but it doesn't work, since the declaration of x inside of the lambda conflicts with the other declaration:

    {

    int x = 5;

    ((Action<int>)(n => { int x = n; }))(3);

    }

  37. Luke P says:

       int x = R() + S();

       N(Q() * x);

    becomes

       N(Q() * R() + S());

    Also has the problem that if Q() throws an exception and R() or S() have side effects then it will change behaviour.

    Do we assume the refactorer is aware of this or stop the refactoring in this case?

  38. googly says:

    @Pavel Minaev:

    ReSharper can deal with that. I've used it many times.

    It'll refactor it to:

      struct X { public int Y; }

      static void Main()

      {

          X x = new X { Y = 123 };

      }

    The quick fix is called "use initializer syntax" IIRC.

  39. Harold D says:

    @Pavel Minaev

    Leave it to me to stick foot in mouth.  I hadn't noticed "struct".  I've done the same thing with classes "new myclass().something = foo" just because I was lazy & didn't need the myclass afterwards.  I didn't know that struct had different behavior in this respect.  

  40. For the case:

       int x = 123;

       N(ref x);

    I believe this could become:

       N(123);

    But only if N is a method on a COM interface in C# 4.0. 🙂

  41. qwertman@hotmail.com says:

    I'm with jader3rd. A much more useful refactoring would be common subexpression factoring, to find all the instances of the same expression in a method, and replace them with a variable.