Performance Quiz #1 (of a series?)

Recently there was a discussion on one of our internal email aliases in which this problem came up. I though it was an interesting problem so I posed this Quiz to assorted people I work with to see what kinds of things they would say.

Considering these three options:

Option 1:

sw.WriteLine(subject + ": " + message);

Option 2:

sw.WriteLine("{0}: {1}", subject, message);

Option 3:

sw.Write(subject);
sw.Write(": ");
sw.WriteLine(message);

Answer these questions:

Q1. Which of these choices offers the best performance?
Q2: Can you characterize the memory usage of each of these?
Q3: Which would the performance team generally recommend, and why?
Q4: What special factors might alter this recommendation?
Q5: What did you have do assume about "sw" to answer the question?

I encourage you to think about this for a few minutes (more is good too) before reading past the line. SPOILERS/ANSWERS follow.


For everyone that took the time to think about this at all, thank you very much.

In the course of my job I'm often asked to comment of the probable performance of an assortment of solutions to give guidance, or at least suggest what measurements should be made. So I approached this quiz the same way and made my best guesses/recomendations as I would if I could not do the measurements. Then I went back and did the actual measurements.

Here are my own answers:

Q1. Which of these choices offers the best performance?

  • Only thing I can say for sure is that #2 will lose to #1 in all cases
  • #3 is going to be best if the output is buffered
  • #1 is going to be best if the output is unbuffered
  • In any case, a typical program's overall performance will be unaffected by the choice

Q2: Can you characterize the memory usage of each of these?

(These answers all proved to be "close but no cigar" due to the unusual WriteLine behavior discussed in the real analysis, see below)

All three probably have allocations associated with buffering the stream, ignoring those as invariant, the allocations unique to each choice are:

#1 single concat operation, one temporary string
#2 assorted allocations, including string builder, underlying string buffer, vararg array (I was close)
#3 no allocations

Q3: Which would the performance team generally recommend, and why?

Even though it's the worst performing, and we knew that much in advance, both of your CLR Performance Architects concur that #2 should be the default choice. In the highly unlikely event that it becomes a perf problem the issue is readily addressable with only modest local changes. Normally you're just cashing-in on some nice maintainability. We do not alter our choice given the data below.

Q4: What special factors might alter this recommendation?

Specific measurements indicating that the code path had become a hotspot.

Q5: What did you have do assume about "sw" to answer the question?

Only that the stream did not have exotic behavior (such as weird cryptographic features that make the cost model very complex) and that it was buffered. In the event of an unbuffered stream of one type or another there are signficant semantic differences between (1 or 2) and (3) and potentially huge perf differences too.

OK, time to see how we did.

To do the analysis below I used the following benchmark program and CLR Profiler, which is one way to look at this data.

namespace Test
{
    using System;
    using System.IO;
 
    class Test
    {
        static private String s1 = "Hello";
        static private String s2 = "Good bye";
        static private int iterations = 5;
        static private volatile int foo = 0;
 
        static private MemoryStream ms = new MemoryStream(100000);
        static private StreamWriter sw = new StreamWriter(ms);
 
        public static void Main(string[] args)
        {
            int i;
 
            for (i=0;i<iterations;i++) Test1();
            for (i=0;i<iterations;i++) Test2();
            for (i=0;i<iterations;i++) Test3();
        }
 
        public static void Test1()
        {
            sw.WriteLine(s1+": "+s2);
            foo++;
        }
 
        public static void Test2()
        {
            sw.WriteLine("{0}: {1}", s1,s2);
            foo++;
        }
 
        public static void Test3()
        {
            sw.Write(s1);
            sw.Write(": ");
            sw.WriteLine(s2);
            foo++;
        }
    }
}

The results below are in the form of an execution trace showing functions and allocations for each of the three options as reported by CLRProfiler.

     
             
   
       
 
       
         
         

             

 
       
   
         
     
       
         
               
                 
     
         
             
             
           
               
             
                 
           
             
                 
             
                 
         
             
             
             
   
         
           
           

     
             

     
         
     
         
 
       
         
         
     

This is already too long but I thought I'd end by sharing a quick summary of the nature of the responses I got internally.

  • Some thought the problem was very easy. Many of these people didn't do very well :)
  • Everyone (including me) was suprised by the allocation in #3.
  • Some people thought all the options would go through the formatting logic, looking for { and } and so forth, that's not the case if there is only one argument to Write/WriteLine.
  • Some people feared the three function calls they could see in Option 3, but didn't think about the internal work assocated with formatting in #2 that they could not see. There was a great tendancy to assume WriteLine was magic or rocketscience -- generally a bad idea. The three virtual function calls are really the least of your problems with #3.
  • Some people assumed no buffering (fair enough), which majorly colors the answer.
  • Some had supreme faith that we had done magic in #2 to make it perfect... alas no.
  • And lastly, Some were afraid to guess at all, must be people that read my blog :)