Have you ever used a profiler? If not, why not?

Dear Reader, I have a question for you…

I am frequently amazed when smart developers ask questions about performance problems, but then it turns out they haven’t so much as tried a profiling tool on it.

I’m curious to understand why this is, and if there is anything I could write that might help people use these tools more effectively.

Are you a smart developer who has a problematic relationship with profilers? If so, what is holding you back?

  • Do you not know what tools exist?
  • Are their interfaces too confusing?
  • Did you try one, but then couldn’t figure out how to interpret the resulting data?
  • Other?

Comments (24)

  1. JoelMartinez says:

    I’d imagine that it’s a combination of all 3.  For example, I have visual studio team system, so I know I can use the built-in one.  But if I didn’t have it, I’d have to spend some time poking around to find one since I’ve never used another one aside from CLR profiler, which kind of only helps with GC issues (correct me if I’m wrong please 🙂 ).

    I think the other main issue is knowing how to interpret the results.  Honestly, even I struggle with this from time to time … I tend to do it so infrequently that I have to reacquaint myself with how to read the results.  It would be good to have a guide on what to look at in different circumstances (similar to your post about how to figure out if you’re cpu or gpu bound).

  2. scypior says:

    Ok, I use nprof frequently because it’s simple:) But I’ve never dived into the details of CLR profiler… simply because it’s not so easy to interpret result at the first sight.

  3. Warenth says:

    I’ve never pursued profilers as I haven’t stumbled across any information about them. When I search for them I typically find marketing buzz that I tend not to trust, rather than honest developer reviews denoting strengths and weaknesses.

    So, I guess mark me down for the category "Do you not know what tools exist?".

    Also: "Do you not know what they are capable of?" might apply as well.

  4. b_lawson says:

    Yes, I have used profilers.  Unfortunately, on VIsta 64 I cannot get the CLRProfiler to run my game.  It launches the .exe but the .exe then crashes immediately.  However, when I double click on the .exe and run it stand alone, it runs as expected.  Not sure what’s up with that.

  5. conkerjo says:

    Without clr profiler and Ants i wouldnt have a DBP entry right now. Profilers for teh win

  6. flipperanubi says:

    I was under the impression that:

    * Profilers didn’t run on XNA on the 360 (our target)

    * Profilers were either expensive or hard to use

    * And I don’t even know what tools exist because of these (perceived?) obscacles

  7. midas79 says:

    I have used them, but infrequently.  Lack of prominence is probably the number one reason.  Microsoft has a profiler available in Visual Studio, as pointed out by JoelMartinez, but it is only available in VSTS versions.  If Microsoft were to push that capability down the chain to Pro (and perhaps even Std) and increase the visibility, they would probably find more people would use it.  Throw in some MSBuild/TeamBuild tasks and we could even set up automatic profiling as a part of the test runs in the build process to track performance impact of changes.  So write your superiors and tell them the average developer needs access to the profiler across the Visual Studio version stack.

  8. CatalinZima says:

    I use CLR Profiler and PIX when I need to.

    When testing on the Xbox, I use the… what’s that name?… "XNA Framework Remote Performance Monitor for Xbox 360"

  9. SandyMaguire says:

    I don’t use profilers because I’ve never really gotten to point where I couldn’t figure out the bottleneck myself. I find it more fun to analyze code myself and try to logically work through the trouble spots.

    It doesn’t always work but it keeps me entertained. In those cases it’s never super critical, so I just don’t get around to it.

  10. CDarklock says:

    In general, I haven’t used profilers since the days of Borland’s Turbo Profiler tool. That was an awesome tool; you simply fired it up, pointed it to the debug build of your project, and it ran the application while injecting the proper code to time every function call.

    I have never had another profiler that was so easy to find and use. In the early days of my work with Microsoft Visual C++, I looked for a good profiler, but everything I found wanted custom code to install its hooks. For many years, my work tended to be input-bound and not processor-bound, so I eventually lost my need for a profiler and just stopped looking.

    I think what we really need in this industry is a solid enterprise-ready profiler with zero setup, like TP, which comes preinstalled in the IDE at all levels – so the developer can stumble across it and "discover" how useful it is.

  11. Ultrahead says:

    PIX + CLR Profiler.

    (one that I’d love to use is nVidia’s PerfHud 6 but I don’t want to mess with the way D3D devices are created)

  12. scypior says:

    Ultrahead: it’s extremaly easy: http://forums.xna.com/forums/t/9204.aspx

  13. Lynch3 says:

    I’ve been pointing these tools out to users in #xna on efnet, and converted a few.  I think what some people really need is the good tutorial that shows them how easy these tools are to use.

    A lot of new xna developers are also new to programming, and their skills might fall short of the concepts within these tool sets.

    As for the smart and experienced developers… they have no excuse, and it’s probably related to the fact many experience developers also recode many classes that are already standard in .NET.  Something about being stuck in their old ways and "not invented here" 😉

  14. DrJBN says:

    Only the clr.  I would love to give Nprof a whirl but I’m too @#* vs ignorant to get it to compile where it will run in Vista 64.

  15. Sir Lantis says:

    Hi Sahawn,

    – Do you not know what tools exist?

    Many people even didn’t/don’t know unit testing (but now VS provides them, so some should notice). So I wouldn’t expect they to know about profiling. Eclipse gives a hint with "Profile as…" but since it doesn’t work for all projects, nobody uses it.

    – Are their interfaces too confusing?

    Some of them are but all of them bring their own ugly IDE. I would be so happy to just click "Profile" in VS with timeinfo directly next to the line-numbers, but ok – even profiling using Eclipse/Java does not work that way.

    – Did you try one, but then couldn’t figure out how to interpret the resulting data?

    nProf website does a great example in introducing itself: Architecture blah, but "No documentation is available yet."

    – Other stuff

    nProf (the tool i liked most) just crashes on my Vista 64bit machine – best free tool, but useless. CLR Profile has similar problems.

    Aside from these problems I would really like to find a good ScreenCast/Tutorial about profiling, especially for how-to-find-* like thread-sync-probs, or graphics-related gpu/cpu bound issues etc.

  16. Ultrahead says:

    @scypior: thanks a lot for the link.

    One thing though: in order to prevent others from profiling your "final" game, those lines of code should be used only on DEBUG or PROFILE mode (not on a RELEASEd compilation).

  17. HerrUppoHoppa says:

    I have tried a variation of different profilers

    * nProf (which only crashes nowadays, even on XP)

    * IJW profiler (very pleasant profiler that shows what each thread is doing, but it crashes sometimes)

    * ProfileSharp (a bit unstable and somewhat confusing interface but is very detailed)

    * ANTS (nice but it costs money)

    * CLRProfiler (love it, use it often)

    * PIX (don’t really know how to use it for profiling so I only use it for debugging)

    I find that the main reason why I don’t use them anymore is because I don’t trust that they will show me information that is applicable to the xbox. This is mostly because we use threading and I don’t have a tripple core processor with hyper threading. This might lead to cases where some of the threads created on the PC gets their own core but others don’t while they would have gotten it on the xbox. Also considering how each core on the PC is significantly faster than those of the Xbox I just don’t see how the profile data could be representable. I don’t know, maybe I’m not thinking right about this but all the inconsistency confuses me and makes want to not use a profiler. And also right now I’m pixel operation bound, which makes it even more irrelevant for me.

  18. MrHelmut says:

    I’m using nProf and CLRProfiler when I’m working on performance sensitive project, or sometimes for the fun, just to see if there’s some bottleneck and/or possible improvements.

    Kind of easy to use, as long as you know how to read a call hierarchy.

    Most of the professional programmers in my surrounding are not aware of profiler and don’t care about generated IL code.

  19. markcusd says:

    I’ve used a few. Sometimes they helped, but most of the time getting clear information out of them was a pain and I end up just using timers and logs.

  20. cdxrd says:

    I gotta learn HOW to use these one of these days. Honestly I’m so lost when trying to figure them out that I give up.

  21. FlyingWaffle says:

    Hmm… I was under the impression that CLR only shows heap allocation (how many objects of each type are dynamically allocated) but not the time spent in each method.

    I’ve used Quantify quite a bit, but it’s not a cheap application. I managed to grab a couple of free .NET profilers and I’m planning to try them out this week.

  22. chrisandharris says:

    For me, it comes down to my time.  I barely get the leisure time I really wish I had for XNA, so I haven’t hit performance concerns just yet.

  23. jasond says:

    I know a profiler is the only real way to know which parts of the code need fixing.  But, like many, I’ve never used one (until my recent usage of the CLR Profiler 2.0 to analyze my Dream Build Play 2008 entry at the last minute).

    So, what’s holding me back?

    1. Do you not know what tools exist?

    No.  I know the CLR Profiler 2.0 exists only because the XNA people directed me towards that.  But, right now, I want to know if my floating point code is what’s slowing down the Xbox 360, or if it’s my collision detection code, or my camera code, etc.  The CLR only tells me about garbage collection and stupid things I’m doing such as newing classes on the heap when they should be structs on the stack for simple value types.

    Which ones do you recommend?

    2. Are their interfaces too confusing?

    I wouldn’t know.  I would love to use them, and answer this question.

    3. Did you try one, but then couldn’t figure out how to interpret the resulting data?

    The CLR Profiler seemed complex, but thanks to Microsoft’s very simple tutorial videos, I was able to make use of it.  Beyond that, it would be hacking and guessing to find any real information out of it, but I’m sure I could.

    4. Other?

    I’m a developer who has LONG wanted to run a profiler to see where my slow pieces of code are — the parts that take most of the CPU time.  And, I don’t even know where to begin at achieving that goal.  Thanks for making this post, and hopefully your efforts can shed some light on the issue for people like myself.

    Take care,

  24. srekel says:

    We used CLRProfiler and NProf for profiling on the PC. We also used the XNA Performance Tool (or something like that) on the xbox. The first two are excellent imo, the third one didn’t really help that much.

    Since the biggest problem was optimizing for the xbox, it would be nice with better profilers for it. We had runtime allocations down pretty low in Windows ( perhaps some hundred kb during the time it takes to complete a level, according to CLRProfiler), but the GC still kicked in on the xbox, so apparently there’s some kind of mismatch between them.