It’s all just a clock measuring contest…


Did you know that SBCL is now faster than Java, as fast as Ocaml, and getting better?! What does this mean for you? Most likely, nothing.

They have various benchmarks that are used to measure the speed of a language implementation; and I think it’s a good thing – it keeps the implementers honest, and I’m not crazy about wasting CPU cycles just because they’re there.

However, these numbers remind me very strongly of the local CompuMart selling the 2.4GHz machine for 1.5x the price of the 2.0GHz machine and relying on the fact that customers have no idea that this will make exactly zero difference for most uses.

Why? Because pretty much all programs nowadays depend on more than just the CPU. They’re competing for RAM, disk access and maybe even network. None of those are nearly as fast as the CPU, hence most often create a bottleneck.

Aside from this, it’s very likely that the code you’re writing will be more dramatically improved by tweaking your algorithms than it would by changing the language.

Couple that with the unfortunate fact that most jobs won’t allow you to just grab SBCL or OCaml because it happens to be at the top of the list today, and you’re left with a measurement that has little real-world value.

What these measurements are good for is preventing those who write the language implementations from getting too lazy. While I think that application programmers should generally be spending CPU cycles to save lines of code, I don’t necessarily believe the same for platform/language programmers.

Just don’t look at this list and get too disheartened about your choice of language. Even if it is Ruby 😉

 

Avi


Comments (6)

  1. Isaac Gouy says:

    > Did you know that “SBCL is now faster than Java, as fast as Ocaml, and getting better?”

    Headlines can be abbreviated to the point of being misleading – in this case the newsgroup post refers only to Java -client

    Look at Java -server on the same measurements and you’ll see Java faster than SBCL 😉

    But wouldn’t it be better to do a direct comparison?

    http://shootout.alioth.debian.org/gp4/lisp.php

    > you’re left with a measurement that has little real-world value

    Given your assumptions about the “real world” – which although reasonable for your “real world” may not apply in other situations.

    > because it happens to be at the top of the list today

    I don’t recall many dramatic changes at the top of that list over the last 18 months – just the addition of Lisaac & Intel C++ .

    > Just don’t look at this list and get too disheartened about your choice of language. Even if it is Ruby 😉

    If it is Ruby we should note that Core 1.9.0 seems better.

    http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=yarv

  2. avip says:

    Yep, as you point out different methods of measurement and different releases will yield different results. Java is faster/slower, Ruby is faster/slower. My point is that it almost never matters.

    There aren’t many programs which are heavily CPU bound, but there are some. Firstly, most of those would likely get more benefit from changing their overall methods of doing things (eg; data structures, algorithms) than they would by just swapping out the runtime.

    Second, consider that most of these programs are probably already hand-tuned in C, allowing them to achieve speeds which should be on par with any other language (all other things being equal), seeing as C is a low level language.

    Lastly, how many of these programs have you seen actually written in Ocaml, Lisp, SBCL, etc? Not many – because altering platforms on a whim is just not something that’s very doable. There are issues of libraries, runtimes, compatibility, third party controls, etc.

    So yeah, there’s the N% (<1) of programmers who will use a language because it has the fastest implementation today, but such a small percentage doesn’t constitute "real world", IMO.

  3. Isaac Gouy says:

    > Not many – because altering platforms on a whim is just not something that’s very doable. There are issues of libraries, runtimes, compatibility, third party controls, etc.

    In “the real world”, in multi-national corporations, I’ve personally experienced the entire software development infrastructure being changed globally – twice.

  4. avip says:

    Isaac, was it done because of the perception that this would give a runtime perf improvement? If so, I would judge that to be a very questionable decision.

    I didn’t say that companies don’t change things, ever. Just that the costs involved generally mean that a simple speed measurement is not sufficient cause.

  5. Isaac Gouy says:

    > Isaac, was it done because of the perception that this would give a runtime perf improvement?

    Yes.

    > Just that the costs involved generally mean that a simple speed measurement is not sufficient cause.

    In some businesses speed is money and the fastest takes all.