Erlang at Yahoo!

Jebu Ittiachen and Arun Suresh of Yahoo! have written an interesting article for Dr. Dobbs about a prototype they’ve built for Yahoo! Harvester using Erlang. The performance gains they achieved are quite impressive.

Some of the more interesting highlights/discussion points I’ve pulled from the article:

  • The language itself is only part of the draw for Erlang users.  The OTP framework provides great value to those writing distributed applications. “Since the restart functionality is already found in the standard library, no extra code was required to be written.”  What parts of the OTP do you feel are essential?  What’s missing?
  • It reiterates some of Erlang’s flaws (we’ve heard lots of complaints about syntax).  I read this as a major opportunity for us to provide a highly differentiated solution and increase developer productivity (which is what Visual Studio is all about!).  “Erlang is not the perfect platform.  It has its share of weaknesses and most of it comes from the design goals behind the language.”  What else besides syntax and the lack of true string data types gets in your way?
  • Performance comparisons.  WOW.  The performance gains on quad-cores are simply incredible.  100% CPU efficiency and a 3750% increase in MAX capacity per machine.  These numbers show major cost savings  for our customers.  Of course, it all depends on the baseline of which you are comparing against but I think it’s a safe assumption that Yahoo!’s developers are capable of writing fairly scalable, fairly performant code. 
  • It’s interesting to note that they’re using Erlang “successfully” in 5 properties.  We’re seeing a lot more commercial products written entirely in Erlang or with components written in Erlang.  Some other noteworthy examples include Amazon, Facebook Chat, and Microsoft’s own PowerSet.  This trend suggests that companies are willing to overcome the learning curve and employ once-obscure technologies to achieve performance gains and, ultimately, cost savings in both hardware scale-down and developer productivity. 


Comments (1)

  1. Josh,

     Thanks for nice words on the article that we did.

     On the questions that you had on our article –

    OTP Essentials

     The supervisor tree is perhaps the single most important aspect, the ability to structure processes as part of a hierarchy, provide restart strategies, add dynamically to the tree all these are must haves. The three core behaviors gen_server, gen_event, gen_fsm absolute strokes of genius, each has its well defined roles providing solid foundations to build on. Actually there are very few things u would want to remove from OTP. For our specific application, we needed SSH libraries, Unit testing frameworks, code coverage utils.. etc OTP come prebundled with everything..

    Whats lacking in OTP?

     Well maybe clearer documentation on the whole release packaging aspects. Also clearer documentation on the return states from each callbacks would be good, but not a blocker since it allows you to peek into the otp source and discover new ways of doing things.

    Once you get the release packaging all sorted out. You gain access to all the hot-deployment goodness OTP has to offer. Agree hot code deployment is not new.. but no other platform can boast such a seamless offering.

    Drawbacks on Erlang as such?

     Tighter integration of dialyzer into erlc, it sucks to figure out that you are calling into a non existing function.

     More wider user of @spec to document the code, more a note to self.

     Better record manipulations ofcourse.

     More BIF’s in guard expressions.

    The crib on the syntax was more of a warning to the beginner. There is a certain element of elegance to the erlang syntax once you become sufficiently well versed with it that you start wondering why none of the others have this clarity of expression.