TPC-E – Raising the Bar in OLTP Performance

Glenn Paulley, Director of Engineering at Sybase iAnywhere, posted a commentary titled “The State of TPC-E” on his blog three weeks ago (10/3/08).  A better title would have been “All TPC-E Results Are On Microsoft SQL Server.  Why?”  Mr. Paulley takes issue with Brian Moran’s statement that “the most rational answer is that Oracle and IBM have tried to top Microsoft’s numbers and simply can’t”.  He says that while it may be true, he doubts it and says there are other plausible reasons why DB2 and Oracle have yet to publish any TPC-E results.  Curiously, he doesn’t say why Sybase hasn’t published TPC-E results.  Since he is, presumably, in a position to know, one can only conclude that he would rather not say.  Readers can reach their own conclusions about what that might mean.


To his credit, he cites this IBM whitepaper for explaining that TPC-E was designed to be more realistic than TPC-C.  There are numerous ways, detailed in the whitepaper, in which TPC-E is far superior to TPC-C.  Let’s compare TPC-E to TPC-C.  As the table below shows, in TPC-E the schema is substantially richer and more complex, there are twice as many transactions, and only TPC-E requires essential capabilities such as referential integrity and RAID protected storage.







Number of database tables



Foreign keys



Tables with foreign keys



Check constraints



Partitioning Characteristic

unrealistic; single dimension common

to 8 of 9 tables


two independent dimensions




Number of transactions



Database roundtrips per transaction


min 1; max 5




Referential Integrity Required



Storage Protection (e.g. RAID) for Database Required

Log Only


Timed Database Recovery test




Mr. Paulley chooses to focus on the query complexity of TPC-E.  While that’s somewhat interesting, a comparison to TPC-C would have provided important context.  For example, TPC-E has 156 DML statements.    Although TPC-C doesn’t include pseudo-SQL the way that TPC-E does, if it did and followed the TPC-E style, it would be fewer than 30 DML statements.  By this measure, TPC-E has more than five times as many distinct DML operations as TPC-C.


But more importantly, TPC-E is not and was never intended to be a query optimizer test.  The pseudo-SQL code in TPC-E is an example, not a requirement.  Unlike TPC-H which strictly limits changing the specified SQL, in TPC-E test sponsors are free to rewrite the SQL anyway they like as long as it is functionally equivalent.  One vendor might rewrite it to remove all joins while another might rewrite it to include more joins or more complex joins.  The same is true of group by and order by clauses.  In our view, Mr. Paulley’s objection that TPC-E isn’t a good optimizer test is misplaced.


After discussing query complexity, Mr. Paulley offers four reasons why Microsoft is the only database vendor publishing TPC-E results.


·         “TPC-E is a moving target” – While it’s true that the TPC-E spec is up to version 1.6.0, the assertion that the workload has changed significantly is unsupported by the facts.  None of the transactions has changed in any way that impacts performance.  All spec revisions have been classified as “minor” changes by the TPC and results across all spec revisions are comparable.  The number of revisions to the spec since it was first released actually reflects a deep commitment by the members of the TPC-E committee to clean up rough edges and address areas of ambiguity before they become issues in published results.  A better gauge of the high quality of the TPC-E spec is that to-date 18 results have been published by six vendors spanning 15 months, but there have been no compliance challenges. 


·         Both DBMS vendors and hardware suppliers have a substantial investment in TPC-C expertise.  On this point we agree with Mr. Paulley.  But we draw different conclusions.  All of the major DBMS companies have spent years picking through every detail of TPC-C.  It has been optimized to such a degree that it long ago stopped driving customer-relevant engineering improvements.  TPC-C is 16 years old and has changed little since 1992.  Saying that we should continue using TPC-C because we know it so well is like saying that we should drive horse and buggies because we have a lot of expertise in blacksmithing.  This is a mindset trapped in the past and doesn’t serve our customers.


·         TPC-E isn’t that cheap.  In fact, TPC-E is substantially less expensive to configure and run than TPC-C.  Two results from IBM within the last month prove the point.  As you can see in the table below, running on the same server, the TPC-C configuration was more than five times more expensive than the TPC-E configuration.  Further, on the four proc server, the TPC-C result had 1361 disks with no data protection, while the TPC-E result had 400 disks with RAID-5.  Which is the more customer-relevant configuration?





IBM System x3850 M2

IBM System x3850 M2

Procs / Cores / Threads

4 / 24 / 24

4 / 24 / 24


684,508 tpmC

729 tpsE


2.58 $/tpmC

457 $/tpsE

Total System Cost

$ 1,763,438

$ 333,646

Publication Date



Availability Date




256 GB

128 GB


1,344 x 73.4GB disks

16 x 500GB disks
1 x 73GB

400 x 73.4GB disks

Data Storage Protection



TPC Result Details



·         Customers continue to desire and reference TPC-C results.”  Granted, TPC-C has stood the test of time.  But today it is outdated, over-optimized, and of questionable relevance.  Customers hold onto TPC-C because it is familiar and available, not because it is better.  Database vendors need to exercise leadership.  As Mr. Paulley says “Microsoft is an early adopter of TPC-E”.  At this point, though, the early adopter window has passed.  TPC-E was ratified 20 months ago.  The first result was published 15 months ago.  There are 18 published results.   We believe that customers will readily embrace TPC-E as a superior benchmark as more results become available.


The more time that goes by, the more one is inclined to believe that Brian Moran is right – other database vendors aren’t publishing because they can’t beat the existing SQL Server results.  We invite Sybase and Mr. Paulley to prove us wrong.  We are confident that once Sybase runs TPC-E instead of just writing about it, Mr. Paulley will gain a new appreciation for just how challenging and technically rigorous TPC-E is compared with TPC-C. 


Charles Levine

SQL Server Performance Engineering

Comments (6)

  1. kirchner says:

    I totally agree with you.

    TPC-C is too old and proves nothing but the fact we can attach about 30 thousand disks to a single server. It’s a very unrealistic configuration and absolutely won’t reflect our real world.

    It may be the fact that other vendors can’t beat SQL Server (after all, what’s faster and better than SQL Server apart from itself???), but I personally believe that people are still giving too much credit on TPC-C.

    While customers don’t realize TPC-E is FAR BETTER and start questioning "why my database vendor has not published any results?", vendors won’t feel the pressure and won’t leave their confort place.

  2. ilovedatabases says:

    Charles, we’ve met – you’re a smart guy..I’m somewhat shocked at the posting. But I commend you on the work you are doing on TPC-E – indeed, it will become the standard, when it becomes the de-facto standard. And yes, I grow tiresome of all the disks in TPC-C too. And I work on all databases….but to make it interesting, the community has to support it.

    (1) One thing about Mr. Paulley is that indeed he is hitting the point home about an optimizer. Indeed, not everyone writes SQL that is succintly aware of the underlying engine like folks in the MS R&D lab do. So to me, all those developers out there should write a T-SQL statement and the database should do what it can to rewrite it and make it faster. Having a guy on your performance team write fast SQL for his database isn’t that impressive to me…. cause I can’t hire him.

    (2) What about the quesiton why isn’t Microsoft also pursuing TPC-C with TPC-E? In fact, after saying no TPC-C for SS08, MS released a TPC-C for 05. AFTER. We’re talking in the last couple of month on a hexa core. If it is so bad, why did you folks do it? If you look at the per core output of top results, MS has long had some issues in TPC-C area. In fact, I seem to even recall an apples to apples comparison a # of years ago with a competitor on the windows platform and same hardwrae (very rare) that beat SQL Server 2005 by 15%! So why the bi-polar TPC-C and TPC-E if this is so true? MS was never going to find a winning per core sustainable mark cause of platforms like POWER and thus WISELY invested early in TPC-E for good marketing…but when it all catches up, what is the answer.

    (3) You compare TPC-C and TPC-E in a chart — but notty notty – you can’t do that.

    (4) You talk about a non-RAIDed TPC-C result (and that is sneaky I admit) but somehow missed that MS’s biggest 1 TB TPC-H result for SS05 launch didn’t raid the disks either.

    (5)You make it sound like MS doesn’t play tricks in TPC-C such as pricing in a single call for support, turning off statistics collection, and some vendors even turn off integrity checks of the page. The point is MS or Oracle when they publish have all kinds of games no one ever does in the real world; if I pull the full disclosure for TPC-E will I find some there (I haven’t, but don’t make me read 400 pages for

    (6) Why do I like TPC-C — NOT because of the results. Because all things being equal, it stresses out the database logger. SQL Server 2005 logs almost 3 times more KB than some other solutions per TX in TPC-C; I’m not sure that is a metric dislosed in TPC-E (it wouldn’t be that great since the SQL wouldn’t be the same) but it certainly tells me something when everyone is doing the same thing.

    Now I do agree with you at the end, any benchmark publication is a lot of work, and you tune towards it. Do you really believe Teradata can’t scale since they don’t publish TPC-H? They feel it a problem solved (I say they don’t like to disclose costs)

    So you say vendors can’t beat existing results; is that they will never or haven’t spent the time investing in that area and when they do they won’t be able to.

    It’s dangerous words for a person of your reputation. TPC-C still has some validity (logger stress on equal footing) and TPC-E has some great modern stuff to.

    But I’d prefer a non attempt at marketing I’m a developer post 🙂

    As for Kirchener above to say nothing is faster than SQL Server — well..I won’t waste the characters on that one. I like SQL Server ,please don’t get it wrong. I don’t like when smart people step into the foggy world of FUD- I would expect this from Oracle!

  3. databasemysterio says:

    I like to hear more on TPC-E — but I agree with above on FACTS not FUD. BOTH MS and IBM (referenced above) use RAID0 in TPC-C so I find the comment disingenuous.

    AS well, TPC-C isn’t just too many disk test — above someone hits on loggings, it’s also SMP scale-up, concurrency, memory management, and more. So it that regards, even if the #s don’t impress you, looking at that stuff does.

    Finally, solid state drives are going to change disk layouts.

    But the biggest reason I’d like MS to post TPC-C is that Oracle still does it and is planning another one from what I see in blogs.

    Either way, I’m cool – I just like GOOD information — not marketing FUD from R&D — let marketing do that I already bought Vista.

  4. glennpaulley says:

    I’ve posted a response to your remarks on my blog.

  5. Len Wyatt says:

    Raid vs. non-RAID

    Ilovedatabases implies that it is disingenuous on my part to criticize TPC-C results for using non-RAID protected storage while at the same time having TPC-H results that use unprotected storage.  This misses my point, which is that it is the benchmark specification itself that establishes what level of storage protection is required.  TPC-E requires storage protection; TPC-C and TPC-H do not.  It’s perfectly fair for vendors to play to the minimum requirements of the benchmark.  The problem is when those minimum requirements are flawed or outdated.  Real customers don’t run mission critical OLTP systems with unprotected storage.  And yet that is what TPC-C requires and how everyone runs it.  TPC-H is also outdated in this regard, but there’s not an alternative benchmark in the DW space.

    Playing by the Rules

    Ilovedatabases says everybody plays tricks when they run the benchmarks.  Of course, one man’s trick is another man’s feature.  Everyone tries to satisfy the requirements, but nothing more.  Over time, everyone gets closer and closer to the edge.  This is why it is important to move the requirements higher.  The only way to do that is by defining a new benchmark, e.g., TPC-E.

    Fighting the Last Battle

    Ilovedatabases says he (she?) likes TPC-C because it stresses the transaction log.  TPC-A stressed logging even more than TPC-C does.  You had to do write ahead logging (WAL) to be competitive.  Anyone who didn’t already have WAL implemented it.  But that’s water under the bridge.  Nobody is going to take it out.  So it’s not a good reason, IMHO, to favor TPC-C (or TPC-A).  

    TPC-C results still being done

    Simply looking at the number of published results on TPC-C versus TPC-E doesn’t tell the full story.  (In reference to Glenn Paulley’s response.)

    Microsoft doesn’t directly publish TPC benchmark results.   Our hardware partners (e.g., HP, Dell, IBM, Unisys, NEC, etc.) publish results.  We provide them a kit and support.  Our level of direct involvement in publication of any particular benchmark result can vary greatly from no involvement whatsoever to actively working with the partner throughout the benchmark process.  

    At the launch of SQL Server 2008 (in February 2008), Microsoft announced that we are shifting our focus to TPC-E and away from TPC-C.  We have not and will not authorize any TPC-C results on SQL Server 2008 or later versions.  To accommodate our partners’ needs, we have allowed them to continue publishing TPC-C results using SQL Server 2005, while encouraging them to transition to TPC-E.  All of the TPC-C results using SQL Server 2005 published this year have been run entirely by the hardware sponsors with no involvement or encouragement from Microsoft.  

    There are more TPC-E results published year-to-date than TPC-C, even though TPC-C results span all database vendors while TPC-E results are (so far) only on SQL Server.   I expect this trend to continue with TPC-C results becoming fewer and further apart as the value of legacy comparisons declines over time.

      -Charles Levine

  6. pwehland says:

    Here we are over 3 years later, and still no RDBMS vendors publishing TPC-E numbers except SQL Server!  I suspect that the other vendors have run the TPC-E benchmark but have come up short, and then decided not to publish.