Cheaters Never Win, Even in TPC Benchmarks

In this column, I want to tell you about one of my favorite aspects of the TPC benchmarks – CHEATING. Keep in mind that I use the term “cheating” in a joking manner with my tongue planted firmly in my cheek.  But I’m also half-serious.  One of the of the things that is great about the TPC benchmarks is that each of the vendors are required to fully describe all of the shortcuts, tweaks, and special operating configurations they use in order to achieve their spectacular performance numbers.  In a sense, the Transaction Processing Counsel requires that all benchmarked platforms declare all of the ways that they cheat in order to achieve peak performance.

read more

Many thanks,

-Kevin

- Follow me on Twitter!

- Google Author

Share and Enjoy:
  • Print
  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Yahoo! Buzz
  • Twitter
  • Google Bookmarks
  • email
  • LinkedIn
  • Ping.fm
  • HackerNews
  • Posterous
  • Reddit
  • Technorati
  • Tumblr

Comments

  1. Oracle should probably get a special mention for ‘best abuse of the rules’.

    ” In April 1993 the Standish Group claimed Oracle had incorporated an extra “discrete transaction” option into its product that would let it perform particularly well against the TCP-A benchmark but offered no noticeable benefit to customers. Standish opined that, by adding this option, Oracle was putting forward for testing a “benchmark special” and that in doing so the company had violated the spirit of the TPC.” http://www.theregister.co.uk/2007/11/12/oracle_tpc_benchmarks/

    I’ve heard that because the TPC benchmark queries were known in advance, Oracle’s ‘tweaked’ query processor would first check that the input query matched one of the benchmark queries. On a match, the processor would quickly choose a precalculated plan that was already known to be the best plan the engine could offer.

    If it’s true, this would actually slow down the processor for any real world query whose best plan is not known in advance, since the processor would do the useless TPC-checking operation before the real search.

  2. Great point, Iain. In fact, many database vendors also claim that simple adherence to the TPC-C standard as Oracle does for transaction processing workloads is enough to warranty marking them as a cheater.

    The TPC-C benchmark is exceedingly simplistic and doesn’t represent the transactional workloads of most enterprises. Consequently, many vendors (Microsoft e.g.) use TPC-E as a more realistic and sophisticated transaction processing workload.

    There’s a similar gap in credibility between the TCP-D and TPC-H business intelligence workloads. When a vendor sticks with an older and less credible benchmark, it erodes rather than builds credibility in my case.

    Thanks for adding to the conversation!

    -Kev

Speak Your Mind

*