Programmers love to see benchmarks and statistics. Admit it! You love it as much as I do. Show us a table of figures or a chart and we go ga-ga.

Who cares that these “authoritative” benchmarks often measure incomplete implementations or fail to re-measure their data when presented with red flags such as the idea that Rails 1.2 is 2-4 times slower than 1.1.6. Or that most omit one of the most basic of statistical measurements.

Many of these are presented innocently and without the intent to deceive but they aren’t models for meaningful benchmarking. Most of the time you’re not looking for big numbers and shocking statistics, you just need to find out which implementation is faster.

If I fragment cache my tag cloud, will it really make the site any faster? Are memcached sessions across the network really any faster than ActiveRecord sessions?

Most of the time this means benchmarking your entire Rails stack: webserver, app server, Rails, database, filesystem. One of the best tools for benchmarking a full stack is httperf. It has a pretty good set of features and it gives you statistically useful information.

Learn the Basics

I’ve spent the last few weeks picking Zed Shaw’s brain and learning all I can about httperf. I’ve put all that information into the latest PeepCode. It starts with an episode on Page, Action, and Fragment Caching, but if you are already familiar with caching you can jump straight to the main course and view the httperf episode.

More and more I’m finding that the best way I can communicate is through a screencast, but here are a few tips that have helped me:

• You can’t do benchmarking without knowing some basic statistics. And no, understanding the concept of “average” isn’t enough! Learn about what standard deviation is and what it says about the data you pulled from your benchmarking run. No server performs at exactly 32.746 requests per second…there’s a range of error and if you don’t express your benchmark along with that range then you haven’t communicated anything. It isn’t very hard to understand and the benefit to your powers of evaluation will be huge. If nothing else, you’ll be able to properly criticize poorly executed benchmarks on other blogs! See also Zed’s rant.

• Learn how to read the output of your benchmarking tool of choice. Even ab gives you some results with standard deviation, but it’s not often used. For httperf, it’s the “Reply rate” line. Here are a few from separate runs against the same server running two kinds of software:

Reply rate [replies/s]: min 100.6 avg 101.0 max 101.4 stddev 0.6 (2 samples)
Reply rate [replies/s]: min 100.6 avg 100.7 max 100.8 stddev 0.1 (2 samples)
Reply rate [replies/s]: min 94.0 avg 96.4 max 98.8 stddev 3.4 (2 samples)
Reply rate [replies/s]: min 103.8 avg 104.4 max 105.0 stddev 0.9 (2 samples)

• Be a scientist. Running a benchmark should be done with the same care that a chemist measures a blood sample for a trace of a fatal disease. Quit all the other programs you have running. Use two separate computers so your benchmarking tool doesn’t compete with your web application for the CPU.

Here’s a script that will parse a text file full of “Reply rate” lines and give you a Gruff graph as seen above.

Calendar

I’m traveling a bit in March. If you’re at any of these places in the next few weeks, find me and I’ll give you a PeepCode t-shirt (limited to availability):

• March 10-13: SXSW in Austin, Texas
• March 13: Seattle.rb lectures at the Northwest Film Forum at 7pm
• March 18-26: Kauai, Hawaii (Is there a Ruby brigade in Kauai?)
• March 27-29: Philly Emerging Tech Conference

If not, you can buy a PeepCode t-shirt at my Shopify store (USA only for the first set).

Resources

• Gregg Pollack on Page Caching
• PeepCode: Page, Action, and Fragment Caching
• PeepCode: Benchmarking with httperf

And a free episode for the first 10 people to figure out how to use this coupon code: PEEP-NUBY

(The coupons were fetched up snappily in about 15 minutes.)