Programmers love to see benchmarks and statistics. Admit it! You love it as much as I do. Show us a table of figures or a chart and we go ga-ga.
Who cares that these “authoritative” benchmarks often measure incomplete implementations or fail to re-measure their data when presented with red flags such as the idea that Rails 1.2 is 2-4 times slower than 1.1.6. Or that most omit one of the most basic of statistical measurements.
Many of these are presented innocently and without the intent to deceive but they aren’t models for meaningful benchmarking. Most of the time you’re not looking for big numbers and shocking statistics, you just need to find out which implementation is faster.
If I fragment cache my tag cloud, will it really make the site any faster? Are memcached sessions across the network really any faster than ActiveRecord sessions?
Most of the time this means benchmarking your entire Rails stack: webserver, app server, Rails, database, filesystem. One of the best tools for benchmarking a full stack is httperf. It has a pretty good set of features and it gives you statistically useful information.
I’ve spent the last few weeks picking Zed Shaw’s brain and learning all I can about httperf. I’ve put all that information into the latest PeepCode. It starts with an episode on Page, Action, and Fragment Caching, but if you are already familiar with caching you can jump straight to the main course and view the httperf episode.
More and more I’m finding that the best way I can communicate is through a screencast, but here are a few tips that have helped me:
Reply rate [replies/s]: min 100.6 avg 101.0 max 101.4 stddev 0.6 (2 samples) Reply rate [replies/s]: min 100.6 avg 100.7 max 100.8 stddev 0.1 (2 samples) Reply rate [replies/s]: min 94.0 avg 96.4 max 98.8 stddev 3.4 (2 samples) Reply rate [replies/s]: min 103.8 avg 104.4 max 105.0 stddev 0.9 (2 samples)
Here’s a script that will parse a text file full of “Reply rate” lines and give you a Gruff graph as seen above.
I’m traveling a bit in March. If you’re at any of these places in the next few weeks, find me and I’ll give you a PeepCode t-shirt (limited to availability):
If not, you can buy a PeepCode t-shirt at my Shopify store (USA only for the first set).
And a free episode for the first 10 people to figure out how to use this coupon code: PEEP-NUBY
(The coupons were fetched up snappily in about 15 minutes.)