memcached Basics for Rails 40 comments
Many speedy sites use memcached to save the results of expensive database queries and intense rendered templates. This is a basic introduction to using memcached with Rails. Thanks to Eric Hodel and Layton Wedgeworth who have answered many questions.
Yes, there is a hash in the sky
Memcached is a lightweight server process that stakes out a fixed amount of memory and makes it available as a quick access object cache.
Some of the things you can do with it are:
- Automatically cache a row from the database as an ActiveRecord object
- Explicitly
render_to_stringand save the results (works well for tag clouds) - Manually store a complicated database query and retrieve it later
I thought there was some kind of fancy voodoo happening, but it turns out that it’s basically just a hash! You can put a key and value into the cache, and get it later. memcache-client also gives you a delete method.
Objects are serialized using Marshal, so it’s very fast.
@tags = Tag.find :all Cache.put 'all_your_tags', @tags Cache.put 'favorite_skateboarder', 'Tom Penny' skateboarder = Cache.get 'favorite_skateboarder' Cache.delete 'all_your_tags'
Installation on Mac OS X
I first installed the memcached server using DarwinPorts but each query was taking five seconds to answer. It turns out that this can be fixed. I wrote a shell script that builds memcached and dependencies with the proper fixes. Use at your own risk…
memcached Install Script for Mac OS X
Other platforms can find the source or an installer at the memcached site.
Install cached_model
The Robot Co-Op has made a few libraries available. cached_model makes it easy to cache single row queries from ActiveRecord tables. memcache-client is a pure Ruby client and is included with the installation of cached_model.
sudo gem install cached_model
cached_model requires Rails 1.1.2 or greater.
Start the memcached server
The memcached server takes several options but the defaults work well to start with. While learning, the -vv argument shows what’s happening as values are entered and retrieved.
# Start the server with the default of 64 MB RAM memcached -vv
Use with ActiveRecord
The easiest way to experiment with memcached is to make a new Rails app with a single database table. Add the following code to the bottom of config/environment.rb:
require 'cached_model'
memcache_options = {
:c_threshold => 10_000,
:compression => true,
:debug => false,
:namespace => 'my_rails_app',
:readonly => false,
:urlencode => false
}
CACHE = MemCache.new memcache_options
CACHE.servers = 'localhost:11211'
NOTE: You should move this to config/environments/production.rb for production.
You’re almost ready. Edit your model so it inherits from CachedModel instead of ActiveRecord::Base.
class Foo < CachedModel end
Try it out
Finally, start ./script/console and watch the output of memcached -vv. It also helps to tail -f log/development.log.
Create a few records, find them by id, and watch as they are automatically placed in the cache. Finding the same record repeatedly should hit the cache, but not the database.
>> Foo.find 1 => MemCache Get (0.134546) active_record:Foo:1 => SELECT * FROM foos WHERE (foos.id = 1) LIMIT 1 => MemCache Set (0.024758) active_record:Foo:1 >> Foo.find 1 => MemCache Get (0.032337) active_record:Foo:1
If you save a record that is already in the cache, CachedModel will automatically update it in the database and in the cache.
CachedModel automatically creates unique keys based on the model name and id number. You can cache arbitrary data with the provided Cache object, but you will need to plan a consistent key system.
>> all_foo = Foo.find :all >> Cache.put 'Foo:all', all_foo => MemCache Set (0.082277) Foo:all >> Cache.get 'Foo:all' => [#<Foo:0x2642ad4 @attributes=...
A few things to remember
Cache.getwill returnnilif the item is not in the cache. You can use this to determine whether something should be refetched and re-inserted into the cache.- Records in
CachedModelexpire after 15 minutes (or you can adjust the expiration time by settingCachedModel.ttl). - Manually inserted objects can be expired, too.
Cache.put('foo', 'bar', 60)will cause thefookey to expire in sixty seconds. - When the allotted memory fills up, older items will be deleted.
Deployment details
Production use of memcached deserves its own article. In the meantime…
- Memcached is only useful for a dedicated server or a VPS that has sufficient memory available.
- Try running an instance of memcached on each of your app servers. Running with 128 MB is sufficient for most applications.
Other Resources
Nuby on Rails
Geoffrey Grosenbach / Ruby / Code / Graphics / Design / Rails / Merb / Javascript / CSS
The official Django documentation about caching with memcached is quite good—
http://www.djangoproject.com/documentation/cache/
Foo.find_all Is deprecated, and has been for quite some time.
Please use Foo.find :all.
Edge rails will warn you now, but there’s no harm having tutorials contain the right information.
A cute add-on would be to create a CacheHash class that works just like a regular hash, but does the Cache.put and Cache.get for you :) I use memcached all the time in Perl but such abstractions are pointless there, but in Ruby.. pretty things can happen.
It’s worth noting that you can get a good increase in performance from using memcached as your session store.
Memcached is only going to be useful for your db queries when displaying that same dataset mulitple times.
@koz: I’ve updated it to use Foo.find :all. Thanks for the note.
@Peter: I was thinking the same thing! I’ll work on that tomorrow and maybe Eric will put it into the library.
@Michael: Agreed, but the fact is that many sites do need to display the same data multiple times, so well-planned use of memcached can speed things up quite a bit.
I don’t think wget is installed on Macs by default.
Just do a find and replace “wget” to “curl -O” and it should work fine.
@peter: ever heard of tie’d hashes in perl?
Excellent article. Thank you kindly.
@Geoffrey
Totally agree, there are many cases where it’s extremely smart to be using memcached. You can see some fairly significant gains.
It certainly played a part in helping livejournal scale and that’s saying something!
Thanks for the run down, I’ll be needing to dig into to this very soon for my current project.
This is really a question for the Robot coop guys, but why make the cached model a class to extend instead of a module? Whether or not a model is cached should be an implementation detail, and shouldn’t define the hierarchy for a class. I know I would rather not use the power of (single!) inheritance just to cache something, when a mix-in should be plenty powerful to do it.
@Peter: CachedHash is not quite straightforward. Making #each work involves client-side tracking because there’s no way to ask memcached what keys exist in the server. It certainly won’t work reliably outside a single process. Without implementing #each its really easy.
@Eric Hodel: you’re right. I tried something like that in my PHP days (slurp!) and to do a foreach I needed to save my keys in memcached too. So basically you needed two hashes: one for the original hash (e.g. employees) and one for the keys of that hash (e.g. employees_keys). The second is just an array of the keys. Personaly if I were to work it out I would do an OrderedHash or OrderedOptions. I’m not a big fan of the messy hashes ;-)
Thank you much for this howto. Some of my more used apps are now memcached-friendly!
I wonder if it make sense to make a Mongrel plugin version of memcached. This would make installation a snap.
Thanks! I couldn’t manually patch this in my port sources for some reason, wasn’t taking. Your script did the trick, very nice!
<3
So much for textile.. that was: Cheers for the article geoff. Love it.
Thank you very much for this—its a nice and much appreciated intro to memcached.
I am a newbie bigtime, and this is a great article (oh, and the podcasts rock) but I don’t see the message when I use CachedModel based queries, such as Brain.find(:all) I DO see it when I explicity put something in the cached in the memcached output.
Any suggestions? All the models now inherit from CachedModel and the queries all return records, I just don’t have the warm, nougaty (sp?) center of confirmation in the memcached output.
CachedModel is for simple queries and will automatically take over when you do a simple search by id, such as:
my_record = MyModel.find params[:id]
my_record = MyModel.find 42
More complicated queries (like :all or :conditions) must be cached manually with the Cache.put and Cache.get methods.
Hmmm….well, when I do this:
tst = Person.find 31637
memcached gives me zero output. Then, when I do this:
Cache.put ‘ruprict’, tst
I get
So, I am either still messing it up (likely) or it isn’t caching the query or it is and it isn’t very forthcoming.
Thanks for the help.
@Rob: I’ve posted why CachedModel is a class on my blog.
NOTE: URL fixed by topfunky
cached_model is nice but it causes some weird behaviour in my unit tests. Its almost like data from previous runs is available to future runs, even though I don’t want it to be.
If it were a module, you could simply dynamically include it for production and development, but because you hard code the relationship its not as easy to disable it.
Is there a way to turn off cached_model so it does nothing when the environment is “test”? I didn’t see anything in the docs about this.
You can turn off CachedModel in your tests.
See the documentation or Eric’s comment here:
Ruprict: I get the same as you when running edge rails, but it works in 1.1.2
I made an app to test that I had memcached setup and working by following the instructions here and it worked fine.
When it came to updating my app, I used exactly the same settings for memcached and updated the models to use CachedModel, but when I deployed it doesn’t use memcached.
Updating the demo app to use edge rails broke the demo app – so looks like something in edge.
For what it’s worth, the latest version of memcached in DarwinPorts (1.2.1 as of writing) doesn’t appear to have the slowdown fixed.
Thanks for pointing out just how easy memcache actually is to use—I thought it would have been more effort!
I’m having the same problem as Ruprict. Has anyone found a solution?
@Mark: There was a tiny change in the latest version of ActiveRecord that changed the way SQL queries are constructed. I submitted a bug report and the tracker shows that it was fixed.
However, a new version of cached_model has not been released yet. See the details in this rubyforge ticket.
That memcached install script was supremely helpful; you saved me hours of work. Thanks a ton!
Glad to hear.
I recently installed the newest version of memcached on CentOS and Debian servers. There were some notes in the changelog that suggested they may have fixed the Mac OS X bug in the 1.2 release.
I wrote a minimal script to download the newest version and install it on Linux. You may have to twiddle with ldconfig to get it to load the libevent library correctly.
http://topfunky.net/svn/shovel/memcached/install-memcached-linux.sh
Guys, I’m new to the world of Ruby On Rails and i have a query. I run a price comparison service based on MySQL and XML and would like to take advantage of Ruby.
Could you advise of how i could get started? I would like to make the site more dynamic for the users firstly.
Regards,
Mike http://www.thepricesite.co.uk
I’ve noticed that MyModel.find(1) doesn’t interact with memcached as it should. It looks like CachedModel#find_by_sql filters out complex requests using a regexp:
I’m running Rails 1.2.3, which for Model.find(1) generates the sql:
Which does not match the above regexp. Has rails changed the sql it generates since this gem was last updated?
Also, is there a more appropriate place to post this?
Hmm, it seems I should read the above comments more closely. I’m changing the regexp in CachedModel#find_by_sql to
Bugs can be filed here:
http://seattlerb.rubyforge.org/memcache-client/
Pretty cool :-)
Brand new to memcache, planning on installing it later today. Anyways, in the beginning of the article you state its possible to automatically cache a row from the database as an ActiveRecord object, but later topfunky says that :all queries have to be manually entered.
Say a customer logs in, which would be customer.find(1…)
this would be automatically cached. Then if you pull in their shopping basket,
product.find(:all …..)
Are you saying its possible to cache that, you just need to add it to the cache yourself? And if you do so, does it remain as an AR object and respond accordingly?
@Bob: Yes, you would have to cache an :all query manually. In fact, I don’t use CachedModel at all since I’m most concerned about caching complicated queries. I do something like
class Product < ActiveRecord::Base def self.find_all_cached Cache.get("Product:find_all", 15.minutes) { find(:all) } end endIt gets more interesting when you can use arguments to form the cache key so the record expires automatically when any update happens.
Well done. One question: if we have
available to us, why do we then need the following dictatorial statement in CachedModel.find() that preempts use of memcache?
I realize a gentlemen above would like CachedModel not to interfere with his unit tests, but I’m one of those eccentrics that like his ‘test’ environment to mimic his ‘prod’ environment as closely as possible.
cheers
Nevermind. I see now this interface hasn’t been maintained in some time.
Hi I am getting the following error while trying to launch ruby script/server
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load—memcache_util (MissingSourceFile)
.. Any clues ? I have all the required gems installed (cached_model and memcache-client) I also downloaded a copy of memcache_util.rb from internet and placed it under gems/memcache-client-1.7.0/lib/ Still no luck! Any idea guys ? I have a deadline hanging over my head. Help please !