townx - Ruby Tuesday: what I've learned about Rails and memcache - Comments

Hi Manoel. I have to confess

elliot — Wed, 16 May 2007 10:37:51 -0500

Hi Manoel. I have to confess to being at a loss here: without being in front of your set up it's hard to be able to see what's going on. You're using memcache-client (I take it), which I never messed with too much. There have also been a couple more versions since I did this testing, and I haven't done much with it since. Also looks like you're doing fragment caching to memcache too, which I never looked into. So I don't think I can be much help.

One thing struck me, which is that :c_threshold => 10_000 looks a bit suspect. Should the value be an integer?

My other suggestion would be to strip the configuration down to its absolute minimum and get that working: the set up you have here looks quite complex. I'd also suggest trying it with a non-memcached session store and making sure that works with Mongrel correctly. Might also be worth trying Ruby-Memcache (though I haven't looked at that for a while and am not sure if it is still a live project). And finally I had my memcache config. outside the Rails initializer block. I wonder if this has any effect?

Mongrel Cluster & Memcached Sessions Issues: session is lost

Manoel Lemos — Tue, 15 May 2007 14:13:59 -0500

Gents,

I'm having very strange issues in my setup:

front-end: 1 x apache + mod_proxy_balancer
app servers: 12 x mongrels
session store: 1 x memcached (512MB)

I configured both the local session date (stored on memcached) for 29
(seems that it cant be more than 30 days, this is a limitation of
memcached) days of expiration and the session cookies for one year of
expiration but my users still have to login again and again in very
short periods of time.

My environment.rb has:


Rails::Initializer.run do |config|
  # Memcache Configuration
  memcache_options = {  :c_threshold => 10_000,
                        :compression => true,
                        :debug => false,
                        :namespace => 'mysite.com',
                        :readonly => false,
                        :urlencode => false }

  CACHE = MemCache.new(memcache_options)
  CACHE.servers = '127.0.0.1:10001'

  # Cache Storage Configuration
  config.action_controller.fragment_cache_store = CACHE, {}

  # Session Storage Configuration
  session_options = {   :cache => CACHE,
                        :session_key => '_bbsession',
                        :session_domain => '.mysite.com',
                        :session_expires => 3.months.from_now,
                        :expires => 29.days }
  config.action_controller.session_store = :mem_cache_store
  ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update(session_options)

end

More, I tried to debug that and I could prove that after
leaving/closing the browser and opening it again (not entering the site
yet) both the local (memcached) and remote (cookies) session data is
available, but mongrels don't find them and then the user needs to login
again.

I'm wondering if this is related to the load balancing or something
else.

Any ideas, suggestions?

Manoel Lemos
Computer Engineer
Email: manoel@lemos.net

Hello Sid. A few

elliot — Fri, 23 Mar 2007 11:23:00 -0500

Hello Sid. A few thoughts:

I'd turn off memcache and check it works off the filesystem, first. Have you got it working without memcache?
I'm not familiar with this construction: (params['user']['username'][%r{[a-z]}i]).nil?. What does it do in the context of this script?
What does @user contain by the time you get to redirect_to(:action=>"home")? Has it definitely been set with a value?
What comes back from session['user'] when you run find_user?
If you try to do a simple insert into session, then read out and reset (as per the example in my blog post), does that work? If it does, it looks like a problem with your script, not memcache or Ruby-memcache.

It could be that the redirect works but that @user is empty. Though I've had another closer look and that seems impossible: you only get redirected to home if @user is not nil, so presumably it has some value, so it should be getting stored in the session. Mmm, perplexing!

Ruby-MemCache

Sid — Thu, 22 Mar 2007 10:43:08 -0500

Elliot,

Sure!! I have a :before_filter on :home which points to :session_auth

The code:
require "digest/sha1"
def signin
flag=0
flag=1 if (params['user']['username'][%r{[a-z]}i]).nil?

#If flag is set to 1 then match phone number else username
if flag==1
@user = User.find(:first, :conditions=>["phoneno = ? and hashed_password=?", params['user']['username'], Digest::SHA1.hexdigest(params['user']['password'])])
else
@user = User.find(:first, :conditions=>["username = ? and hashed_password=?", params['user']['username'], Digest::SHA1.hexdigest(params['user']['password'])])
end
if @user.nil?
flash[:notice] = "Sign In Failed!"
redirect_to(:action=>"login")
else
session['user'] = @user
redirect_to(:action=>"home")
end
end

def find_user
session['user']
end

def session_auth
@user = find_user
if @user.nil?
flash[:notice]="Your Session Has Expired!"
redirect_to(:controller=>"top", :action=>"error")
end
end

In the :session_auth function the session is completely blank. I removed the filter and let the redirect continue to :home but the session was still blank. Other than that the before and after example on the main page works perfectly.
I even thought that maybe putting an object into the session variable isn't such a good idea, simple strings also get flushed :(
Sid.

Hello Sid. Without seeing

elliot — Thu, 22 Mar 2007 10:15:00 -0500

Hello Sid. Without seeing the specifics of the code, it's hard to debug. If you've got a code snippet I can try out myself (providing your application isn't so horrendously complicated you can't send one) I'd be happy to have a look over it. Have you made sure it's not the session['logged_in'] vs. session[:logged_in] thing, or something like it? Can you see what's happening on the client and inside memcache? Do the sessions work without memcache?

Ruby-MemCache

Sid — Thu, 22 Mar 2007 06:44:02 -0500

Hi,

I'm using Ruby-MemCache and I have the same problem. It's not just about filters though, a redirect_to somehow manages to come up with a blank session. Any thoughts?
cheers,
Sid.

Glad you sorted it. Comments

elliot — Fri, 09 Mar 2007 03:46:41 -0600

Glad you sorted it. Comments are always welcome, even though it's not a forum. I've had that string vs. symbol before. Most irritating.

uhh... never mind

Guest — Thu, 08 Mar 2007 15:32:36 -0600

i know this isnt a forum, but the issue was sybols vs. strings with memcache-client. session[;user_id] is not something that memcache can sustain... session['user_id']. Lame.

sessions breaking when logging in to private area

Guest — Thu, 08 Mar 2007 14:59:56 -0600

I was wondering if you have encountered memcahced not being able to maintain a logged in status in a simple admin/private area. I have not changed my code from moving from files to memcache, your tutorial code works, but i am getting kicked out of my before_filer :make_sure_logged_in area. Thanks in advance.

Ruby Tuesday: what I've learned about Rails and memcache

elliot — Tue, 03 Oct 2006 04:11:53 -0500

memcached is the cool new thing people are using to store Rails sessions and cache models. To quote the website:

"memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

What that basically means is that you can run memcached on one or multiple servers, and they are bundled together to present one big key store you can write values into or read values out of. The key store is volatile, in that it sits in memory: if all the memcache servers fall over, all the data stored in them is lost. But this is fine for caching and sessions. In the case of Rails' use of memcache, the session object itself is marshalled and written into the key store when a session value is added or updated; and the same key is used to retrieve the session data.

The main reason for using memcached is so you have a single session store which can be written to and read from multiple Mongrel instances. If you're running a Mongrel cluster, this means a client can be served by any member of the cluster, and all the Mongrel instances have access to the client's session.

In the rest of this article, I'll go through:

How to install memcached (the daemon which runs memcache instances) on Ubuntu Linux
How to install the Ruby libraries for talking to memcached
How to configure your Rails application to use memcached for sessions
Some benchmarking I've done on the various Ruby memcache libraries
A plugin I've written to help an app. recover if memcached disappears

I won't be covering:

How to configure memcached as a service
How to cache models into memcached
How to secure memcached and mongrel
How to setup a Mongrel cluster (I might cover this another day, but not now)

memcached installation

memcached is packaged for Ubuntu Breezy, so I just did:

apt-get install memcached

I then started a memcached instance as non-root with:

memcached -d -l 127.0.0.1 -p 17898 -m 256 -P /tmp/memcached.pid

The flags mean:

-d = run as a daemon
-l <ip address> = bind to ip address
-p <port> = run on this port
-m <num> = use num Mb of memory for the store
-P = where to put the pid file

When you're debugging, it's useful to use this variant instead, which makes it extra verbose and doesn't daemonize:

memcached -vv -l 127.0.0.1 -p 17898 -m 256 -P /tmp/memcached.pid

Ruby memcached bindings installation

There is some debate about which of the Ruby libraries is best for use with memcached. I've formed my own opinion after some very basic testing, but I'll explain both before we get to that.

The two choices are:

memcache-client from http://rubyforge.org/projects/rctools/. This is the one everyone says is fastest. It is quite basic and difficult to debug.
Ruby-MemCache from http://www.deveiate.org/projects/RMemCache/. This is older but (to my mind) has better error correction and debugging information.

Installation can be done from a gem. For memcache-client:

gem install memcache-client -v 1.0.3

Note that I've specified this version because the current version (1.1.0) doesn't appear to work properly on my machine. This is because the threading code has been substantially rewritten, but some of the checks for whether to use multithreading have not been applied correctly (see this bug report). So I'd stick with version 1.0.3 if I were you.

For Ruby-MemCache:

gem install Ruby-MemCache

By the way, I wouldn't recommend installing both at the same time: they appear to tread all over each other's namespaces, so I never felt confident that I knew which was being loading when I did my testing.

Setting up a Rails testing application

Regardless of which memcached client library you are using, the Rails configuration is the same, and is done in environment.rb. For testing, I set up a new Rails project called mem:

rails mem

I then created a controller for my testing:

cd gem
script/generate controller Sess

And put this content into app/controllers/sess_controller.rb:

class SessController < ApplicationController
  def index
    @before_id = session['id']
    session['id'] = Time.now
  end
end

This tries to read something out of the session, then resets it. This enables you to test both read and write operations.

I then added a view in app/views/sess/index.rhtml to show what's in the session:

<p>Before: <%= @before_id %></p>
<p>After: <%= session['id'] %></p>

Remember this isn't a brilliantly-written application with valid HTML, just a piece of junk to test sessions with.

I then edited environment.rb so that ActiveRecord doesn't load (I'm not going to be using a database, as this just complicates things). Find this line:

#config.frameworks -= [ :action_web_service, :action_mailer ]

and edit it to look like this instead:

config.frameworks -= [ :active_record, :action_web_service, :action_mailer ]

Now we're ready to test whether sessions work in the default fashion, and check our application is OK:

mongrel_rails start -e production

Notice that I've used the production environment throughout as this speeds things up somewhat. Now browse to http://localhost:3000/sess/ and you should see something like this:

The top line is blank as there's nothing in the session, and we're trying to print something from it. Refresh the page and you should get the value set on the previous request. This is how you know sessions are working.

Switching sessions to use memcached

Now we're going to switch session management over to memcached, instead of the default file-based sessions system. Let's get memcached started in debug mode:

$ memcached -vv -l 127.0.0.1 -p 17898 -m 256 -P /tmp/memcached.pid
<3 server listening

Notice the second line, which shows memcached listening on the specified port.

Stop the Rails application, then edit environment.rb. First find this line:

# config.action_controller.session_store = :active_record_store

and edit it to look like this:

config.action_controller.session_store = :mem_cache_store

Next, add this line to the top of the file (before the Rails::Initializer.run do |config| line):

require_gem 'Ruby-MemCache'

I'm using Ruby-MemCache here, but you could do require_gem 'memcache-client' if you prefer. I require the gems rather than the libraries because this makes it clear which library I'm using.

Then add these lines at the bottom:

memcache_options = {
   :compression => true,
   :debug => false,
   :namespace => "mem-#{RAILS_ENV}",
   :readonly => false,
   :urlencode => false
}

memcache_servers = [ '127.0.0.1:17898' ]

CACHE = MemCache.new(memcache_options)
CACHE.servers = memcache_servers
ActionController::Base.session_options[:cache] = CACHE

The memcache_options are reasonably self-explanatory, but just for clarity:

compression: use compression when communicating with memcached
namespace: store keys for this application inside this "chunk" of the memcache
readonly: allow writes into the memcache

The memcache_servers array should list all of the host/IP address combinations for your memcache servers. We've only got one on localhost, hence our setting.

The last three lines make the new memcache instance available to our controllers.

The moment of truth

Restart your application in a new console (remember to leave memcached running):

mongrel_rails start -e production

Browse to http://localhost:3000/sess/. You should see something similar to last time (see the image above). Refresh the page, and you should see that both the Before and After paragraphs include a time.

Now check your memcached instance in the other console window, and you should see something like this:

<7 new client connection
<7 get mem-production:session:0121026304505352eceeb4d82d849433
>7 END
<7 set mem-production:session:0121026304505352eceeb4d82d849433 1 0 81
>7 STORED

This indicates that Rails is correctly storing session data into memcached. Hurrah!

Benchmarking

I was quite interested in the suppose performance differences between memcache-client and Ruby-MemCache. Bear in mind the application is a toy one, it doesn't have a database, and is very simple. (Though these could be considered strengths, as it's taking a lot out of the equation.) Anyway, I ran some testing on the two different libraries, as shown in the table below. Here's my setup:

Mongrel 0.3.13.4
Rails 1.1.6
memcached 1.1.12
Ruby-MemCache 0.0.1
memcache-client 1.0.3

I ran the application in production mode, using one of the two libraries; I used no debugging on Mongrel or on the memcache libraries. I used ab for the benchmarking, with this command:

ab -n2000 -c20  http://localhost:3000/sess/

(2000 requests, concurrency of 20.)

Here are the results:

Library	Total time for 2000 requests (seconds)	Mean time per request (ms)	Requests per second
Ruby-MemCache	36.310	18.155	55.08
memcache-client	31.787	15.894	62.92

In both cases, all 2000 requests were served without error. You can see that memcache-client is faster, but not much.

And here my curiosity got the better of me

I like breaking things. So I decided to see what happens if you switch off memcached: how does the Rails application respond? Can it carry on serving pages? Can it recover? What happens to sessions? I know it's unlikely that memcached would disappear completely from anyone's setup, as it is distributed, and likely to run across multiple machines. But what if the server network cable got unplugged or the card broke? Or you only had two memcached servers and they both broke simultaneously?

So I tried running my application, then switching memcached off. The results were quite interesting:

Rails with memcache-client just falls over. Once memcached is down, the connection is lost and isn't recovered. This causes the whole Rails application to crash and serve only 500 error pages.
Rails with Ruby-MemCache continues to serve the application, but sessions are irretrievably broken.

In both cases, if you restart memcached, the application can't pick up the connection again until you restart Mongrel. This also means that before you start your app., memcached must be running and available, otherwise your application won't start correctly. Of the two, Ruby-MemCache obviously offers the nicest end result, as at least it doesn't fall over.

Respawning connections to memcached

I decided a decent solution might be to monitor the memcached instances from Rails and ensure that there is at least one available. If none is available, attempt to respawn the connection; keep doing this until it is re-established. This adds some overhead (checking memcached is OK, respawning if not), but could be worth it where memcached is not highly distributed, or the network is unpredictable.

To this end, I've bundled together a plugin which will monitor memcached. While Ruby-MemCache monitors memcached servers, memcache-client doesn't. Consequently, this code only works with Ruby-MemCache.

To use it:

Unzip the zip file below and drop the resulting folder into vendor/plugins.
Edit environment.rb and change the bottom part of the file (where you set up MemCache) to this instead:

MEMCACHE_OPTIONS = {
   :compression => true,
   :debug => false,
   :namespace => "mem-#{RAILS_ENV}",
   :readonly => false,
   :urlencode => false
}

MEMCACHE_SERVERS = [ '127.0.0.1:17898' ]

CACHE = MemCache.new(MEMCACHE_OPTIONS)
CACHE.servers = MEMCACHE_SERVERS
MEMCACHE_MONITORING = true
ActionController::Base.session_options[:cache] = CACHE

Note I've changed the option settings into constants (so the MemCache options can be accessed by the plugin), and added a MEMCACHE_MONITORING variable, which toggles monitoring (set to false to disable the monitoring code). To test it:

Start memcached, then Mongrel.
Check your sessions are working and Rails can access memcached.
Stop memcached. Your sessions should now stop working, while the app. still gets served.
Restart memcached. Sessions should be re-established.

Enjoy! Any comments welcome as always.

The plugin adds some overhead, but not much: I'll put some stats. up shortly.