Apache mod_proxy balancing with PHP sticky sessions - Mark's blog

Apache mod_proxy balancing with PHP sticky sessions

I've been investigating the new improved mod_proxy in Apache 2.2.xfor use in our new production environment, and in particular thebuilt-in load balancing support. It was always possible to build aload-balanced proxy server with Apache before, using some mod_rewritevoodoo, but having a whole set of directives that do all the hard workfor you is a great feature.

There is however, a catch. It won'twork out of the box with PHP sessions, or many other applications. I'vesince worked out a way around this which enables you to continue usingall the great features mod_proxy_balancer offers and still bind requeststo an originating server. All you need is a little mod_rewrite magic :Read on for more details...

If you're going to use mod_proxy to build a reverse proxy (cachingor otherwise) in front of your backend application servers, you'reprobably going to want to use sticky sessions so that a user of yoursite always gets "bound" to the server that the session originated on.Otherwise, you'll get people randomly getting logged out and all mannerof other nastiness. According to the docs, you can use the "stickysession" key in your configuration :

The value is usually set to somethinglike JSESSIONID or PHPSESSIONID,and it depends on the backend application server that support sessions.

Sounds great, but it doesn't actually work that way with PHP. If you set your configuration to something like this :

ProxyPass / balancer://cluster/ lbmethod=byrequests \
stickysession=PHPSESSID failover=Off

You'llfind that regardless of whether a session exists or not, you'll stillend up randomly pinging between the backend servers, which is definitelyNOT what you want to happen. So why is this happening ? Well, it turnsout that the Apache documentation is actually a little misleading. Ifyou look at the relevant code in modules/proxy/mod_proxy_balancer.c,lines 195 to 210, you'll see it's actually looking for a session ID,then a period (.) character and a route. Tomcat can be configured to dothis, but PHP can't; PHP sessions can only be alphanumeric and don'tspecify an originating server or "route".

So, as a workaround, youcan create your own cookie and use that instead when binding requeststo a backend server. It does however mean that any request gets bound toa server,not just when sessions start but this shouldn't be too much ofan issue, particularly if you are caching static content as well. Aneasy way to do this on recent versions of Apache is to use mod_rewriteto set the cookie for you (this feature was only added in later versionsof 2.0.x and up - I recommend running 2.2.x to be certain all this willwork).

Say you have 2 backend servers, www1.example.com andwww2.example.com. You'd add the following to your backend vhostconfiguation :

 

RewriteEngine On
RewriteRule .* - [CO=BALANCEID:balancer.www1:.example.com]

 

Andthen do the same for www2, but obviously changing the cookie value toreflect this. You then need to tell your frontend proxy that it shouldlook for this cookie, and which server each "route" refers to :

ProxyPass / balancer://cluster/ lbmethod=byrequests stickysession=BALANCEID
ProxyPassReverse / balancer://cluster/

  BalancerMember http://www1.example.com route=www1
  BalancerMember http://www2.example.com route=www2

 

Andthen give it a kick. What you then should see (if you switch LogLevelto debug) is the following the first time a request comes in :

proxy: BALANCER: Found value (null) for stickysession BALANCEID
proxy: Entering byrequests for BALANCER (balancer://cluster)

So it doesn't have a preferred route, and it switches to the defaultload balancing algorithm (byrequests) to get a random server. Nexttime, the cookie will have been set, and you'll see :

proxy: BALANCER: Found value balancer.www1 for stickysession BALANCEID
proxy: BALANCER: Found route www1
proxy: BALANCER (balancer://cluster) worker (http://www1.example.com) 
rewritten to http://www1.example.com/

Andyou should then be good to go. Each new request that comes in will bedirected to a backend server according to your load-balancing method,and any subsequent requests from that user (assuming they have cookiesenabled) will then go back to the same backend server. When they closetheir browser and the cookie expires, the "binding" is reset and they'llget a new random server next time they connect.

Posted by Mark Round on Friday, October 27. 2006 at 11:53 in Sysadmin

TrackbacksTrackback specific URI for this entry
No Trackbacks
CommentsDisplay comments as(Linear | Threaded)
Axel-Stéphane SMORGRAV says,

Friday, March 23. 2007 at 08:38 (Reply)

Mark,

Thanks for putting me on to the right track. I ended up setting a cookieusing a randomised RewriteMap (mod_rewrite) on my reverse proxy.

-ascsBaldeep Hira says,

Tuesday, March 27. 2007 at 01:40 (Reply)

This is a naive questionbut how do I configure vhost to write the BALANCEID cookie? I have thefollowing code and cookie doesn't get written from within theVirtualHost block.

NameVirtualHost 10.1.1.115:8080
NameVirtualHost 10.1.1.116:8080
VirtualHost 10.1.1.115:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app1:.example.com:120:/app]
/VirtualHost
VirtualHost 10.1.1.116:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app2:.example.com:120:/app]
/VirtualHost


ProxyPass /app balancer://app_cluster/app stickysession=BALANCEID
ProxyPassReverse /app http://10.1.1.115:8080/app
ProxyPassReverse /app http://10.1.1.116:8080/app

Proxy balancer://app_cluster
BalancerMember http://10.1.1.115:8080 route=app1
BalancerMember http://10.1.1.116:8080 route=app2
/ProxyMark says,

Thursday, March 29. 2007 at 21:49 (Link)(Reply)

Just a quick idea (I can'tget access to a server to check right now), but have you tried using theactual hostname as part of your BalancerMember config ? You're settingthe cookie to be valid for the .example.com domain, but are requestingit via an IP address - I can't remember off the top of my head if Apachewill serve the cookie to you under those circumstances.

Anyway, I will have a look and check...Baldeep Hira says,

Friday, March 30. 2007 at 15:46 (Reply)

Thanks for the quick replyMark. If I move the rewrite rule outside the VirtualHost block thecookie gets set and I can see everything working fine. Of course allrequests are serviced by a single internal server in this case. I wantto intercept the request processing once the load balancer has chosen aninternal server from the cluster and add the cookie in that phase, notsure how BalancerMemeber interacts with VirtualHost at this point.Gjøran Sæther says,

Friday, March 30. 2007 at 11:31 (Reply)

Thanx a lot. this was very helpfullPaulo Roberto Delpizzo says,

Sunday, July 1. 2007 at 03:20 (Link)(Reply)

Hi All.

Mark, I have a similar problem here and I´ll appreciatte a lot if you help me.

I have two instances of ColdFusion running on the same server. As youknow, CF runs on the JRun and therefore, is an application java thathave session controlled by JSessionID.

Well, my problem consists as following. When I make login, I can´t logonin the application, because instance CF that validates login is not thesame one that it answers request of the initial page of theapplication. As the session does not exist for this second instance, Iam redirected again for the logon screen.

I use Apache 2.2x with mod_proxy_balancer. I think that this solve my problem. What is your opinion?

Regards,

Paulo RobertoDave says,

Wednesday, August 15. 2007 at 22:41 (Link)(Reply)

If you want two seperate instances to have access to the same sessions you will need to store them in the same place.

I'm not sure where ColdFusion configures this but in PHP you can specifythe session storage directory in the php.ini file. I suspectColdFusion has an equivalent in it's config files.

Alternatively, you could switch to storing sessions in a database. Thatwill come with it's own set of problems such as placing extra load onyour database server and potentially being slower than storing them onlocal disks but it will solve the problem of accessing a session frommultiple web servers.Naveenkanth says,

Thursday, January 31. 2008 at 19:17 (Reply)

Mark,

Thanks for the post.

Paulo,

JRun's JSessionID will not help for session stickiness here. So I wouldpropose writing an extra filter in your web application(in your case CF)and add a custom cookie in both the instances such as cookie.node1,cookie.node2 Then add this custom cookie name in thestickysession=${Custom_cookie}

This should help to maintain session stickiness in the balancer, though it will break in case of fail over.

-JRBeniji says,

Wednesday, September 5. 2007 at 16:20 (Reply)

Thanks for the guide (mightcome in handy one day). I can't help thinking though that if the PHPsite has sufficient traffic to warrant load balancing, perhaps it shouldmove to memcached for session storage (in RAM) and not be usingfilesystem based sessions (on magnetic disk) in the first place.

My experience with sticky balancing is "best to avoid it if possible"!Uwe says,

Friday, August 1. 2008 at 16:00 (Reply)

hi,

I checked this and it work so far.

But what are you doing, when a server fails? I mean completely.
So far I've got a so called healthcheck script which checks the BalancerMember and reconfigures the apache.
Now, the cookie is still set and the proxy sends the request to thefailed server. Do you have any idea how to get into that request andsend to another BalancerMember?

Best Regards,
UweHardayal Patil says,

Sunday, August 3. 2008 at 19:12 (Reply)

I am trying to use load_balancer for HTTP load balancing without using servlets or Tomcat. Has anyone tried this?

I have 5 prescription servers in the back-end that support HTTP tunneledrequests. They have an internal sessionid called CustSessID that iscoming from cookie. I want stickyness so that the requests from samebrowser go to the same Prescription Server. I also have URL rewritingthat connects to Generic server or BrandName Server.anil says,

Thursday, September 25. 2008 at 16:10 (Reply)

in your above examle you are using http like below.

BalancerMember http://www1.example.com
BalancerMember http://www2.example.com

in my case i have two apache as BalanceMember and they are listing at secure port 443.
so my links are

BalancerMember https://www1.example.com
BalancerMember https://www2.example.com

this is not working????? why??
more over in my real servers i am re writing the urls too.

and while i request my loadbalancer apache it just redirect my requestto real server and in my browser i see the address of my real server.which i dont like.

can anybdy help me.Scott says,

Thursday, October 2. 2008 at 13:59 (Reply)

My cookie value is getting URL encoded. In my log I see:

proxy: BALANCER: Found value balancer%2Ewww1 for stickysession BALANCEID

So the route is never getting
discovered.

I'm setting the cookie with an ASP page. Anyone know whats going on?Raj says,

Wednesday, October 15. 2008 at 09:54 (Reply)

Hi,

I have Problem in getting sticky session.
i used phpBB application to test sticky session.i have installed phpbb in two diffrent machine connected to diffrent database.

here is my config
_____________________________________
NameVirtualHost *

ServerName test
ServerAlias test
ProxyRequests Off


Order deny,allow
Allow from all


ProxyPass /balancer-manager !
ProxyPass / balancer://mycluster/ stickysession=BALANCEID nofailover=On
ProxyPassReverse / http://192.168.11.23/phpBB2
ProxyPassReverse / http://192.168.11.43/phpBB2

BalancerMember http://192.168.11.23/phpBB2 route=http1
BalancerMember http://192.168.11.43/phpBB2 route=http2
ProxySet lbmethod=byrequests



SetHandler balancer-manager

Order deny,allow
Allow from all



Every time i refresh the page it takes me to diffrent server and doesnot allow me to login, because it's checking session on bothe the server

Please help....Mark Round says,

Wednesday, October 15. 2008 at 11:01 (Reply)

Have you set the cookie correctly on the backend servers ? Can you see this cookie being set in your browser ?Travis says,

Monday, October 20. 2008 at 11:23 (Reply)

Great post. I got this working, however, I have run into a practical issue.

If you have Apache running as a load balancer with two web nodes, andthe user gets a cookie associating them with www1, there is an issuewhen www1 goes offline. It seems that Apache is smart enough to knowthat www1 is offline and requests should now go to www2, but that onlyhappens with fresh requests. The previous user with the cookie pointingto www1 is now just getting sent to the www1 server, which throws them aServer Unavailable message (via the load balancer). Any way aroundthis? I guess I would be looking for a way to check that a server is upbefore Apache uses that balance member. Thoughts?

-TravisMark Round says,

Monday, October 20. 2008 at 13:20 (Reply)

The only way I found to dothis (it would be great if Apache natively supported it!) is to monitorthe backend servers with something like Nagios. When a server goes down,an event handler removes that from the proxy pool and performs agraceful restart.

Alternatively, you could write a script that uses the balancer-manager page to dynamically add/remove backend servers.Brian Kearney says,

Thursday, October 30. 2008 at 05:18 (Link)(Reply)

I think Apache 2.2 does natively support what you are trying to accomplish.

I see that maxattempts and nofailover parameters to the ProxyPassdirective are described on this page of the HTTPD configuration site:

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

Does that look like what you need?Travis says,

Thursday, November 20. 2008 at 20:09 (Reply)

Hmm, this might work. Butthe problem is that Apache is just responding to what it gets from theuser's cookie value, which is set to the dead server. I need a way to dosomething different (i.e., go to the non-dead server) BEFORE theconnection attempt is made. If maxattempts is set to 1, it's still goingto try at least once on the bad server PER REQUEST, which will faileach time. Does that make sense (and am I correct?)

Since I wrote this post, I was also messing around with other loadbalancing options, so forgive me if my explanation is a bit incorrect,since I'm trying to put myself back into this configuration to fullyunderstand it...Miles Crawford says,

Tuesday, May 26. 2009 at 23:09 (Reply)

Just a heads-up of a common gotcha with this setup.

One common reason for using a reverse proxy is to enable caching. Bydefault, mod_cache records and replays the Set-Cookie header!

This means that you can wind up with a cached route, so that anyone whorequests the cached file will get assigned to the same backend host.

CacheIgnoreHeaders Set-Cookie clears this up.Richard Wiseman says,

Friday, February 5. 2010 at 10:16 (Reply)

Thanks for the article, it almost works for me!

Any idea why I might be getting:

proxy: BALANCER: Found value balancer.www8888 for stickysession balancerID
proxy: BALANCER: Found route www8888

followed immediately by:

proxy: Entering byrequests for BALANCER (balancer://cluster)

It appears to be ignoring the route=www888 in my BalancerMember entry.

Thanks!Richard Wiseman says,

Friday, February 5. 2010 at 10:49 (Reply)

Sorry, typo in my last post! It should read: "It appears to be ignoring the route=www8888 in my BalancerMember entry."Ali says,

Thursday, March 25. 2010 at 01:28 (Reply)

The last sentence of this article is incorrect, IMO:

"When they close their browser and the cookie expires, the "binding" isreset and they'll get a new random server next time they connect."

I manage an infrastructure that was built off of this page. The cookienever expires. A user is bound to a specific server forever. If thatserver dies, then the user is going to go to that dead server.

This is bad. I'm searching for a solution to this now.Mark Round says,

Thursday, March 25. 2010 at 09:00 (Reply)

Interesting. This is the complete opposite of the behaviour I experience - what browser are you using ?

Using Firefox (purely because it's got good developer tools like the WebDeveloper toolbar and Firebug that lat you inspect this sort ofbehaviour), when I visit a site that sets the cookie, I see thefollowing in my cookie store :

Name BALANCEID
Value balancer.www1
Host .example.com
Path /
Secure No
Expires At End Of Session

And it's set by the following header from the webserver :

Set-Cookie BALANCEID=balancer.www1; path=/; domain=.example.com

When I close my browser down completely and then restart it, I can checkmy cookie store and the cookie isn't there anymore. When I revisit thesite, my request headers do not send any cookies.

-MarkdAm2K says,

Tuesday, April 6. 2010 at 14:07 (Link)(Reply)

It's working OK for me.
Thank you.quangtin3 says,

Tuesday, June 22. 2010 at 10:01 (Reply)

Thanks Mark,

After four years, your article still valuable. In Apache 2.2 documentsand many other (Tomcat, Apache) cluster configuration tutorials, theydidn't specific the relation between "route" in Apache's httpd.conf


BalancerMember ajp://localhost:8009/passport route=node00
BalancerMember ajp://localhost:8019/passport route=node01
BalancerMember ajp://localhost:8029/passport route=node02


and "jvmRoute" in Tomcat's server.xml



Thanks for your article!Peter says,

Thursday, August 5. 2010 at 13:15 (Reply)

We had the problem of two IIS behind an Apache balancer. The application (ASP) needs sticky session, but when you give
ProxyPass /BLAH/ balancer://test/BLAH stickysession=ASPSESSIONID thatwill not works because there was many sessioncookies with differentnames at the end (ASPSESSIONIDABCDE; ASPSESSIONIDEFGHI...) and thestickysessionparameter in ProxyPass seems not to allow wildcards likeProxyPass /BLAH/ balancer://test/BLAH stickysession=ASPSESSIONID*.
So your solution works PERFECT.
Thanks!Sandeep says,

Tuesday, August 31. 2010 at 00:32 (Reply)

Hi Mark,
Your article is excellent. I was wondering if it fails with rewriteinstead of directly proxypassing. I have apache on frontend and anothercouple of httpd.s on backend serving pure content. my setup is like
______________________

BalancerMember http://10.10.10.1 route=cmshttp1
BalancerMember http://10.10.10.2 route=cmshttp2
ProxySet lbmethod=byrequests nofailover=Off


RewriteRule ^/publish/(.*)$ balancer://cmshttp/publish/$1 [P,QSA]

______________________
The cookie on backend is properly set and I can check by directly goingto http://10.10.10.1 (firebug). But I see in myhttp://10.6.32.154/balancer-manager that they are elected almost equally.
Any thoughts ? Thanks a lot.