So I am trying to test my picture storage app in a load balancing environment.
I got the client all set up to point at the load balancer and all the instances in the farm are up and running.
When I try to login from the client application, I always get a “Username/password pair do not match”. So what could be the problem ? It was working fine on my machine (typical developer lame escuse)
Looking into the apache logs of the server machines I noticed that the HTTP_AUTHORIZATION http header was missing when the login request was reaching the servers. So I started to find where in the request path the header was being lost.
I started suspecting the load balancer was stripping the headers from the request and not passing it to the app servers. Below is the configuration of the load balancer :
BalancerMember http://django1 ping=5 disablereuse=on retry=5 ttl=120
BalancerMember http://django2 ping=5 disablereuse=on retry=5 ttl=120
BalancerMember http://django3 ping=5 disablereuse=on retry=5 ttl=120
ProxyPass "/*" "balancer://django"
ProxyPassReverse "/*" "balancer://django"
I searched the net and could not find any hint that apache load balancing did anything special to the request headers.
So I tried looking at the receiving end of the process. I even had a couple of log statements on the authentication class on the server that logged the request headers and the request path.
I even added a header to the client and the server that had a custom name so to eliminate the hypothesis that only the authorization header was being stripped from the request.
Upon inspection of those log statements I found that neither the authorization header nor the custom header reached the authentication code.
So I started suspecting the apache server on each server that serves the app through mod_wsgi. While searching for a way to log the headers coming in to apache I came to know about the mod_log_forensic.so module. Since I had compiled apache on a similar machine to the server ones I just copied the .so to those apache installations and installed them.
Ran some tests again, and observed that the server apaches were in fact receiving both headers. So I nailed it down to the mod_wsgi python module.
So I did another search on Google for “request.META missing headers”. I came across a Stack Overflow post that finally cleared the mistery. Here’s the link to the post :
My custom header had an underscore, and for security reasons mod_wsgi removes headers with those characters from the request. See the post on why it does that. So now my custome header is reaching the authentication code with a “HTTP_” string prepended. That’s what mod_wsgi does to http headers, so that’s fine.
Now to tackle the real issue. Why is the “Authorization” header not getting through. Well, the previous SO post I mentioned before talks about another post that explains that :
In the post there are two possible solutions :
– configure mod_wsgi
– use url rewriting to pass the header
I prefer the former, so that’s what I did, and it did the trick.