At Matsuri, we are starting to move some parts of our infrastructure to Kubernetes in a project we named Project MakubeX (Matsuri on Kubernetes), which may be covered in more details in another post. This short one will be all about HTTP request headers.
Back in the Heroku-only world we are coming out from, requests reached our single instance AP through Cloudflare’s proxy. As a result, the real client IP address appear in the Heroku logs and we didn’t need to think too hard and didn’t give tracing much thought.
And then we added load balancers (nginx) between the AP(s) and Cloudflare and the managed cluster we were using didn’t have or allowed
externalTrafficPolicy: Local, meaning we couldn’t rely on the ingress to provide us with the client IP. On our (still hosted with Heroku) AP we only received the cluster’s IP. And it seems that the ingress might also be rewriting
X-Forwarded-For instead of appending to it.
As it turns out, Cloudflare sets some very useful headers for us. One of them,
cf-connecting-ip gives us exactly what we are looking for: the client IP. Heroku logs use the
X-Forwarded-For header, as described in their documentation. To provide Heroku logs with the client IP, we then simply need to add this to our nginx configuration:
proxy_set_header X-Forwarded-For $http_cf_connecting_ip,$proxy_add_x_forwarded_for;
We decided to keep
$proxy_add_x_forwarded_for as it may become useful as we add some redundancy to our load balancers by splitting them accross other providers.
While we’re at it however, it’s worth noting that the client IP is not very useful. Actually, even in a single AP scenario IP alone is not that helpful for troubleshooting. What we should really be using is the tracing ID. Heroku looks for
X-Request-ID and sets it to a generated value if the header isn’t set. Nginx does the same, so we also can’t rely on it. Cloudflare comes to the rescue again, via the
cf-ray header. The Ray ID is also what is presented to the user by the Cloudflare error page, which makes it extra useful. Once again, passing this to Heroku is as simple as adding this to our nginx configuration:
proxy_set_header X-Request-ID $http_cf_ray;
One other useful aspect about the
cf-ray header is that it also provides a hint of to the location of the client: the
cf-ray ID is terminated by a three-letter airport code. Of course, Cloudflare also provides another useful header,
cf-ipcountry, for that, which removes the necessity of using an ip-to-country module inside nginx.
One caveat is that all of the above only works if all traffic gets proxied through Cloudflare. This is the case for us, and we go further by enabling Cloudflare’s Authenticated Origin Pull feature (mTLS) and allowing only requests going through Cloudflare.