Why you can not always trust web server logs?

Sketching the situation

Let’s suppose we do a server post-breach analysis and manage to state the following:

• external access is possible only through a web application, and the web server is running with the privileges of an unprivileged user,
• the application is out-of-date and contains publicly known RCE vulnerability (remote code execution),
• the access_log, error_log files have the appropriate permissions and there are no signs of vulnerability in them.

In this situation, can we think that the attack vector was different? It turns out that not necessarily, because it is possible that the attacker has obtained root and modified the logs. But what if the server is properly configured and we exclude such a possibility? It might seem that you should look elsewhere, but I asked myself: What if the queries were not logged in? The question may seem absurd, but I have taken the subject seriously.

Searching for answers on the Internet

The first step was research on the Internet, and the only point of reference was the question on StackOverflow. The author asked about the possibility, in which the server does not log some queries. However, no answer was given.

Searching for answers on your own

I proceeded to take action myself: I went to the Apache documentation to learn more about the access_log file. The first sentence turned out to be interesting:

The server access log records all requests processed by the server.

For verification, I checked the nginx documentation and there:

NGINX writes information about client requests in the access log right after the request is processed.

We can conclude from this that logs are saved only for queries that have been processed. This also confirms the sample entry:

Number 200 means status, so try to find a situation where the server will stop processing and will not return any response.

Testing environment

We choose the easiest and probably the most popular configuration of the web server: Apache + PHP, where Apache uses prefork MPM and communicates with PHP using the built-in module.

Consider various ways to terminate the code processing:

• syntax error
• throwing an exception
• exit ()
• die ()
• tiggererror () trigger_error ()
• __haltcompiler()
• killing the process (eg by itself)

For each method, a file is prepared, for example, 01_syntaxerror.php:

The whole is closed in Docker containers, with redirection of access_log, error_log to standard output. To this scripts are added that allow full testing automation. The code used is available in the public repository andrzej1_1/rce-weblogs-bypass.

Tests

Running the run.sh script returns. the following logs:

As you can see, most of the scripts left a trail behind them, and only 07_selfkill.php did not result in the creation of any entry. Here are its contents:

The code operation is simple: the kill command is executed on the current process.

Further tests

We managed to execute a script that did not leave any trace, however, it is worth checking if the behavior will be identical for other configurations. There are many different ways to run a server that supports PHP, because we have a choice of different servers and interfaces for communication. In the case of Apache, we can additionally choose different processing modules (MPMs), of which the prefork, worker and event are stable. Due to the great possibilities, we will test the most popular of them:

1. Apache (prefork MPM) + mod_php (this configuration was already tested)
2. Apache (prefork MPM) + FastCGI
3. Apache (worker MPM) + FastCGI
4. Apache (event MPM) + FastCGI
5. Apache (prefork MPM) + FPM
6. Apache (worker MPM) + FPM
7. Apache (event MPM) + FPM
8. nginx + FastCGI
9. nginx + FPM

A separate container has been created for each configuration, and running the run.sh script will test each one and generate results in the form of log files.

However, after running the tests and reviewing the results, it turns out that all configurations outside the first one are “resistant” to the considered methods.

Conclusions

The tests show that the configuration of Apache with the prefork module and mod_php allows to bypass the login of queries in the case of RCE vulnerability. It is certainly not a common situation, but it is worth knowing that such a “trick” exists.

Author: Andrzej Broński

Tagged: Analysis, Apache, RCE