Linux security monitoring: auditd + OSSEC integration part I

This article is devoted to the integration of two well-known and proven open source tools for security monitoring: change audit software for Linux (auditd) and Host IDS OSSEC. The aim of this article is to learn the limitations and use the advantages of both of these tools so that by acting in tandem they can detect suspicious behavior at the level of system calls (syscalls).

After reading part I, you will know how to:

• run a Linux audit
• write audit rules for system calls and files/directories
• log on to the remote server all commands related to running programs on Linux
• use the search and event analysis tools: ausearch and aureport

In Part II, we will focus on the advanced configuration of OSSEC (writing set-top boxes, writing rules and the Active Response module) and using information from auditd by this HIDS system.

Using comparisons with military tactics, auditd+OSSEC will carry out the tasks of a sniper pair with us, where the auditor will be an auditor and the fire task will be carried out by OSSEC using the Active Response module. So we start training and hunting for intruders.

Why monitor system calls?

System calls (syscalls) are requests directed to the kernel of the operating system that result in the execution of privileged procedures. The basic syscall systems are:

• open() – reading data
• write() – data entry
• open() – opening the file
• fork() – create a new process
• exec() – running the program
• and many more

As we can see, monitoring system calls related to running new programs, and reading and saving files can provide defenders with information about intruders’ actions in attacked systems; e.g., using a 0-day vulnerability. Despite the fact that the burglary took place, we as a defender are not yet in a lost position, provided that we have implemented an appropriate monitoring system and procedures that will allow us to detect and stop the ongoing attack. The attacker after the successful break-in will most often try to increase their powers, recognize the attacked system from the inside, steal data, and run malicious software. Such activities in most cases leave traces, and the essence of the actions of security teams is to pick up these traces and take the lead.

To monitor the above-mentioned activities, I will propose an auditd; i.e., software that at the Linux kernel level is able to perform system audits including system calls.

Observer: auditd

The auditd installation can be done using the package manager:

Debian,Ubuntu

Fedora, CentOS, RHEL (usually auditd should be installed)

The auditd package contains a number of binaries that are used to manage the auditd daemon and search and analyze the recorded events:

• auditctl: a tool for the current configuration of the auditd daemon, checking the status and running rules
• audispd: daemon is the multiplication of logged events to other log aggregation programs
• aureport: a tool for quick reports generated on the basis of a log (auditd.log)
• ausearch: tool to search events from the log (auditd.log)
• autrace: a tool for analyzing the interaction of programs with the kernel
• aulast: tool with the last command functionality but using log auditd.log
• aulastlog: tool with the functionality of the lastlog command but using log auditd.log
• ausyscall: mapping the system call ID to the name
• auvirt: auditing information on virtual machines

Launching an audit at every system startup (Fedora, CentOS, RHEL):

We introduce the observer to the game:

and we check the status of the service:

AAt the beginning, the auditd does not contain any rules, which can be checked each time by using the -l switch:

It’s time to put tasks to our observer; that is, we write rules. The rules can be added to the audit.rules file (most often in the /etc/audit location) or using the auditctl tool with the -k switch.

Below I have placed a few rules that will allow, among other things, to monitor all commands executed in the operating system and set a trap for intruders who would not only like to modify system files but “only” read them. We can also be more granular and enable or disable monitoring rules for users with specific UID, programs, files, directories, and System call ID.

-a always,exit -F arch=b32 -S execve -k execv	logging of program start calls (32-bit architecture)
-a always,exit -F arch=b64 -S execve -k execv	logging of program start calls (64-bit architecture), it is recommended to use both rules
-w /etc/passwd -p war -k watch_passwd	logging of reading (r), writing (in), attribute changes (a) of file /etc/passwd
-w /etc/shadow -p war -k watch_shadow	logging of reading (r), writing (in), attribute changes (a) of file /etc/shadow
-a never,exit -F path=/bin/grep	disable certain instances from logging in (note: the rule must be placed at the top of the file)
a always,exit -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -k delete	logging of selective calls related to the deletion of files by users whose UID is greater than 500
-w /etc -p wa -k watch_etc	llogging of write and attribute changes in the /etc directory

Examples of interesting, detailed rules for auditd can be found on the github site:

• alienone user rules
• auditd rules to comply with PCI-DSS (Payment Card Industry Data Security Standard) requirements
• auditd rules to comply with NISPOM-DSS (National Industrial Security Program)
• auditd rules for compliance with the requirements of the LSPP-DSS (Labeled Security Protection Profile Data Security Standard)
• auditd rules to comply with the NIST and DISA STIG requirements for RedHat

In this article, we will focus on a full review of the situation, because we do not want any suspicious behavior to escape us. Therefore, we will use the first two rules to keep track of what is happening in the monitored system:

In this way, we have configured a comprehensive observer who will follow all activity in the system and will additionally be focused on any access to critical files (here the flagship duo—passwd and shadow). It seems that auditd is an ideal tool and will not miss it, and although in most cases this is true, we must be aware of its limitations. For example, running this tool in a system previously infected with a well-prepared rootkit operating at the kernel level of the system may cause malicious activity to be hidden in whole or in part. An example of an attack on software that intercepts system calls can be found in the article Exploiting Concurrency Vulnerabilities in System Call Wrappers. At the beginning, the auditd may also overwhelm the enormity of information that it registers in its log, and the overload of information may sometimes be worse than its absence. Everything is a matter of subjugating auditd and its proper use.

However, before we move on to the integration of HIDS OSSEC, several examples of using auditd solo, which will help reduce the revulsion of the single-event multi-line audit logs.

Auditd example 1: search auditd.log

The event analyzed is a hacking to the server using an unprivileged user account. The intruder tried to copy the shadow file, so he had to be noticed by auditd:

The log contains the following entry regarding this event, which the auditd assigned to the number 182858:

The PATH line describes the file descriptor to which the event applies.

The CWD (current working directory) line indicates which location the command was issued from.

The SYSCALL line describes the details of the system call:

• system call number (syscall = 2 – open())
• the result of the operation completed in this case failure (success=noexit=-13)
• process number and process number of the parent (ppid=26215 pid=26768)
• terminal number (tty=pts4)
• system command and bin location (comm=”cp” exe=”/bin/cp”)
• information about the user (auid=1005 uid=1005 gid=1003 euid= 1005 suid=1005 fsuid=1005 egid=1003 sgid=1003 fsgid=1003)

Execution of the same command but with elevated privileges using the sudo functionality:

gives the following result in the audt.log log:

It is worth noting that this time the reading of the file was successful (success=yes), and the operation was carried out with elevated privileges as indicated by the euid field (effective uid), which now has the value 0 (the root UID). The comm field contains information about the sudo program, but we have no information about the arguments of this command. To this end, we will use that the auditd monitors all running commands along with the arguments by using a rule called execve and we will look up all the events related to the sudo command and the execve monitoring rule:

Resulting in such an event:

The EXECVE line containing all the arguments of the command is of particular interest. Thus, we can see that by properly formatting the EXECVE line from the audit.log log, we can receive current information about which commands and programs are run in the monitored system (Example No. 5).

Auditd Example 2: Analysis of the Source of Malware

After detecting suspicious activity on the host, auditd is able to help us determine the source of the malware. For this purpose, we use the ppid (parent process id) field and check how the file copy command was run.

The SYSCALL line contains information that the parent process is a dash system shell, otherwise it could be a script or binaries. However, in each of these cases we can determine where the source of suspicious behavior is to ultimately verify whether the source is malware, hacking or authorized administrators. Often you have to check several offspring processes to finally reach the infected library.

If we analyze the following processes step by step, we come to a situation where their source is, e.g., some of the system binaries. For example less, the auditd package contains an autrace tool, which has functionality similar to strace. Using autrace, we can analyze the references to files and libraries used by a given binaries, which can also bring us closer to the source of malware.

We get a hint to use the ausearch -i -p 8312 command. In this example with the malware search, I suggest to use some:

Note: the autrace command precedes clearing the rules using auditctl -D.

Auditd example 3: advanced search (ausearch) and report creation (aureport)

Here are some useful switches to search the audit.log log:

searches for an event from the current day related to deleting files using the rm command. Other interesting time periods that we can specify using the -ts to now, recent switch (last 10 minutes), today, yesterday, this-week, this-month, this -year.

Searches for user activity with UID 1001.

The -i switch converts numerical values to names such as UID’s for user names.

The -k switch allows you to search for events for which we have prepared rules in the audit.rules file.

In the ausearch manual, we will find further examples of switches that we can use to query the auditd about what we observed.

In addition to precise responses in the form of extracting information from the log using ausearch, the auditd package also contains aureport binary, which is used to create reports and summarize system audits in a wider time-contact.

Using aureport we can generate a report about all subsequent launches of executable programs between 12.00 and 13.00:

Here is a report on all programs performed on the current day with a summary of the number of calls to these programs:

Report on failed logins using the -au switch :

The summary about account modifications can be obtained using the -m switch:

A useful option is also to generate an anomaly report using the -n switch.

Interestingly, both ausearch and aureport commands can be connected together:

With the help of the above command, we have summarized all the programs that got access to the /etc/shadow file on a given day, and the disturbing copy command (/bin/cp) of this file was executed six times.

Auditd example 4: central login

A good practice of monitoring is to use a central log-in, and although it seems obvious, the realities leave much to be desired. Auditd contains the audispd plugin, which makes the configuration of sending the audit log to the remote login system quite simple. In this example, we’ll start with the rsyslog server configuration:

/etc/rsyslog.conf file on server:

The syslog server will listen on port 2514 TCP, and log audit.log will be placed in /var/log/rsyslog/%HOSTNAME% with the auditd format, which means that you can use tools such as ausearch or aureport by specifying the file location using the -if switch, e.g.

file /etc/rsyslog.conf at the client:

When configuring syslog, you need to be particularly careful in the configuration files for separating selectors and actions using tabs rather than spaces:

In the configuration of the audispd plugin at the client in the file /etc/audisp/plugins.d/syslog.conf enable the option to send events to the syslog:

Client-server communication containing such sensitive data as the system audit log must be encrypted, and on the official rsyslog.com site you can find the TLS configuration tutorial for rsyslog.

Auditd example 5: view all commands in real time (script in Python formatting audit.log)

Although the first examples should make log audit.log no longer cause a headache, and tools such as ausearch and aureport improve the search and analysis of it, but it is worth having your own tools that allow you to extract the essence from the log, in our case – all commands that run real-time executable programs. For this purpose, you can use a short script in Python, which will format log audit.log for our needs.

tailAuditLog.py

The logging of all commands at the level of system calls is best checked by opening two terminal windows in parallel:

1. In the first run the script with the command: python /root/tailAuditLog.py
2. In the second window, we issue any shell commands, run programs, etc.

The result of the script operation are the next registered EXECVE system calls, where the event ID was placed in the first column, which can be used for detailed analysis of auditd package tools.

If we configured the central login described in the example number 4, the script is best run on the server by changing the location of the auditd.log file in the script.

Summary

In the first part of the article we got to know the first player – auditd, whose task is to observe system calls that take place in the monitored system. It is worth getting to know this audit tool due to the wide range of logging options for events related to both system calls and file descriptors. What’s more, the flexibility of building rules allows to meet the requirements set by many standards regulating the details of the audit (NIST, PCI_DSS, etc.). I hope that the examples of using ausearch and aureport tools have made it easier to get used to the large amount of information provided by auditd. Auditd sees a lot and registers a lot, but does not contain an alerting module and does not block unwanted activity.

In the second part of the article I will present the necessary OSSEC HIDS configuration for cooperation with auditd. In other words, the topic will concern writing your own decoders, writing OSSEC rules and using the Active Response module, thanks to which OSSEC is not just an intrusion detection system, but also for precise blocking of intruders. Through the use of these tools, we increase our arsenal, which means that we are not vulnerable. What’s more, it is also time to switch from the seemingly obvious “waiting for the incident” to the strategy of going against the opponent. The strategy change consists in an offensive attitude towards defence. By thinking like a burglar, we can anticipate their movements and set their own ambushes there. Auditd is certainly a tool with which we can implement it, because it radically increases our picture of the situation.

Author: Bartek Jerzman

Tagged: Linux