How to centralize server logs with Rsyslog?

By | January 26, 2016

Centralizing Server logs with Rsyslog

1. Overview

Centralizing logs from multiple servers on one server can be of great interest to the security level within a system of information. Indeed, it is easier for log analysis tools to compare, read and scan files being on a single server rather than doing it remotely or through remote agents. Also in case of server crash, you will be able to recover the mistakes and actions on your server before it does crash, facilitating the restoration activity of this and future security.

On most of the modern Linux systems then log management tool is by default Rsyslog and it is one that we will use in this tutorial. Here, we will work on two machines running Debian 7, one acting as a server, the other as a “client” that will send its logs to the server. In both cases, the configuration file to edit is always /etc/rsyslog.conf. One can nevertheless put in /etc/rsylog.d/ and make a “include” in the main configuration file to import.

2. Server Configuration

For the server, simply set the server to accept logs from outside by opening the appropriate port, for the time we will open port (UDP 514) for simplicity. For this, will uncomment the following two lines:

We can then restart the rsyslog Service:

To verify that our server is listening, you can perform the following command:

We then see our open UDP port 514…

3. Client Configuration

At the client level, we will go to the same file and then redirect all logs on the server. Add the following line to the end of the file:

Note: The “@” must appear in the line
It will then restart rsyslog our Service:

Then we can generate logs; for example, restart any service to see if they are on the server rsyslog. By default, as we did, customer logs will be in the same file as the server logs, for example in the /var/log/syslog server. You just see the name of the host that generated the log line is not that of the server, but the client’s.

4. TCP / UDP

Previously, we configured our client and our server to exchange using the transport protocol (layer 4) UDP. Briefly, UDP allows a single exchange to transmit a packet is a protocol that does not try to synchronize, track packages status or exchanges between two assets. Instead, TCP will try to find for each packet if it has been received by the correspondent, which generates additional packets over UDP. TCP is able to check whether an information exchange or not been received and retransmitted if necessary that does not UDP.

All this is to underline that in our case, TCP exchanges have a disadvantage to be taken into account in relation to the UDP. Indeed, the TCP will generate more trade (and therefore resource consumption) over UDP, but will ensure that the information received is in full. So there is a choice which depends on your available resources and the number of customer and information managed by your server. To trade in TCP mode, you must refer these lines in the server configuration file, note that both methods can coexist if we put them on different ports, for example:

Otherwise, it will review the top two lines. On the client side, add this line in the configuration for that trade is TCP:

Note: The @@ must appears in the command line, the fact that there have two stated that exchange will TCP.

One can also leave two lines to indicate that the logs will be sent once TCP and once UDP. This is of little interest but for understanding that we can also send our logs to several different IP servers if you change one the two lines. Then we restart the server and the client if needed:

5. Redirect or copy according to the priority or services

When talking about log management especially in Linux, the services are categories in which the logs will store to better archive and sort. These services include, for example:

1) auth: used for events relating to the security or authentication through access applications (SSH type)
2) authpriv: Used for messages related to access control
3) daemon: Used by different systems and application process
4) kern: used for messages about the kernel
5) Email: Used for Event services email
6) user: default services when none is specified
7) local7: Indicates the boot messages
8) *: Refers to all services, for simplicity this is what we specified when we set first redirection rule slightly above logs
9) none: Indicates no services

In addition to these services, we find services for each severity level (called Priority) that goes from worst to mere information:

1) Emerg: Emergency System unusable
2) Alert: Immediate action required
3) Crit: Critical System Error
4) Err: Operation error
5) Warning: Warning
6) Notice: normal Reportable Events
7) Info: For information
8) Debug: debug Message

One then finds many ways to specify the logs that interest us and therefore those we will redirect. For example if you try to redirect to our server logs only display critical messages and higher for messages on UDP port 514, add the following line:

It can also redirect all mail logs:

One can also enter in a line several types of facilities and priority; there is no example in the default configuration file lines like these:

Here we see that all the debug priorities are redirected to the file /var/log/debug and that all priorities info notice and warn will be in /var/log/messages. For these filters are redirected to the log server, just specify the server IP and the port has done above in place of the file name.

6. Redirect logs to a folder / file by host

It is also to facilitate prioritization and archive our logs when you have a large number of customers Rsyslog use a tree with a folder/file by host rather than putting all logs in the same file as the server logs. We will do this using a template that we will after the block ‘RULES’ in the server configuration file:

We will then apply this template to all incoming logs:

It then suffices to restart rsyslog our services and generate logs from clients. Then we will end up with a file “/ var / log / clients /” containing a folder by IP / customer name and a file containing respectively “syslog.log” with logs of each respective client, which simplifies the search for information in the logs for a specific customer…