Remote System Logging

This article attempts to describe how to configure multiple servers to use a single syslog host in Linux.

What is syslog?

Syslog is a mechanism dating back to the days of BSD Unix (NOTE: I may be incorrect here...) whereby applications, services, and the OS kernel were able to communicate messages for logging. In the day to day life of a system administrator, this is considered a critical part of the operating system as a whole as a means of auditing applications, retaining statistical information, and troubleshooting issues.

Why do I want to use remote system logging?

Some system administrators like being able to collate all logs on a single server whereby additional utilities can be run without having to collate multiple sources. Let us pretend for this example that you are a system administrator very keen on server security and want to run analytics on sources of failed authentication attempts.

syslogd or rsyslogd?

In most popular linux distributions there are now two different syslog daemons:

• syslogd (traditional 'compatible' daemon that behaves like the original from BSD Unix)
• rsyslogd (new version designed specifically for remote system logging)

Both daemons are capable of being configured to handle remote and local system logging, but the approaches are different.

Configuring the syslog server (using rsyslogd)

The 'newer' rsyslogd daemon makes it easy to configure for accepting logging for remote sources. This particular daemon is now used by default in (among others...):

• Ubuntu 10.04 LTS (Lucid Lynx)
• Ubuntu 11.10 (Oneiric Ocelot)
• Red Hat Enterprise 6.x
• CentOS 6.x

rsyslogd uses an 'include' structure for its configuration starting with the file /etc/rsyslogd.conf. Additional configurations are stored (usually starting with numbers to control the order of load) in the directory /etc/rsyslogd.d. To start, we will want to edit the file /etc/rsyslogd.conf to enable to 'remote' log listening modules. Use your favorite text editor and either uncomment or add the sections:

$ModLoad imudp $UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514

This will then tell rsyslogd to listen on TCP and UDP ports 514 (syslog standard) for any messages that might need to be logged.

WARNING: it is strongly advised that firewall rules be in place to restrict traffic to port 514 UDP and TCP ONLY to those hosts you wish to log from - otherwise you may be open to denial of service attacks!

Configuring the syslog server (using syslogd)

The classic syslogd (sometimes packaged as 'sysklogd') requires a bit of a different approach. For starters, it only supports UDP listening (which should be fine...). Additionally, the methods for enabling remote listening is not based on modules but rather based on command line arguments. Different distributions handle this in different ways...

• Ubuntu 8.04:

  modify file: /etc/default/syslogd and set environment variable:

  SYSLOGD="-r"

• CentOS 5.x:

  modify file: /etc/sysconfig/syslog and set environment variable:

  SYSLOGD_OPTIONS="-m 0 -r"

  NOTE: -m 0 is the CentOS 5.x default option meaning 'disable Mark' messages. You can retain that or not as you wish, just make sure to add the -r flag.

WARNING: it is strongly advised that firewall rules be in place to restrict traffic to port 514 UDP and TCP ONLY to those hosts you wish to log from - otherwise you may be open to denial of service attacks!

Configuring the 'client' servers

Now that we have configured the syslog server and restarted the syslog service (this will vary in distributions - /etc/init.d/{rsyslogd,syslogd,sysklogd,syslog} restart), we need to tell our servers to send log messages there.

rsyslogd

For servers that use rsyslogd as the syslog daemon, we can either send *all* logging to the remote server, or we can be judicious in our logging. For the purposes of our example, we will want to forward the log messages for failed authentications. We can do this by editing the file: /etc/rsyslog.d/50-default.conf (may be different based on distribution). Add the following line to the top of the file:

auth,authpriv.* @ip.address.of.syslogserver

Of course, fill in the actual ip address of your syslog server to make this a reality. Once the change is in place (at the top of the file, right?) then restart the local syslog (rsyslogd or syslogd or syslog) for the changes to take effect.

This is of course a very simplified walk through - there are many other pieces of flexibility available - and I may write more about them in the future. Keep tuned for a real life example of a web server cluster (high availability) for a single web site being configured to use syslog in this manner to store single location statistics via webalizer and awstats!