syslog centralized logging

Syslog is a standard in computer networks for the register of messages provided by different technologies. It is used for the debugging messages generated by applications, security auditing or other relevant information that we need to know about our systems or applications in any moment. We can classify syslog messages by their facility and priority, the facility refers to the source of the log message, there are many predefinied facilities:

The facilities are associated with a severity level, used for give a priority to the log messages. The list of severity levels:

Every entry in the syslog configuration is composed by one facility.priority action. We can use wildcard for the configuration:

•  , : separate one facility or priority in the same rule.
• * : All possibilites.
• = : Grants exclusivity to a facility or priority.
• ! : Exclude a facility or priority to a rule.
• != : Exclude only the selected priority.
• ; : separates a facility.priority selector for the same output rule.

The possible actions could be:

• Users
• Files
• Pipe
• Remote servers

There are some implementations of syslog, like syslog-ng, sysklogd, rsyslog… The used in this post is rsyslog, this implementation is preinstalled in the most GNU/Linux distributions and available BSD based systems. rsyslog implement extended features like content based filtering, rich filtering capabilities and the possibility of use TCP for transport. In this post I’ll introduced for the configuration of rsyslog for centralize the logging messages in one server with some useful examples of content filtering and the use of templates for the transform of the output messages.

rsyslog server

– Edit /etc/rsyslog.conf

#### MODULES ####
# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514

#### TEMPLATES ####

#Default Template in use, timestamp format.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template RemoteHost,"TAG: %SYSLOGTAG% MSG: %msg%n"
$template FilePerHost,"/var/log/system-%fromhost-ip%/%timegenerated:1:10:date-rfc3339%.log"

#### RULES ####

#Condition for all log messages from IP address 172.16.0.1 send to a separate log file
if $fromhost-ip == '172.16.0.1' then /var/log/cisco-router.log
# Send all user severities to the user.log file with the output format of RemoteHost template
user.* /var/log/user.log;RemoteHost
# Create one file for each host following FilePerHost template format
*.* ?FilePerHost

With this configuration we’ve a simple rsyslog server listening for TCP/UDP 514 port. First I created two templates, RemoteHost with a personalized format and FilePerHost used for create one directory per host named system- and in each directory rsyslog will creates a log file every day. The rules created are:

• From the IP address 172.16.0.1 the output log will be /var/log/cisco-router.log
• All severities for the user facility will be stored in /var/log/user.log using the RemoteHost template fromat
• All the facilities with all the severities will be stored using the FilePerHost template.

Now we can restart rsyslog for the changes take effect:

# service rsyslog restart

Cisco Router client

RCENTRAL(config)#logging host 172.16.0.2 transport tcp port 514

OR

RCENTRAL(config)#logging host 172.16.0.2 transport udp port 514

rsyslog client

– Edit /etc/rsyslog.conf

user.* @192.168.0.111
*.* @@172.16.0.2
$ActionExecOnlyWhenPreviousIsSuspended on
& @@172.16.0.3
& /var/log/local-buffer
$ActionExecOnlyWhenPreviousIsSuspended off

With this client configuration rsyslog will sends all user facility to the server 192.168.0.111 using UDP (@). All the facilities will sends to the server 172.16.0.2 using TCP (@@). If fails the communication with this server, It will try with the server 172.16.0.3 with the same selector (&&). If also fails, it will be stored in localhost in the file /var/log/local-buffer. With this configuration we add high availability to our syslog platform if one of the servers goes down. The use of TCP is required with this configuration because is the most reliable way to check if one of the server fails.

– Restart the service:

# service rsyslog restart

Now we can use the logger command for make an entry in the system log to test our configuration:

# logger Hello remote server, This is an example message

For more information about rsyslog for content filter and file format you can see:

– Filter conditions:
http://www.rsyslog.com/doc/rsyslog_conf_filter.html

– Property replacer:
http://www.rsyslog.com/doc/property_replacer.html

Or RFC 5424 about the functioning of syslog:

– Syslog RFC:
http://tools.ietf.org/html/rfc5424