What is Syslog?

Syslog is the message logging utility used on Unix, Linux and Network Device operating systems (e.g. Cisco, Juniper).  

Like the Windows Event Log, syslog collects messages from processes and the operating system, but unlike Windows, syslog has a built in architecture for automatically forwarding messages to remote syslog receivers.

Syslog was created in the 1980’s as part of BSD Unix, but was not formally codified by the Internet Engineering Task Force (IETF) until August 2001 in RFC 3164.  In March 2009, RFC 5424 superseded RFC3164, expanding the syslog transport mechanism from UDP only to the more secure Transport Layer Security (TLS), providing guaranteed delivery to receivers and better handling of congestion due to message volume.

The original 1980’s syslog daemon for Unix systems was named syslog, and is still currently in use, but the newer syslog-ng and rsyslog implementations are now more common.  While network device operating systems have their own built-in syslog implementations which are configured through their own management interface, unix syslog daemons are usually configured through edits to syslog configuration files in the /etc directory.

How Syslog works

Given that syslog has been around for more than 30 years and is available for a wide variety of OS versions, the exact details of how events generated by the system get to the syslog daemon will vary.  The basics are:

1. The syslog daemon can be configured to receive events from multiple sources.  Local events are typically sent over a socket (e.g. /dev/log) and remote events through a port (default= UDP 514).
   
   
2. The OS kernel uses klogd or a similar daemon to write to syslog.
   
   
3. Program output to syslog can be configured in most cases.  For example, nginx can be configured to specify socket, port, facility and severity for each of its log files.
   
   
4. In addition to message content, syslog messages have properties for facility, which indicates their source, their severity, and, if from a remote server, their host server.  The priority property is a combination of facility and severity.
   
   
5. Syslog receivers process incoming messages based on the message’s properties. The extent of filtering and forwarding options available in the configuration file will vary based on the syslog daemon in use, but at a minimum, syslog records can be grouped and filtered by their facility and severity properties, and written to log files or forwarded to another syslog receiver.

Syslog message properties:  facility, severity and priority

The syslog receiver’s configuration file processes messages primarily based on their facility and severity.  However, when the syslog message is generated, it is sent with a priority value that has been calculated from the facility and severity rather than the individual properties.

Each facility and severity has both a name and a numeric value.  Facility values range from 0 - 23, and include 8 “local use” fields that are often used by vendors for their custom messages - for example, Cisco uses local7 for Cisco IOS devices and local4 for PIX firewalls.

In syslog configuration files, the facility values are often referred to by abbreviations - e.g. kernel messages are referred to by kern.  The following table lists the facility numeric values, names, and the abbreviations commonly used for them in syslog configuration files:

Severities range from 0 (Emergency) to 7 (Debug):

_{* these terms have been deprecated}

The priority is calculated as:

So, a kernel alert would be priority 1 (= 0*8 + 1), and a mail information message would be priority 22 (= 2*8 + 6).  The following table lists the priority based on severity and facility:

Configuring syslog: Syslog configuration file

The syslog daemon uses one or more configuration files to set up syslog receivers and message processing.  For our example, we’ll be looking at a basic /etc/rsyslog.conf file for rsyslog version 8.26.1 on CentOS 7.1, but the general concepts are about the same for other syslog daemons.

The rsyslog.conf file has the following sections:

1. Modules   The modules section allows you to select only the modules you need for your system and configure their basic parameters.  Some of the basic modules and the syntax to load them are:
   
   

   Local system logging	module(load="imuxsock")
   Kernel logging	module(load="imklog")
   UDP syslog receiver	module(load="imudp")
   TCP syslog receiver	module(load="imtcp")

   
   The UDP or TCP receivers also need to be configured to listen on one or more ports - the syntax for that is:
   
   
   • input(type="imudp" port="514")
   • input(type="imtcp" port="514")
   
   Note that 514 is the default port for transmitting syslog messages, but any available port can be used.  RFC 3164 originally specified using UDP, which does not provided guaranteed delivery of messages, but is still widely in use.  If you’re using syslog to send messages that must be retained, use TCP instead of UDP.
   
   
   
2. Global settings
   
   This section is used to set up default settings for handling messages.  The out of the box settings are:
   
   

   $ActionFileDefaultTemplate RSYSLOG_TradtionalFileFormat	default message format
   $IncludeConfig /etc/rsyslog.d/*.conf	additional configuration files to include

   
   This section also contains a commented out option for file write synchronization:
   
   #$ActionFileEnableSync on
   
   File synchronization is available for backwards compatibility with the original syslogd implementation, and is not recommended as enabling this feature can cause a significant slowdown in I/O.
   
   
    
3. Rules
   
   The rules section specifies how syslog should handle each incoming message. The basic configuration is: {facility.severity} {action} For example:
   
   

   mail.*    /var/log/mail.log	directs all mail facility messages to /var/log/mail.log
   *.err;local7.emerg    @@syslogalert.local.net:5140	directs all error messages and local7 emergency messages to syslog receiver syslogalert.local.net using  TCP port 5140
   *.*    @syslogarchive.local.net	directs all syslog messages to syslog receiver syslogarchive.local.net using  UDP and the default port (514)

   
   In the examples above, the abbreviations listed for the facility and emergency names are used to specify messages, and the messages are either written to a log file or sent to a syslog receiver via TCP or UDP (@ before the receiver name for UDP,  @@ for TCP).  
   
   The basic “facility.severity  action” format for rules covers most syslog message handling - however, for more complex needs, rsyslogd can also use regular expressions, can filter by message text and hostname, and can change the format used to output the message. Check the documentation for your OS and syslog daemon for specific details.

Troubleshooting

• Run “rsyslogd -N1” to verify that there are no syntax errors in /etc/rsyslog.conf, and restart rsyslog to make sure the current configuration file is loaded.
• For syslogs sent to remote receivers, verify that firewalls on the syslog sender, syslog receiver, and firewalls in the network between them allow the port and protocol used to send syslog (default = UDP 514).
• In addition to sending syslog to the receiver, send to a local file as well to verify that the messages are being received locally, e.g.:
  *.crit @syslogreceiver
  *.crit /var/log/syslog_debug.log
• If events are missing and they are being transmitted via UDP, check the volume of events being received.  Messages sent via UDP are not guaranteed and congestion can mean more messages are dropped.  If congestion is an issue, try filtering the messages being sent to cut down on volume and changing the protocol to TCP.
• Run rsyslog in debug mode using “rsyslog -d”.  
  Make sure that you’ve shut down the system rsyslog daemon.  To make it easier to analyze the output use either the “script” command or pipe the output through the “tee” command to save the output to a file.  Note that there is a “rsyslog -D” option, but this provides development level debug output and is usually overkill.

Conclusion

Syslog provides a standardized way to consolidate and prioritize messages across a wide variety of network devices and Unix/Linux operating systems.  While there are multiple versions of syslog that can provide advanced message handling options, basic configuration settings often meet most logging needs.

Want to learn more?

Download a FREE trial of Splunk  - Download a copy today and see how Splunk makes it simple to collect, analyze, and act upon the untapped value of big data.