Blog

Configuring and Collecting Syslog

May 18, 2017 | Susan Bilder

What is Syslog?

Syslog is the message logging utility used on Unix, Linux and Network Device operating systems (e.g. Cisco, Juniper).  

Like the Windows Event Log, syslog collects messages from processes and the operating system, but unlike Windows, syslog has a built in architecture for automatically forwarding messages to remote syslog receivers.

Syslog was created in the 1980’s as part of BSD Unix, but was not formally codified by the Internet Engineering Task Force (IETF) until August 2001 in RFC 3164.  In March 2009, RFC 5424 superseded RFC3164, expanding the syslog transport mechanism from UDP only to the more secure Transport Layer Security (TLS), providing guaranteed delivery to receivers and better handling of congestion due to message volume.

The original 1980’s syslog daemon for Unix systems was named syslog, and is still currently in use, but the newer syslog-ng and rsyslog implementations are now more common.  While network device operating systems have their own built-in syslog implementations which are configured through their own management interface, unix syslog daemons are usually configured through edits to syslog configuration files in the /etc directory.

 

How Syslog works

Given that syslog has been around for more than 30 years and is available for a wide variety of OS versions, the exact details of how events generated by the system get to the syslog daemon will vary.  The basics are:

  1. The syslog daemon can be configured to receive events from multiple sources.  Local events are typically sent over a socket (e.g. /dev/log) and remote events through a port (default= UDP 514).

  2. The OS kernel uses klogd or a similar daemon to write to syslog.

  3. Program output to syslog can be configured in most cases.  For example, nginx can be configured to specify socket, port, facility and severity for each of its log files.

  4. In addition to message content, syslog messages have properties for facility, which indicates their source, their severity, and, if from a remote server, their host server.  The priority property is a combination of facility and severity.

  5. Syslog receivers process incoming messages based on the message’s properties. The extent of filtering and forwarding options available in the configuration file will vary based on the syslog daemon in use, but at a minimum, syslog records can be grouped and filtered by their facility and severity properties, and written to log files or forwarded to another syslog receiver.

 

Syslog message properties:  facility, severity and priority

The syslog receiver’s configuration file processes messages primarily based on their facility and severity.  However, when the syslog message is generated, it is sent with a priority value that has been calculated from the facility and severity rather than the individual properties.

Each facility and severity has both a name and a numeric value.  Facility values range from 0 - 23, and include 8 “local use” fields that are often used by vendors for their custom messages - for example, Cisco uses local7 for Cisco IOS devices and local4 for PIX firewalls.

In syslog configuration files, the facility values are often referred to by abbreviations - e.g. kernel messages are referred to by kern.  The following table lists the facility numeric values, names, and the abbreviations commonly used for them in syslog configuration files:

 
Numeric
Code
Facility File Abbreviation
0 kernel messages kern
1 user-level messages user
2 mail system mail
3 system daemons daemon
4 security/authorization messages auth
5 messages generated internally by syslogd syslog
6 line printer subsystem lpr
7 network news subsystem news
8 UUCP subsystem uucp
9 clock cron
10 security/authorization messages authpriv
11 FTP daemon ftp
12 NTP subsystem ntp
13 log audit security
14 log alert console
15 clock daemon cron
16 local use 0 local0
17 local use 1 local1
18 local use 2 local2
19 local use 3 local3
20 local use 4 local4
21 local use 5 local5
22 local use 6 local6
23 local use 7 local7
   

Severities range from 0 (Emergency) to 7 (Debug):

Numeric
Code
Severity File Abbreviation
0 Emergency: system is unusable emerg (or panic*)
1 Alert: action must be taken immediately alert
2 Critical: critical conditions crit
3 Error: error conditions err (or error*)
4 Warning: warning conditions warning (or warn*)
5 Notice: normal but significant condition notice
6 Informational: informational messages info
7 Debug: debug-level messages debug

* these terms have been deprecated

   

The priority is calculated as:

priority =  facility * 8  + severity

So, a kernel alert would be priority 1 (= 0*8 + 1), and a mail information message would be priority 22 (= 2*8 + 6).  The following table lists the priority based on severity and facility:

 
  severity emerg alert crit err warning notice info debug
facility   0 1 2 3 4 5 6 7
kernel 0 0 1 2 3 4 5 6 7
user 1 8 9 10 11 12 13 14 15
mail 2 16 17 18 19 20 21 22 23
daemon 3 24 25 26 27 28 29 30 31
auth 4 32 33 34 35 36 37 38 39
syslog 5 40 41 42 43 44 45 46 47
lpr 6 48 49 50 51 52 53 54 55
news 7 56 57 58 59 60 61 62 63
uucp 8 64 65 66 67 68 69 70 71
cron 9 72 73 74 75 76 77 78 79
authpriv 10 80 81 82 83 84 85 86 87
ftp 11 88 89 90 91 92 93 94 95
ntp 12 96 97 98 99 100 101 102 103
security 13 104 105 106 107 108 109 110 111
console 14 112 113 114 115 116 117 118 119
cron 15 120 121 122 123 124 125 126 127
local0 16 128 129 130 131 132 133 134 135
local1 17 136 137 138 139 140 141 142 143
local2 18 144 145 146 147 148 149 150 151
local3 19 152 153 154 155 156 157 158 159
local4 20 160 161 162 163 164 165 166 167
local5 21 168 169 170 171 172 173 174 175
local6 22 176 177 178 179 180 181 182 183
local7 23 184 185 186 187 188 189 190 191

 

Configuring syslog: Syslog configuration file

The syslog daemon uses one or more configuration files to set up syslog receivers and message processing.  For our example, we’ll be looking at a basic /etc/rsyslog.conf file for rsyslog version 8.26.1 on CentOS 7.1, but the general concepts are about the same for other syslog daemons.

The rsyslog.conf file has the following sections:

        1. Modules   The modules section allows you to select only the modules you need for your system and configure their basic parameters.  Some of the basic modules and the syntax to load them are:

          Local system logging module(load="imuxsock")
          Kernel logging module(load="imklog")
          UDP syslog receiver module(load="imudp")
          TCP syslog receiver module(load="imtcp")

          The UDP or TCP receivers also need to be configured to listen on one or more ports - the syntax for that is:

          • input(type="imudp" port="514")
          • input(type="imtcp" port="514")

          Note that 514 is the default port for transmitting syslog messages, but any available port can be used.  RFC 3164 originally specified using UDP, which does not provided guaranteed delivery of messages, but is still widely in use.  If you’re using syslog to send messages that must be retained, use TCP instead of UDP.


        2. Global settings

          This section is used to set up default settings for handling messages.  The out of the box settings are:

          $ActionFileDefaultTemplate RSYSLOG_TradtionalFileFormat default message format
          $IncludeConfig /etc/rsyslog.d/*.conf additional configuration files to include

          This section also contains a commented out option for file write synchronization:

          #$ActionFileEnableSync on

          File synchronization is available for backwards compatibility with the original syslogd implementation, and is not recommended as enabling this feature can cause a significant slowdown in I/O.


           
        3. Rules

          The rules section specifies how syslog should handle each incoming message. The basic configuration is: {facility.severity} {action} For example:

          mail.*    /var/log/mail.log directs all mail facility messages to /var/log/mail.log
          *.err;local7.emerg    @@syslogalert.local.net:5140 directs all error messages and local7 emergency messages to syslog receiver syslogalert.local.net using  TCP port 5140
          *.*    @syslogarchive.local.net directs all syslog messages to syslog receiver syslogarchive.local.net using  UDP and the default port (514)

          In the examples above, the abbreviations listed for the facility and emergency names are used to specify messages, and the messages are either written to a log file or sent to a syslog receiver via TCP or UDP (@ before the receiver name for UDP,  @@ for TCP).  

          The basic “facility.severity  action” format for rules covers most syslog message handling - however, for more complex needs, rsyslogd can also use regular expressions, can filter by message text and hostname, and can change the format used to output the message. Check the documentation for your OS and syslog daemon for specific details.

   

Troubleshooting

        • Run “rsyslogd -N1” to verify that there are no syntax errors in /etc/rsyslog.conf, and restart rsyslog to make sure the current configuration file is loaded.
        • For syslogs sent to remote receivers, verify that firewalls on the syslog sender, syslog receiver, and firewalls in the network between them allow the port and protocol used to send syslog (default = UDP 514).
        • In addition to sending syslog to the receiver, send to a local file as well to verify that the messages are being received locally, e.g.:
          *.crit @syslogreceiver
          *.crit /var/log/syslog_debug.log
        • If events are missing and they are being transmitted via UDP, check the volume of events being received.  Messages sent via UDP are not guaranteed and congestion can mean more messages are dropped.  If congestion is an issue, try filtering the messages being sent to cut down on volume and changing the protocol to TCP.
        • Run rsyslog in debug mode using “rsyslog -d”.  
          Make sure that you’ve shut down the system rsyslog daemon.  To make it easier to analyze the output use either the “script” command or pipe the output through the “tee” command to save the output to a file.  Note that there is a “rsyslog -D” option, but this provides development level debug output and is usually overkill.

Conclusion

Syslog provides a standardized way to consolidate and prioritize messages across a wide variety of network devices and Unix/Linux operating systems.  While there are multiple versions of syslog that can provide advanced message handling options, basic configuration settings often meet most logging needs.

 

Want to learn more?

Download a FREE trial of Splunk  - Download a copy today and see how Splunk makes it simple to collect, analyze, and act upon the untapped value of big data.

 

Start Your Free 60 Day Trial of  Splunk Today!

 

Sign Up for the Blog

Heroix will never sell or redistribute your email address.