syslog Overview

syslog Overview
Last modified08-Jul-2003
Comments totbird
On this page:
Background
syslog Architecture
Quick Start
Troubleshooting
For More Information
The goal of this document is to describe the use of UNIX syslog asa logging and auditing mechanism for operating systems and applications; toprovide information on customizing your syslog configuration for localor remote log collection; and to document some possible syslog clientconfiguration, for monitoring the health and well-being of a UNIX computer.We‘llalso establish naming conventions and sample network topologies for the other syslog configuration documents.
syslog Architecture
syslog is a consolidated audit mechanism for UNIX operating systemand application messages. It’s designed to make it easier for applicationdevelopers and system analysts to incorporate logging into their projects, andto provide system administrators with a single point of management for collecting,distributing and processing audit data. In most cases, a lot of different peopleinfluence the data that eventually ends up in a system log. The developer whowrote the operating system component or application starts the ball rolling,by deciding
what events will create a log entry
how significant the events are
what kinds of information will be contained in the log entry
how the information will be formatted
whether or not the end user has any control over the quantity or types of logs that get generated.
The local system or application administrator gets to decide what actions toapply to the log data, based (in the default version of syslog ) onthe system component or application that generated the log entry, and the significanceassigned to the log entry by its developers. It‘s a pretty coarse-grained system,which is why there are a lot of alternatives for collecting and acting on syslog data (see theLog Analysis website‘s listing ofsyslogreplacements).The administrator can choose amongst the following options:
write the log entry to a local file, which may be specific to that application or may be a monolithic repository
write the log entry to a remote system running syslog (we‘ll refer to such a remote server as a loghost) throughout these configuration documents
broadcast the log entry to any or all users logged in on the host computer
write the log entry on the system console
The architecture is designed to maximize flexibility and ease-of-use. But thereare few, if any, guidelines available for developers or system administrators,which means that implementing an enterprise logging system involves a lot oftrial and error.
The behavior of syslog -- how data from different applications andsubsystems on the host operating system, and of different levels of severity,is stored or distributed -- is controlled on a per machine basis by the file/etc/syslog.conf. This configuration file consists of multiplelines like this:
mail.info /var/log/maillog
The format of configuration lines is:
facility.level action
For historical reasons, the key, not a simple blank space, is used to define white space between the selectoron the left side of the line and the action on the right side. Throughoutthe Log Analysis configuration documents, we‘ve used the to remind you of this -- but of course, when you look at the file, you‘llonly see white space.
The facility is the application or operating system component thatgenerates a log message. The level is the severity or significanceof the message that’s been generated. The action defines what’sdone with any newly-arrived message that matches the facility and level. Thiscombination of facility and level, referred to as the selector , allowssystem administrators to customize message handling, based on which parts ofthe system are generating data, and how critical the data is.
The standard UNIX syslog facilities are
kern – kernel
user – application oruser processes (this is the default if the application sending a message doesnot specify the facility)
mail/news/UUCP/cron –electronic mail/NNTP/UUCP/cron subsystems
daemon – system daemons
auth – authenticationand authorization related commands
lpr – line printer spoolingsubsystem
mark – inserts timestampinto log data at regular intervals
local0-local7 – 8 facilitiesfor customized auditing
syslog – internal messagesgenerated by syslog itself
authpriv – non-systemauthorization messages
* -- on most versions of UNIX,refers to all facilities except mark
syslog levels are nominally defined as:
emerg – system is orwill be unusable if situation is not resolved
alert – immediate actionrequired
crit – critical situations
warning – recoverableerrors
notice – unusual situationthat merits investigation; a significant event that is typically part of normalday-to-day operation
info – informationalmessages
debug – verbose datafor debugging
Unfortunately, there aren’t any clearly defined standards dictating howlevels are determined for a particular application or computer subsystem. Alltoo often, the best way to figure out how any given application determines itslevels is to set it to log everything. Then read what gets logged as you performdifferent actions (logging in, changing configurations, etc).
syslog can perform these actions:
filename – write messageto the specified file on the local machine
@hostname or @ipaddress– forward message to remote loghost
user1,user2,… -- writemessage to consoles of users named in list, if user is logged-in
* -- write message to all logged-in users
When a selector is defined in /etc/syslog.conf,it causes all messages from the specified facility and the specified level orhigher to be subject to the defined action. So setting the following line:
kern.crit root,@loghost
will cause messages matching kern.crit,kern.alert and kern.emergto be written to the remote loghost and to the root user’s console (ifroot is logged in).
As a sample of how facilities, levels and actions can be used to customizea syslog installation, here’s a sample /etc/syslog.conffrom RedHat Linux v7.1. The default configuration has been modified towrite all messages of crit andhigher level to the root userconsole, the remote loghost, and a local file:
# Log all kernel messages to theconsole.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of levelinfo or higher.
# Don‘t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in oneplace.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages,plus log
# them on another machine.
*.emerg *
# Save mail and news errors of levelerr and higher in a
# special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# send messages level crit and higherto root console
# and loghost
*.crit /var/log/messages,@loghost,root
Remember that those empty spaces between the selector and the action are tabs,not spaces.
To verify that your syslog configuration is correct, use the UNIX utility loggerto write arbitrary messages. The -pflag allows you to set the facility and level. Typing the following command:
# /usr/bin/logger –p local0.warning“This space intentionally left blank”
will cause the message This spaceintentionally left blank to be sent to the local system‘s syslog.
Quick Start
When you‘re getting started with centralized logging and monitoring in yournetwork, you may choose either to look at the data being produced by the defaultlogging configuration on your devices, or to configure things to produce lotsand lots of log data so you can learn about your network‘s idiosyncracies ingory detail. Starting off with one or two computers in "log everything"mode gives you the chance to get comfortable with reading log files, so that‘swhat we show here. After the operating system logging configuration is done,the next step is to configure your applications – like Apache Web server,FTP servers, or whatever else is running on the system that doesn’t logto syslog by default – to send data to the local syslog .There are links to configuration documents for a variety of operating systemsand applications in theFor More Information sectionbelow.
The following instructions use the word loghost to refer to the IPaddress or the valid DNS hostname of the central syslog server. Pleasesubstitute the appropriate address or name for your site. The entries in the following lines are a reminder to use the characterto delimit white space between the selector and the facility, not blank spaces.White spaces in the following lines are added to improve the legibility of theinformation, and should not be included in your working syslog configurationfile.
To send all syslog data from a Solaris hostto the loghost, add the following line to /etc/syslog.conf:
*.debug @loghost
and then restart syslog :
/etc/init.d/syslog stop ; /etc/init.d/syslogstart
To send all syslog data from an OpenBSD host to the loghost, add the followingline to /etc/syslog.conf:
*.* @loghost
and then restart syslog :
kill –SIGHUP `cat /var/run/syslog.pid`
To send all syslog data from a Linux host tothe loghost, add the following line to/etc/syslog.conf:
*.* @loghost
and then restart syslog :
/etc/rc.d/init.d/syslog restart
On Nokia IPSO, the proprietary FreeBSD-based operating system used by NokiaFW-1 and RealSecure appliances, syslog configuration is controlledthrough the IPSO Voyager interface. Bring up Voyager, and select the Configradio button. Scroll through the list of options until you get to the SystemConfiguration option; under System Configuration ,select System Logging . Add the loghost‘s IP address as thenew remote IP address to log to. Apply and then save your changes.
To integrate Windows NT and Windows 2000 into your syslog infrastructure,please see the separateLogAnalysis link covering these operating systems.
Troubleshooting syslog
First, the syslog daemon must be running on the local computer, andthe loghost must be on the network and ready to accept data. The local machinemust have network access to the loghost. The local /etc/syslog.confmust contain the appropriate facilities and levels. If required, applicationsmust also be configured to log to syslog . Here are some suggestionsfor checking each of these requirements.
1) Is syslog running on the local machine? If not, the most likelyculprit is a stray whitespace in the configuration file. Verify that /etc/syslog.confuses and not whitespacesto separate selectors from actions. (The syslog daemon dies a lonelyand depressing death without those tabs.)
2) Is the loghost on the network? Is it accepting data from other remote systems?
3) Does the local machine have network connectivity to the loghost? Verify networkconnectivity using ping or yourown favorite network troubleshooting tools. Network sniffers (such as snoopor tcpdump, e.g. tcpdump-nli eth0 udp port 514) are invaluable here. If you can’t establishconnectivity, are the machines on separate subnets on your network that requirerouting changes? Are there access control lists or firewall rules restrictingtraffic between the devices?
4) Is /etc/syslog.conf on thelocal machine configured to send the appropriate data to the loghost? Test thisby sending the “This space intentionally left blank” message tothe loghost, using the loggerutility described above.
5) Are your applications configured to send data to syslog ? This canbe tricky to test – there’s no one mechanism for generating a logmessage that’s guaranteed to work for all applications – but youcan try, for instance, stopping and starting a server, changing a configuration,or using the application, to generate log data. If the data is not appearingin syslog , check your vendor documentation to see if there’ssupport for syslog built-in. If the application does not support syslog by default, you can usually can use logger to get data into syslog . If the application produces binarylogs, we need to find some way to decode them before invoking logger.
For More Information
For more information on syslog , including a lot more sample configurations,see
Nemeth et al., UNIX System Administration Handbook (Prentice Hall)
Frisch, A., Essential System Administration (O‘Reilly)RFC 3164, on the syslog protocol
the rest of theLog Analysis Web site
theLog Analysis Mailing List