The Purpose of System Logging

Log files

• Contain detailed records of activity on a Unix system
• Can be used to track hardware and software problems
• Can be used to track configuration problems
• Can be used to compute system statistics
• Can help identify security problems

Security and logging

• Log files can be tampered with 
• It is possible to put a digital signature on your log files

Most of the logging on your system is configured in the /etc/syslog.conf file.

Daemons

Where did the word daemon come from?

The word was first used "by a British gentleman who was working on the CTSS programming staff at MIT during the early 1960s." (From Unix System Administration Handbook page 821 who got it from Jerry Saltzer at MIT, via Dennis Richie.)

From the Oxford English Dictionaly meaning "an attendant spirit that influences one's character or personality."

CTSS was the project to beget Multics which beget UNIX back in the 60s. CTSS, Compatible Time-Sharing System,  allowed the transition from a batch environment, to a time-sharing / interactive system.

• programs that run in the background running various system tasks
• not part of the kernel
• some are started at boot time
• some only run when requested

• Responsible for running other daemons
• inetd: manages internet processes such as ftp, telnet, finger
• init: the first process to run after system boot
• starts getty processes allowing users to login
• starts all processes for default run-level
• starts system logging daemons: syslogd and klogd
  
• portmapper: manages RPC (Remote Procedure Call) system used by nfs daemons, yp daemons

• Running scheduled programs
• crond

• Managing system tasks for the kernel
  

• kswapd or swapper: monitors page faults vs memory references and moves processes to swap space to prevent thrashing
• bdflush and kupdate: writes updates to disk. 

• bdflush writes memory buffers to disk when  memory is running low
• kupdate writes buffers to disk periodically

• Monitoring user actions
• authentication
• disk utilitzation
  

• The point being that all write status to log files!

Default system logging behavior

The messages file

• By default this is your primary log file

Format of /var/log/messages

• timestamp
• hostname
• name of program
• message text

The configuration file for system logging is /etc/syslog.conf
The logging daemon is syslogd

• watches for messages submitted by programs
• manages all calls to the syslog function
• writes log messages according to syslog.conf

One last thing before we look at the configuration file for syslog.

Viewing system Boot Messages with dmesg

dmesg

• dmesg displays the contents of the kernel ring buffer
• small area of memory that holds kernal messages
• when the area becomes full, the oldest messages are discarded
  
• these same messages scroll across your machine during system boot
• contains details about your system hardware and how it is initialized
  

$ dmesg | less

Linux version 2.4.7-10 (bhcompile@stripples.devel.redhat.com) (gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-98)) #1 Thu Sep 6 17:27:27 EDT 2001
BIOS-provided physical RAM map:
 BIOS-e801: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e801: 0000000000100000 - 000000000a000000 (usable)
Scanning bios EBDA for MXT signature
On node 0 totalpages: 40960
zone(0): 4096 pages.
zone(1): 36864 pages.
zone(2): 0 pages.
Kernel command line: auto BOOT_IMAGE=linux ro root=801 BOOT_FILE=/boot/vmlinuz-2.4.7-10
Initializing CPU#0
Detected 299.951 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 598.01 BogoMIPS
Memory: 157188k/163840k available (1269k kernel code, 5104k reserved, 90k data, 220k init, 0k highmem)

*Since the ring buffer will eventually get over written, RH saves a copy of your original boot messages in /var/log/dmesg.

$ less /var/log/dmesg

syslog.conf

This file is fairly difficult to configure so hang on
Each line consists of a selector <Tab> action

Sample line from syslog.conf

*.info;mail.none;authpriv.none            /var/log/messages

|_____||________||____________|           |________________|
selector;selector;selector                  action

*Multiple selectors on the same line are separted by semi-colons

Selectors consist of facility.priority

• Facility: which type of program is being selected
• Priority: severity of message submitted for logging
• must be chosen from a list of defined values
  

Priority	Meaning
emerg	Panic situations, could crash the system
alert	Urgent situations, cause a program to shutdown and may affect other programs
crit	Critical situations, may cause a program to shutdown
err	Error conditions occuring within a program
warning	Warnings about potential problems with a program
notice	Something that might need investigation, noteworhy event
info	Information about what a program is doing
debug	Used by programmers or sysadmins for debugging purposes
*	All priorities match
none	exclude all messages matching a certain facility

Actions

• write to a given file
• write to a given terminal
• write to all user terminals
• asterisk (*) in the action field refers to all users logged into the system
  
• write to log file on remote system
• consolidate important messages
• prevent tamporing of local log files
• enhances security
  

Ok, first make sure you have the package sysklogd :

% rpm -q sysklogd

Some basic rules of syntax for syslog.conf

• '*' indicates all in facility or priority
• *.emerg
• all messages of type emergency
  
• '*' in action field refers to all users on the system
• facility and priority are separated by a period
• keywork none excludes all messages maching a facility
• separate mulitple selectors with a ';'
• *.info;mail.none;authpriv.none
  
• usa a comma to separate mulitple facilities/priorities
• uucp,news.crit
• anything for news and uucp that is critical 
• selecting a priority indicates that all messages of that priority and higher are included
• the same messages can be logged to more than one place
• include the same selector on another line with a different action
• '!' excludes all priorities above a certain level
  

Suggestions for modifying  /etc/syslog.conf

Here are the lines you may want to include in you syslog.conf file. Comments begin with '#'.

* Please note logging will NOT work unless you use TABS not spaces  is the syslog.conf file.

# Everybody gets emergency messages
*.emerg                          *

# Log all messages for info level and higher to syslog file
# except mail and authpriv
*.info;mail.none;authpriv.none   /var/log/syslog

# Log all kernel messages to the new file /var/log/kernel
kern.*                           /var/log/kernel

# Log all authorization messages to secure
authpriv.*                       /var/log/secure

# Log all the mail messages in one place.

mail.*                           /var/log/maillog

# Log all logins to /var/log/loginlog

auth.*;user.*;daemon.none        /var/log/loginlog

# printer debug errors
lpr.debug                        /var/log/lpd-errs

# sudo -- if configured
local2.debug                     /var/log/sudo

# named -- if you are running it
daemon.*                         /var/log/named

# Log additional data to the Alt-F7 and Alt-F8 screens (Pseudo TTY 7 and 8) for real-time look

*.info;mail.none;authpriv.none  /dev/tty7
authpriv.*                      /dev/tty7
*.warn;*.err                    /dev/tty7
kern.*                          /dev/tty7
mail.*                          /dev/tty8

Now that the syslog.conf file is modified, you will need to make a few additional changes to the log files and their permissions.

• Touch each new log file and chmod the permissions:

# touch /var/log/syslog /var/log/kernel
# touch /var/log/named /var/log/lpd-errs /var/log/sudo
# chmod 700 /var/log/syslog /var/log/kernel
# chmod 700 /var/log/named /var/log/lpd-errs \
/var/log/sudo

# killall -HUP syslogd
or
# service syslod restart

Logging to a remote machine

• Syslog can be configured to accept log information from another machine.
• If an attacker compromises a machine and its log data, you will have a backup of the logs on another host.
• The backup logs can be compared to those on the compromised machine
• Protect your central log host server
  

• an attacker could overload your machine with og messages in a denial of service attack
  
• or your log messages could be intercepted by an attackers machine
• Be sure that UDP port 514 is blocked at your firewall!
• The syslog daemon has a sordid history of buffer overflows
• keep the sucker patched!
  

SYSLOGD_OPTIONS="-r -m 0"

3. estart the deamon on both machines

Log Management Utilities

Red Hat (and maybe other distributions) use the logrotate utility to rotate log files after a certain time period.

• done to keep log file size manageble.
• frees up space on hard disk
• old log files can be saved and compressed
  

You will need to add addtitional code to the logrotate config file to accomodate the new syslog and  kernel log files.

• Make sure you have the binary file, klogd, and check the path. 
• It is sometimes installed in /usr/sbin.
• You can use rpm -i to install the sysklogd rpm file. 

/etc/logrotate.d/syslog

• allows you to specify logfiles to rotate
• what command to execture after rotating
• compression
  

The file should look something like this:

/var/log/messages /var/log/secure /var/log/maillog \
/var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
# getting rid of errors from cat and kill commands

/var/log/kernel {
compress
postrotate
/usr/bin/killall -9 klogd
/sbin/klogd &
endscript
}

/var/log/syslog {
compress
postrotate
/usr/bin/killall -HUP syslogd
endscript
}

postrotate: execute commands between postrotate and endscript after rotating log file
sharedscripts: postrotate script only run once, not once for each log which is rotated

/etc/logrotate.conf

This file configures the frequency of rotatation. You should keep at least the last four weeks of rotated logs before deletion. It is recommended that you do NOT rotate the utmp and wtmp files . The are commented out below.

This is a good start:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

## no packages own lastlog or wtmp -- we'll rotate them here
#/var/log/wtmp {
#    monthly
#    create 0664 root utmp
#    rotate 1
#}

#/var/log/lastlog {
#    compress
#    monthly
#    rotate 1
#}

To check the status of your rotated logs read this text file: /var/lib/logrotate.status.

logrotate state -- version 2
"/var/log/messages" 2002-3-24
"/var/log/secure" 2002-3-24
"/var/log/maillog" 2002-3-24
"/var/log/spooler" 2002-3-24
"/var/log/boot.log" 2002-3-24
"/var/log/cron" 2002-3-24
"/var/log/xferlog" 2002-3-24
"/var/log/wtmp" 2001-11-1
"/var/log/rpmpkgs" 2002-3-24
"/var/log/samba/smbd.log" 2002-3-17
"/var/log/samba/oaxaca.log" 2002-3-24
"/var/log/samba/dstreet.log" 2001-11-25
"/var/log/kernel" 2002-3-24
"/var/log/syslog" 2002-3-24
"/var/log/samba/pavones.log" 2001-11-15
"/var/log/samba/smbmount.log" 2002-3-24
"/var/log/samba/colima.log" 2002-3-24
"/var/log/samba/mammoth.log" 2002-3-15
"/var/log/samba/ponto.log" 2002-3-10

*Keep an eye on your disk utilization in /var

LogWatch: log monitoring system

• uses standard search utilities and perl script to search for lines in log files
• you specify conditions to watch
• mails warning messages to root
  
• freely available, comes with RH 7.2 distribution