Configure logs

In the last chapter Update Linux server we updated the operating system.

In this chapter we will look at where the log files are stored and how we can read them. Admittedly, this is not the most exciting chapter and the minor reconfiguration we will do will not bring any security or performance optimization. You could just skip this chapter and your server would run just as well. However, every administrator should know about log files. If something doesn't work, they are the first place to go for troubleshooting.

The log file, which logs logins and logouts, is additionally written to by background processes. This makes it confusing and difficult to read. Therefore, we will make a setting to record the "uninteresting" events separately. This way the entries are neatly separated, which can be analyzed more easily.

Log files

The central location for log files is /var/log.


__$ sudo ls -l /var/log

Output:


total 11264
-rw-r--r--  1 root      root              31252 Jan  4 22:20 alternatives.log
drwxr-xr-x  2 root      root               4096 Jan  4 22:19 apt
-rw-r-----  1 syslog    adm             1294710 Jan  4 22:23 auth.log
-rw-r--r--  1 root      root             104003 Jan  3  2021 bootstrap.log
-rw-rw----  1 root      utmp            1100928 Jan  4 20:40 btmp
-rw-r--r--  1 syslog    adm              474428 Jan  4 22:28 cloud-init.log
-rw-r-----  1 root      adm               23633 Jan  4 22:28 cloud-init-output.log
drwxr-xr-x  2 root      root               4096 Jan  3  2021 dist-upgrade
-rw-r--r--  1 root      adm               61927 Jan  4 22:28 dmesg
-rw-r--r--  1 root      adm               61624 Jan  4 22:17 dmesg.0
-rw-r--r--  1 root      adm               16365 Jan  4 15:02 dmesg.1.gz
-rw-r--r--  1 root      adm               15996 Jan  4 15:00 dmesg.2.gz
-rw-r--r--  1 root      adm               16027 Jan  4 19:35 dmesg.3.gz
-rw-r--r--  1 root      adm               16092 Jan  4 19:34 dmesg.4.gz
-rw-r--r--  1 root      root             497519 Jan  4 22:21 dpkg.log
-rw-r-----  1 root      adm               20285 Jan  4 10:53 fail2ban.log
-rw-r--r--  1 root      root              32064 Jan  4 21:31 faillog
drwxr-xr-x  3 root      root               4096 Jan  4 12:09 installer
drwxr-sr-x+ 3 root      systemd-journal    4096 Jan  4 12:14 journal
-rw-r-----  1 syslog    adm             2322573 Jan  4 22:23 kern.log
drwxr-xr-x  2 landscape landscape          4096 Jan  4 12:18 landscape
-rw-rw-r--  1 root      utmp             292584 Jan  4 22:23 lastlog
-rw-r-----  1 syslog    adm                2627 Jan  4 22:12 mail.err
-rw-r-----  1 syslog    adm              303319 Jan  4 22:23 mail.log
drwx------  2 root      root               4096 Jan  3  2021 private
-rw-r-----  1 syslog    adm             1547818 Jan  4 22:23 syslog
-rw-r-----  1 syslog    adm             1683517 Jan  4 00:00 syslog.1
-rw-------  1 root      root                  0 Jan  3  2021 ubuntu-advantage.log
-rw-------  1 root      root                157 Jan  4 22:20 ubuntu-advantage-timer.log
-rw-r-----  1 syslog    adm             1723687 Jan  4 22:23 ufw.log
drwxr-x---  2 root      adm                4096 Jan  4 12:14 unattended-upgrades
-rw-rw-r--  1 root      utmp              83328 Jan  4 22:23 wtmp

Log files are typically divided into four categories:

Application
Event
Service
System

The SSH service logs to the file /var/log/auth.log. Let's have a look in there:


__$ sudo less /var/log/auth.log

At the beginning of each line is the date and at the end log message. As you may have noticed, there are also login events that cannot be from you. These contain the phrase pam_unix(cron:session) and come from internal processes. But more about that later.

For example, we can filter the output with grep and a search word. The next command shows all lines where the phrase user tom is mentioned:


__$ sudo less /var/log/auth.log | grep "user tom"

Another example with the phrase cron:session:


__$ sudo less /var/log/auth.log | grep "cron:session"

Output the latest events and read them live

With tail a certain number of the last lines can be output. For example the last 10:


__$ tail -10 /var/log/auth.log

Log files can be read live. This helps to not have to constantly close and reopen the log file. New entries or changes are output with the parameter -f:


__$ tail -f /var/log/auth.log

The key combination CTRL+c terminates the output.

At the moment, hardly anything is happening. This will be more interesting later when we observe the logs from the server service.

Logrotate

After a while, a lot of data ends up in the logs. The files are bloated and consume corresponding hard disk space. To counteract this, the log files are emptied at certain intervals, but of course copied and archived beforehand. The archives get the original file name and additionally a sequential number. This process is called "logrotate". Old log archives are then eventually deleted. Each installed package or service that logs can store its own configuration for this at the logrotade.d service and design the process individually.

Check version of logrotade:


__$ sudo logrotate --version

The configuration files are located in the folder of the same name /var/logrotate.d. There we will create another Logrotate configuration, for the SSH service.


__$ sudo ls -l /etc/logrotate.d

Excursus Cron

To make the next section more understandable, here is a brief explanation of crons. A cron daemon is a service that runs in the background and triggers processes at specified times. Also called a "cron job." Hourly, daily and weekly intervals are configured by default. Each user can have their own cron table. The cronjobs from the root user are in the /etc/crontab file. Let's have a quick look:


__$ sudo cat /etc/crontab

For me, the crontab table looks like this:


# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

You probably have other numbers, because they are set at the time of the Linux installation. The lower lines or the non-commented lines represent a table. With the first columns, the interval can be set and at the end are the commands that are then triggered. In my case: the hourly cronjob at the 17th minute (around the clock) with the command cd / && run-parts --report /etc/cron.hourly. Any executable in the etc/cron.hourly directory will be run with run-parts. Let's take a quick look in the folder:


__$ sudo ls -l /etc/cron.hourly

There are probably no executable files at the moment.

The situation is different for daily crons:


__$ sudo ls -l /etc/cron.daily

Redirect cron sessions

It happens that applications launched at cronjobs authenticate themselves to the system via the PAM interface. They generate the cron:session log entries mentioned above. They are actually uninteresting as an entry and just make the /var/log/auth.log messy. Therefore, we will redirect them to another file.

Let's first create a log file for the cron session events with touch:


__$ sudo touch /var/log/pam-cron-session.log

We change the file permissions with chmod to 0640:


__$ sudo chmod 0640 /var/log/pam-cron-session.log

Since we are the owner of the file, there would only be problems if background processes without permission wanted to write in there. As new owner we appoint the system user syslog from the group adm. We change owner and group with chown:


__$ sudo chown syslog:adm /var/log/pam-cron-session.log

To catch said log events, we place a configuration file at the rysylog.d service. This service evaluates events and writes log messages to appropriate log files. Let's create a new file (/etc/rsyslog.d/30-cron.conf) with nano:


__$ sudo nano /etc/rsyslog.d/30-cron.conf

And write in the following:

Excerpt from /etc/rsyslog.d/30-cron.conf


# Cronjob-Spam redirect
:msg, contains, "pam_unix(cron:session):" -/var/log/pam-cron-session.log
& stop

Save and close (CTRL+s, CTRL+x).

The configuration says: if pam_unix(cron:session): is in a log message (:msg, contains), then write it to the file /var/log/pam-cron-session.log. The hyphen -/ first caches or collects the events before writing them. This reduces the write accesses on the hard disk.

Let's optimize this further with a logrotate. As discussed above we create a configuration file for the logrotate.d service:


__$ sudo nano /etc/logrotate.d/pam-cron-session

And write in the following:

/etc/rsyslog.d/30-cron.conf


/var/log/pam-cron-session
{
  rotate 4
  weekly
  missingok
  notifempty
  compress
  delaycompress
  sharedscripts
  postrotate
    invoke-rc.d rsyslog rotate > /dev/null
  endscript
}

Save and close.

This will rotate weekly (weekly) and no more than 4 archives will be created. The fifth archive, which is the oldest, is deleted.

All that remains is to restart the rsyslog.d service to apply the new configuration:


__$ sudo systemctl restart rsyslog.service