Silicon Soul

Logs are important to help us understand our GNU/Linux systems. They provide an avenue to track and audit system activity, which can be valuable for security and performance purposes.

There are several GNU/Linux logging implementations. The implementation that you encounter will depend on the GNU/Linux distribution that you are using.

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

systemd Logging

In a traditional GNU/Linux logging system, the syslog daemon accepts log messages on UDP port 514 or the /dev/log socket. From there, it usually writes the messages to text files or forwards them to other hosts where they are written to files.

In a systemd system, background services can write log messages to their standard error output channel and systemd will arrange for them to be passed to the logging service. With systemd, log files are not text files. Instead, log messages are written to a binary database that can then be queried according to diverse criteria.

systemd-journald, the systemd logging daemon, logs to a binary file and annotates each log entry with metadata. Anything that a daemon writes to the console is saved as a proper entry in the database.

Modern syslog implementations (e.g., Rsyslog or Syslog-NG) are capable of writing log messages to a database, but it is your responsibility to come up with a suitable database schema, to configure Rsyslog or Syslog-NG, and to develop software that allows you convenient access to the log messages. By default, systemd includes all of this functionality.

The systemd journal is not confined to textual log messages. For example, it is possible to store core dumps of crashed programs in the journal, as long as they are not oversized.

systemd's log system can interoperate with the traditional logging approach. If desired, it logs messages that arrive on UDP port 514 or /dev/log, and can pass messages on to a traditional syslog daemon (or modern re-implementation).

systemd enables you to see the most recent log messages by a service using the systemctl command:

systemctl status ex_service

Overlong lines are shortened, but to see them in full, use the -l (--full) option with the command (e.g., systemctl -l status ssh).

The systemd Journal

There are some disadvantages of traditional logging systems' use of text files for log files:

• They can be difficult to search.
• Targeted evaluation is either tedious or requires additional non-standardized software.
• There is no type of cryptographic protection against the manipulation of log entries, and the amount of information that can be written to the log is limited.

The systemd journal is an integrated part of systemd. In the simplest case, systemd uses a limited-size ring buffer in /run/log/journal/ to store a certain number of log messages in RAM (which is sufficient if you want to pass the messages on to a traditional log service).

To take advantage of all journal features, you should ensure that the log messages are permanently stored on disk. This is done by creating the directory for storage (many distributions, like Debian and Fedora, are already configured to use /var/log/journal/ for a persistent journal).

# mkdir -p /var/log/journal/
# systemctl --signal=USR1 kill systemd-journald

The SIGUSR1 signal gets systemd to transfer the RAM-based journal (i.e., /run/log/journal/) to the new file on disk (i.e., /var/log/journal/).

systemd-journald and /etc/systemd/journald.conf

The journal is configured by means of the /etc/systemd/journald.conf file. The [Journal] section of this file contains the Storage parameter, which can assume any of the following values:

• auto - Similar to persistent, but the existence of the /var/log/journal/ directory determines whether a persistent journal is written. If the directory does not exist, the volatile journal is used.
• none - No log messages are stored in the journal. You can still pass messages to a traditional syslog service.
• persistent - Log messages are preferably stored on disk (/var/log/journal/). The directory will be created if it does not exist. During early boot and if the disk is not writable, systemd falls back onto /run/log/journal/.
• volatile - Log messages are only stored in RAM, even if /var/log/journal/ exists.

Interesting parameters besides Storage include:

Compress

Specifies whether log files (or those exceeding a certain size) are transparently compressed. The default value is yes.

RateLimitInterval, RateLimitBurst

These parameters are supposed to make it more difficult to flood the log with messages. If a service produces more than RateLimitBurst messages during a period of time given by RateLimitInterval, then all further messages until that period of time is over are ignored (the log will contain only one message detailing the number of ignored messages).

By default, the limit is 10000 messages in 30 seconds. If you set either of the parameters to zero, the limitation is lifted.

Seal

Lets you ensure that persistent journal files are protected against clandestine manipulation by means of a cryptographic signature. You will need to provide an encryption key.

SyncIntervalSec

Specifies how often the journal will be synced to disk. The journal is always immediately saved after a message of priority crit (or above) has been logged. As long as no such message arrives, journald will wait for the interval specified by SyncIntervalSec before saving it again. The default value is 5 minutes.

systemd-journald tries to make sensible use of available disk space, i.e., new messages are normally appended to the journal, but if a certain upper limit for the size of the journal is reached, it tries to remove old log messages.

The following /etc/systemd/journald.conf parameters can be be used to set limits:

RuntimeMaxUse

Describes how much space the journal may take up under /run/log/journal/.

SystemMaxUse

Describes how much space the journal may take up under /var/log/journal/.

RuntimeKeepFree, SystemKeepFree

Determine how much space must be kept free on the file system in question.

systemd-journald takes into account all of the above parameter values and confines itself to the minimum size dictated by both the MaxUse and KeepFree parameter values. The Runtime values are used when the system is booting or no persistent journal is used. The System values apply to the persistent journal if the system has been booted far enough. When determining the space used by the journal, only files whose names end in .journal are considered.

Space amounts may be specified in bytes or via one of the following binary units:

• K
• M
• G
• T
• P
• E

If there is less space available when systemd-journald starts than the KeepFree values dictate, the limit is reduced even further such that space for other material remains.

Like the logrotate command did in traditional logging systems, systemd rotates the journal to make room for new messages. To do so, the available space is subdivided into a number of files, so that the oldest can be continually discarded. This rotation is transparent to users because systemd-journald does it of its own accord when required and journalctl always evaluates the full journal, no matter how many files it consists of.

The subdivision is governed by the SystemMaxFileSize and RuntimeMaxFileSize parameters within the /etc/systemd/journald.conf file. These parameters specify how large individual journal files may become.

Also, you may make the log file rotation depend on time via the following parameters:

MaxFileSec

Determines the maximum time period before systemd starts a new log file (usually the size-based rotation is adequate). The default value is 1month (0 means unlimited).

MaxRetentionSec

Specifies an upper limit for how long old log messages are kept around. Setting this to 0 disables the mechanism.

In /etc/systemd/journald.conf, you can also configure log forwarding to a traditional syslog system. To do so, set ForwardToSyslog to yes:

[Journal]
ForwardToSyslog=yes

Access Rights

As the root user, you get to see the complete systemd journal log. As an ordinary user, you will only be able to peruse your own log, specifically the messages submitted by programs that you started yourself (or that the system started on your behalf).

If you want to have full access to the log, even as an ordinary user, you will need to ensure that you are part of the adm or wheel group (this will depend on the distribution that you are using).

For example, on Debian, you can add yourself to the adm group like so:

# usermod -a -G adm ex_username

After you restart your system, this change will be effective.

journalctl

The journalctl command is used to direct queries to the journal:

$ journalctl
-- Logs begin at Mo 2015-07-27 13:37:14 CEST, end at Mo 2015-07-27
22:20:47 CEST. --
Jul 27 13:37:14 tails systemd-journal[138]: Runtime journal is using 4.
Jul 27 13:37:14 tails systemd-journal[138]: Runtime journal is using 4.
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpuset
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpu
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpuacct
Jul 27 13:37:14 tails kernel: Linux version 3.16.0-4-amd64 (debian-kern
Jul 27 13:37:14 tails kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.

The output will strongly resemble what would be found in /var/log/messages, but includes various improvements:

• The log is displayed using your favorite display program for text files. This is determined by the value of the SYSTEMD_PAGER environment variable, failing that the value of PAGER, and failing that, less.

  Using the SYSTEMD_LESS environment variable, you can specify options for less (if you do not use the system default, this variable is ignored). Alternatively, you can also put options directly into SYSTEMD_PAGER for any pager program, including less.

• If you invoke journalctl with the --no-pager option, or set SYSTEMD_PAGER to cat or an empty string, the output will not be displayed page by page.

• The output includes all accessible log files, including rotated ones.
• Time stamps are in the local (zone) time, not UTC.
• Log messages of priority notice or warning are displayed in bold face.
• Log messages of priority error (or higher) appear in red.

Real-Time Journal Monitoring

To follow new messages being written to the journal, use journalctl's -f (--follow) option:

journalctl -f

This will display ten lines of output before journalctl waits for further messages to arrive.

As with the tail command, you can set the initial number of lines shown with journalctl using the -n (--lines=) option.

Services and Priorities

journalctl's -u (--unit=) option can be used to restrict the output to those log messages written by a specific systemd unit:

$ journalctl -u ssh
-- Logs begin at Mo 2015-07-27 13:37:14 CEST, end at Di 2015-07-28
09:32:08 CEST. --
Jul 27 13:37:23 tails sshd[428]: Server listening on 0.0.0.0 port 22.
Jul 27 13:37:23 tails sshd[428]: Server listening on :: port 22.
Jul 27 13:56:50 tails sshd[912]: Accepted password for amnesia from 192.16
Jul 27 13:56:50 tails sshd[912]: pam_unix(sshd:session): session opened

Instead of a specific unit name, you can also provide journalctl a shell search pattern to include several units, or use several -u options.

To only display messages of certain priorities, use the -p (--priority=) option. This takes either a single numerical or textual priority and limits the output to messages of that priority or above. Alternatively, you can specify a range to only see those messages whose priority is in that range:

journalctl -p ex_priority_1..ex_priority_2

The -u and -p options can be combined. For example, journalctl -u nginx -p err displays all error messages (or worse) from the NGINX web server.

The -k (--dmesg) option limits the output to messages logged by the operating system kernel. This considers only messages written since the last system boot.

Time

Log output can be filtered by time. The -S (--since=) and -U (--until=) options let you specify a date or time in the YYYY-MM-DD hh:mm:ss format, and only messages written since or until that point in time will be output.

You can leave off the time value, in which case a time of 00:00:00 is assumed. If you just leave off the seconds, :00 is implied. If you leave off the date (in which case you will need to specify a time, with or without seconds), journalctl will assume today as the date.

The yesterday, today, and tomorrow keywords stand for 00:00:00 yesterday, today, and tomorrow, respectively.

Relative time specifications are also allowed. For example, -30m stands for half an hour ago and 1h stands for in one hour.

Every system boot is assigned a unique identifier, and you can limit your search to the part of the journal between one boot and the next. In the simplest case, using journalctl's -b (--boot=) option will only consider messages from the current boot:

journalctl -b -u nginx

With the --list-boots option, journalctl will output a list of boot identifiers to be found in the journal, together with the periods of time for which there are log entries:

$ journalctl --list-boots
-1 30eb83c06e054feba2716a1512f64cfc Mo 2015-07-27 22:45:08 CEST—
Di 2015-07-28 10:03:31 CEST
0 8533257004154353b45e99d916d66b20 Di 2015-07-28 10:04:22 CEST—
Di 2015-07-28 10:04:27 CEST

You may refer to specific boots by passing their index number to the -b option (e.g., 1 stands for the chronologically first boot in the log and 2 for the second boot) or the negative offset in the first column of the output of journalctl --list-boots (e.g., 0 refers to the current boot and -1 stands for the boot prior to the current boot).

The 320 character alphanumeric boot ID from the second column of journalctl --list-boots can be specified to search the journal for that boot only. A positive or negative offset can be appended to the boot ID to identify boots before or after it, respectively.

For example, if 8533257004154353b45e99d916d66b20 represented the second boot ID in the log, the following would be another way of saying journalctl -b 1:

journalctl -b 8533257004154353b45e99d916d66b20-1

Arbitrary Search Operations

If you specify a path name as an argument, journalctl tries to act on it in a reasonable fashion:

• If it refers to an executable file, journalctl looks for journal entries made by that program.
• If it refers to a device file, journalctl looks for entries concerning the device in question.

These search operations are special cases of a more general search mechanism offered by the journal. systemd does log more information than the traditional syslog mechanism. This can be demonstrated by invoking journalctl with the -o verbose (--output=verbose) option:

$ journalctl -o verbose -u sshd
Mo 2015-07-27 13:37:23.580820 CEST [s=89256633e44649848747d32096fb42
68;i=1ca;b=30eb83c06e054feba2716a1512f64cfc;m=11a1309;t=51bd9c6f
8812e;x=f3d8849a4bcc3d87]
PRIORITY=6
_UID=0
_GID=0
_SYSTEMD_SLICE=system.slice
_BOOT_ID=30eb83c06e054feba2716a1512f64cfc
_MACHINE_ID=d2a0228dc98041409d7e68858cac6aba
_HOSTNAME=tails
_CAP_EFFECTIVE=3fffffffff
_TRANSPORT=syslog
SYSLOG_FACILITY=4
SYSLOG_IDENTIFIER=sshd
SYSLOG_PID=428
MESSAGE=Server listening on 0.0.0.0 port 22.
_PID=428
_COMM=sshd
_EXE=/usr/sbin/sshd
_CMDLINE=/usr/sbin/sshd -D
_SYSTEMD_CGROUP=/system.slice/ssh.service
_SYSTEMD_UNIT=ssh.service
_SOURCE_REALTIME_TIMESTAMP=1437997043580820

The first line in the output above is a time stamp for the message together with a cursor. The cursor identifies the message inside the journal and is needed, for example, to store log entries on remote computers.

Subsequent lines are journal fields that refer to the message in question. Field names without a leading underscore derive from information submitted by the logging program (i.e., the program filing the log entry) and, as such, are not necessarily reliable (the program could, for example, misrepresent its Process ID or name in SYSLOG_IDENTIFIER). Field names with a leading underscore are supplied by systemd and cannot be manipulated by the logging program.

PRIORITY, SYSLOG_FACILITY, SYSLOG_IDENTIFIER, SYSLOG_PID, and MESSAGE derive from the syslog protocol and are self-explanatory. _BOOT_ID is the identifier of the current boot and _MACHINE_ID identifies the logging system according to its entry in /etc/machine-id.

_CAP_EFFECTIVE specifies the special capabilities of the logging process and _TRANSPORT describes how the message reached systemd (apart from syslog, common sources are stdout for messages that the program wrote to its standard output or standard error output, or kernel for messages submitted by the operating system kernel).

_COMM, _EXE, and CMDLINE all describe the command being executed. _SYSTEMD_SLICE and _SYSTEMD_CGROUP specify where in systemd's internal process management the logging process may be found.

More detailed information is available by running man 7 systemd.journal-fields.

These fields can be searched by specifying them on journalctl's command line:

journalctl _HOSTNAME=enceladus _SYSTEMD_UNIT=nginx.service

• Search terms using different fields are implicitly joined using AND. If the same field appears in several search terms, these are implicitly joined using OR.
• There is also an explicit OR. For example, journalctl _HOSTNAME=enceladus _UID=80 + _HOSTNAME=deimos _UID=90 shows all processes with the User ID (UID) 80 on the host enceladus, as well as all processes with the UID 90 on the host deimos (this will only work if you consolidate both of these journals on your system).
• You can freely combine these search terms with options, e.g., to set up time limits or save typing, journalctl -u nginx _HOSTNAME=enceladus.

If you cannot remember which values a search term can assume, you can ask the journal using journalctl's -F (--field=) option:

$ journalctl -F _SYSTEMD_UNIT
session-2.scope
udisks2.service
session-1.scope
polkitd.service
dbus.service
user@1000.service

Command line completion works for field names and values.

Commands

Useful systemd logging commands include:

ls -l /run/log/journal/

View a long listing of the runtime journal directory.

ls -l /var/log/journal/

View a long listing of the persistent journal directory.

less /etc/systemd/journald.conf

View the contents of the journald configuration file. After any changes are made to this file, run # systemctl restart systemd-journald to make them effective.

journalctl Commands

The journalctl command is used to query the systemd journal. If called without parameters, it will show the full contents of the journal, starting with the oldest entry collected.

To view journal entries for a specific binary, you can provide the binary's absolute path to journalctl:

journalctl ex_binary_absolute_path

Helpful journalctl options include:

-b, --boot=

Show messages for the current boot.

This option can be provided a specific boot number relative to the current boot (e.g., -1 for the previous boot from the current boot, which is 0) or a boot ID (caf0524a1d394ce0bdbcff75b94444fe) as an argument to only show messages for that boot.

--disk-usage

Show the current disk usage of all journal files.

-e, --pager-end

View the end of the journal.

-f, --follow

Show only the most recent journal entries and continuously print new entries as they are appended to the journal.

-k, --dmesg

Show only kernel messages. This is an alternative to dmesg | less.

--list-boots

Show a tabular list of boot numbers (relative to the current boot), their IDs, and the timestamps of the first and last message pertaining to the boot.

-n ex_number, --lines=ex_number

Show the most recent journal events and limit the number of events to ex_number.

-o ex_formatting_option, --output=ex_formatting_option

Controls the formatting of the journal entries that are shown.

-o takes one of the following options:

• cat
• export
• json
• json-pretty
• json-sse
• json-seq
• short
• short-full
• short-iso
• short-iso-precise
• short-precise
• short-monotonic
• short-unix
• verbose
• with-unit

-p ex_priority_level, -p ex_priority_level..ex_priority_level, --priority=ex_priority_level, --priority=ex_priority_level..ex_priority_level

View journal entries logged at a specific priority level/priority range or higher priority.

Show/Hide Log Priority Levels

Log Priority Levels

Level	Tag	Meaning
0	emerg	Emergency: The system is unusable.
1	alert	Alert: Immediate action is needed to prevent a failure.
2	crit	Critical: The system has reached a critical level.
3	err	Error: Some part of the system encountered an error.
4	warn	Warning: Something happened that may have been processed incorrectly warning.
5	notice	Notice: Not an error condition, but may need special handling.
6	info	Informational: A normal log entry about a routine event.
7	debug	Debug: A message about internal state that is for debugging problems.

-r, --reverse

View the journal in reverse order (i.e., newest entries shown first).

-S 'YYYY-MM-DD hh:mm:ss', --since='YYYY-MM-DD hh:mm:ss'

View journal entries since a specific time. Can be used with -U (--until=).

If the date is omitted, the current date is assumed. If the time component is missing, midnight is assumed. If seconds is left off, 00 is used.

Relative values can be used, e.g., --since yesterday, --since today, --since '1 hour ago'.

-U 'YYYY-MM-DD hh:mm:ss', --until='YYYY-MM-DD hh:mm:ss

View journal entries until a specific time. Can be used with -S (--since=).

If the date is omitted, the current date is assumed. If the time component is missing, midnight is assumed. If seconds is left off, 00 is used.

Relative values can be used, e.g., --until yesterday, --until today, --until '1 hour ago'.

-u ex_unit, --unit=ex_unit

Show messages for the specified systemd unit. This parameter can be specified multiple times (e.g., -u ex_unit -u ex_unit -u ex_unit).

Shell search patterns are supported.

You can use systemctl list-unit-files --type=ex_unit_type to view all available units of a specific type and systemctl list-units --type=ex_unit_type to view all running units of a specific type.

--utc

View the journal in Coordinated Universal Time (UTC).

--vacuum-size=ex_size, --vacuum-time=ex_time, --vacuum-files=ex_files

Remove the oldest archived journal files until the disk space they use falls below the specified size (specified with K, M, G and T suffixes), or all archived journal files containing no data older than the specified time-span (specified with s, m, h, days, months, weeks and years suffixes), or no more than the specified number of separate journal files remain, respectively.

These three options may be combined into a single invocation to enforce any combination of a size, time, and number of files limit on the archived journal files.

Field Filters

journalctl output can be filtered via the use of numerous field filters (you can also view all field filters by running man 7 systemd.journal-fields).

journalctl ex_field=ex_value

View messages using a specific field filter.

journalctl -F ex_field, journalctl --field=ex_field

View all available values for a given journal field.

journalctl _PID=ex_pid

View messages by Process ID (PID).

journalctl _UID=ex_uid

View messages by UID.

journalctl _GID=ex_gid

View messages by Group ID (GID).

Documentation

For more information on systemd logging, run the following commands:

• man 8 systemd-journald.service
• man 5 journald.conf
• man 7 systemd.journal-fields
• man 1 journalctl

syslog Logging

Not all GNU/Linux distributions use the systemd initialization system and, therefore, do not use systemd logging. Alternative logging systems are usually based on syslog.

The Linux kernel and system/network services running in the background are not connected to user terminals. When such processes want to output a message, they might write it to the system console's screen. On a system using the X Window System (X11), such messages might show up in xconsole windows (xconsole displays system log messages in X windows).

In multi-user mode, writing a system message to the system console only is not sufficient. First, it is not clear that the message will actually be read by the system's administrator. Second, these screen messages cannot be saved and may easily get lost.

A solution to this problem is the syslog daemon, syslogd. Instead of directly outputting messages, system messages with a specific meaning can be output using the syslog() function, which is a part of the Linux C runtime library. These messages are accepted by syslogd via the local socket /dev/log.

Kernel message are handled by a different program, klogd. This program preprocesses the messages and usually passes them along to syslogd.

syslogd logs the different system messages and is useful for debugging. Usually, it is started via an init script when the system is booted. When it receives messages, it can write them to a file or send them across the network to another computer, which manages a centralized log.

syslog Implementations

There are various implementations of syslog logging:

• syslogd
• rsyslogd Meant to be a rocket-fast implementation of a syslog daemon with support for plugins, alternate storage mechanisms, and more flexible rules processing. It can store logs in a database, filter the logs based on keywords, and keep statistics. Also, it is backwards-compatible with syslogd.
• syslog-ng The next generation syslog server. It has many of the same features of rsyslogd, but with a simpler filter syntax.

syslogd

The syslogd configuration file is /etc/syslog.conf and it specifies where log messages should go. /etc/syslog.conf consists of two columns and looks something like this:

kern.warn;*.err;authpriv.none       /dev/tty10
kern.warn;*.err;authpriv.none       |/dev/xconsole
*.emerg                             *
*.=warn;*.=err                      -/var/log/warn
*.crit                              /var/log/warn
*.*;mail.none;news.none             -/var/log/messages

Each rule consists of a selector and a destination, separated by white space.

The first column (i.e., the selector) of each line determines which messages will be selected and can be made up or one or more patterns. Each pattern is a facility and a priority (severity), separated by a period (.). Each pattern is separated by a semicolon (;). A wildcard (*) can be used in the pattern.

The second column (i.e., the destination) determines where the messages go. A - in front of a destination tells syslogd that it should not commit each log entry to disk as it is logged, but to let the kernel write to disk when it has time. If a destination is just a wildcard (*), the message is sent to the console of any logged in users.

Comments in this file start with a sharp (#).

syslogd supports logging to a remote host. Moving your log files from the local system to a different computer on the network can be a valuable administrative and security measure.

For example, you could add a line like this to /etc/syslog.conf:

*.* @ex_IP_address_of_loghost

Then, you would restart syslogd to activate the change:

# service syslog restart

Facilities and Priorities

The <facility> denotes the system program or component creating the log message. If you specify a * in place of a facility, this serves as a place holder for any facility.

It is not easily possible to define additional facilities. However, the local facilities local0 to local7 serve as user-definable facilities and should be sufficient for most purposes.

The <priority> specifies how serious the message is.

A message is logged with a facility and a priority. Whomever uses the syslog function, namely the developer of the program in question, must assign a facility and priority to their code's messages. Many programs allow the administrator to at least redefine the message facility.

syslogd takes care of processing the message. Different facilities can be logged at different priority levels.

When you log at a particular level, you are saying that you are only capturing logs at that or a higher priority (i.e., a lower priority level number) and discarding anything else. If you just want to capture messages of a single priority, you can do this by using the - character (e.g., ex_facility.-ex_priority).

The * character can be used to target messages of any priority (e.g., ex_facility.*. Also, you could just specify the debug priority, ex_facility.debug).

A preceding ! denotes negation (e.g., lpr.!info deselects messages from the printing subsystem at a priority of info and higher). ! and = may be combined (e.g., lpr.!=info deselects exactly those messages from the printing subsystem with priority info).

You can also specify multiple facilities with the same priority (e.g., lpr,mail.info, which selects messages of priority info and higher that belong to the lpr or mail facilities.

Log Message Management

Log messages can be handled in different ways:

• They can be written to a file. The filename must be specified as an absolute path. If there is a - in front of the path, the file will not immediately be written to disk (unlike normal syslogd operations).

  This means that in case of a system crash, you might lose pending log messages. The filename may also refer to a device file (e.g., /dev/tty3).

• Log messages can be written to a named pipe (First In, First Out; FIFO). The FIFO name must be given as an absolute path with a preceding |.

• They can be passed across the network to another syslogd instance. This is specified as the name or IP address of the target system with a preceding @ character.

  This is especially useful if a critical system state occurs that renders the local log file inaccessible, as it deprives malicious actors from hiding their activity trail, and enables administrators to collect the log messages of all hosts in a network on a single computer for easier processing.

  On the target host, syslogd must have been started using the -r (remote) option in order to accept forwarded messages.

• They can be sent directly to users. The usernames in question must be given as a comma-separated list. The message will be displayed on the listed users' terminals if they are logged in when the message arrives.

• They can be sent to all logged-in users by specifying a * in place of a login name.

Log files are generally created below /var/log/, but the specific filenames vary.

Some log files contain messages concerning users' privacy and should only be readable by the root user. Most distributions err towards caution and restrict access rights to all log files.

The log files created by syslogd can be perused using the less and tail commands. Following a log with tail's -f (--follow) option is particularly useful.

The messages written by syslogd normally contain the date and time, the hostname, a hint about the process or component that created the message, and the message itself.

Typical messages may look like this:

Mar 31 09:56:09 tails modprobe: modprobe: Can't locate ...
Mar 31 11:10:08 tails su: (to root) user1 on /dev/pts/2
Mar 31 11:10:08 tails su: pam-unix2: session started for ...

An overly large log file can be removed using the rm command (it is probably wise to save the log file first by renaming it with an extension like .old). A new log file will be created when syslogd is next restarted. There are also more convenient methods to remove large log files (e.g., logrotate).

Conventional log locations include:

/var/log/boot.log

Log entries from daemons as they were started during bootup.

/var/log/boot.msg

All of the messages displayed on screen during system boot.

/var/log/cron

Logs of scheduled job activity.

/var/log/dmesg

Hardware detection information.

/var/log/exim4/

All logs related to email (Debian).

/var/log/failog

Failed authentication attempts.

/var/log/firewall

Firewall log entries.

/var/log/maillog

All logs related to email (Fedora).

/var/log/messages

General purpose log messages that are not in one of the other logging files.

/var/log/rpmpkgs

A list of installed RPM packages (Fedora).

/var/log/secure

Security logs.

/var/log/warn

Warning messages.

/var/log/wtmp

Log entries from the xinetd daemon.

/var/log/xferlog

Logs of local File Transfer Protocol (FTP) server activity.

The Linux kernel does not send its log messages to syslogd, but puts them into an internal ring buffer. They can be read from there in various ways, e.g., via a specialized system call or the /proc/kmsg file. Traditionally, a program called klogd is used to read /proc/kmsg and pass the messages on to syslogd.

During system startup, syslogd (and possibly klogd) are not immediately available. They must be started as programs and therefore cannot directly handle the kernel's start messages. The dmesg command makes it possible to access the kernel log buffer retroactively and look at the system start log. With a command like # dmesg > boot.msg, you can write these messages to a file for further perusal or debugging purposes.

You can use the dmesg command to delete the kernel ring buffer and set a priority for direct notifications using dmesg's -c (--clear) and -n (--console-level ex_level) options, respectively. Messages meeting or exceeding the priority that you set will be immediately sent to the console.

Kernel messages have priorities from 0 to 7 corresponding to the syslogd priorities from emerg down to debug, respectively. For example, the # dmesg -n 1 command causes only alert and emerg messages to be directly written to the console.

All message will be written to /proc/kmsg in every case. Here, it is the job of post-processing software, like syslogd, to suppress unwanted messages.

dmseg can tell you system information about the amount and usage of Random Access Memory (RAM), available Central Processing Units (CPUs), disks and other mass storage devices (e.g., IDE and SCSI), Universal Serial Bus (USB) devices, and network cards.

logger

The syslogd mechanism can be tested using the logger program, along with its -p ex_priority (--priority ex_priority) and -t ex_tag (--tag ex_tag) options:

logger -p ex_facility.ex_priority -t ex_tag ex_message

ex_priority can be specified numerically or in the ex_facility.ex_priority form shown above. For example, logger -p local0.err -t TEST 'Hello, world!' produces a log message of the form Sep 20 17:27:45 tails TEST: Hello, world!.

To include the PID in the log entry, add the -i option.

If you want to log more messages than are already being collected by syslogd, you can modify the /etc/syslog.conf file and send syslogd a SIGHUP signal to get it to re-read its configuration file.

Documentation

For more information on syslogd logging, run the following commands:

• man 3 syslog
• man 8 syslogd
• man 5 syslog.conf
• man 1 logger

Also, syslog protocol information can be found in RFC 5424.

rsyslogd

The rsyslogd configuration file is /etc/rsyslog.conf and it specifies where log messages should go. It is largely compatible with what syslogd uses. rsyslogd gets by without a separate klogd program because it directly takes care of kernel log messages by itself.

Besides greater efficiency, rsyslog's goal is to support various sources and sinks for log messages. For example, it writes messages not just to text files and terminals, but also to a wide selection of databases.

The basic ideas behind rsyslog are:

• Source pass messages to rulesets. There is one standard built-in ruleset (RSYSLOG_DefaultRuleset) but, as the user, you get to define others.
• Every ruleset may contain arbitrarily many rules.
• A rule consists of a filter and an action list. Filters make yes-no decisions about whether the corresponding action list will be executed.
• For each message, all of the rules in the ruleset will be executed in order from the first to the last (and no others). All rules will always be executed, no matter how the filter decisions go, although there is a stop processing action.
• An action list may contain many actions (at least one). Within an action list, no further filters are allowed. The actions determine what happens to matching log messages.
• The exact appearance of log messages in the output may be controlled through templates.

/etc/rsyslog.conf

In /etc/rsyslog.conf, you may use three different styles of configuration settings in parallel:

1. The traditional /etc/syslog.conf syntax (sysklogd).
2. An obsolete rsyslog syntax (legacy rsyslog). This syntax can be recognized by commands that start with dollar signs ($).
3. The current rsyslog syntax (RainerScript). This syntax is best suited for complex situations.

sysklogd syntax and legacy rsyslog syntax are line-based. In RainerScript, line breaks are irrelevant.

For simple applications, you can, and likely should, use the sysklogd syntax. If you want to set configuration parameters or express complex control flows, RainerScript is probably more appropriate. It is likely best to avoid the obsolete rsyslog syntax, unless you need to use a rsyslog feature that is only accessible using that syntax.

Empty lines and comment lines in /etc/rsyslog.conf are ignored. Comment lines include both lines (and parts of lines) that start with a # (the comment then stops at the end of the line) and C-style comments that go from a /*, disregarding line breaks, until a */.

C-style comments may not be nested, but # comments may occur inside C-style comments.

Extended Filter Expressions

rsyslog offers various features that surpass those of BSD syslogd, e.g., you can use extended filter expressions for messages. Extended filter expressions always consist of a colon at the left margin, a property that rsyslog takes from the message, a filter operator, and a search term.

:ex_property, ex_filter_operator, ex_search_term ex_destination

For example, the following sends all log messages whose text contains the character sequence FOO to the /var/log/foo.log file:

:msg, contains, "FOO" /var/log/foo.log

Apart from msg (the log message proper), the properties you may use include:

fromhost

The name of the computer that forwarded the message to rsyslog.

hostname

The name of the computer sending the message.

input-name

The rsyslog module name of the source of the message.

pri

The category and priority as a text string, with the number appended, as in local0.err<133>.

syslogfacility, syslogseverity, syslogfacility-text, and syslogseverity-text

For direct access to the category and priority.

timegenerated

When the message was received.

The allowable comparison operators are:

• contains
• eregex
• isequal
• regex
• startswith

regex considers its parameters as a simple regular expression and eregex as an extended regular expression according to POSIX. All comparison operators take upper and lower case into account.

The startswith comparison is useful because it is more efficient than a regular expression that is anchored to the start of the message (as long as you are looking for a constant string).

However, you need to consider that the start of the message and what rsyslog thinks of that can be quite different. For example, if rsyslog receives a message via the syslog service, this will look something like <131>Jul 22 14:25:50 root: error found.

As far as rsyslog is concerned, mgs does not start at the e of error, but with a space character in front of it. So, if you are looking for messages that start with error, you should say :msg, startswith, " error" /var/log/error.log.

With traditional syslogd, an entry like local0.* @enceladus.example.com will forward log messages to a remote host via the (UDP-based) syslog protocol. With rsyslogd, you may also write local0.* @@enceladus.example.com to transmit log messages via TCP.

Potentially, this is more reliable, especially if firewalls are involved. At the other end of the TCP connection, there must be a suitably configured rsyslog listening for messages. You can ensure this in /etc/rsyslog.conf like so:

module(load="imtcp" MaxSessions="500")
input(type="imtcp" port="514")

This does the same thing, but in the obsolete syntax:

$ModLoad imtcp
$InputTCPMaxSessions 500
$InputTCPServerRun 514

Consider that only the UDP port 514 is officially reserved for the syslog protocol. The TCP port 514 is used for a different purpose. You can specify a different port, e.g., local0.* @@enceladus.example.com:10518 (this works for UDP, too, if required). The changes required on the server will be easy to deduce.

Filters can be based on expressions that may contain arbitrary Boolean, arithmetic, or string operations. These always start with an if at the beginning of a new line:

if $syslogfacility-text == "local0" and $msg startswith " FOO" and \
    ($msg contains "BAR" or $msg contains "BAZ") then /var/log/foo.log

With this rule, messages of category local0 will be written to the /var/log/foo.log file, as long as they start with FOO and also contain either BAR or BAZ (or both). Watch for dollar signs at the start of the property names.

Modules

rsyslog supports a large number of modules that determine what should happen to log messages. For example, you might forward important messages by email. To do so, you might put something like this into your /etc/rsyslog.conf file:

module(load="ommail")
template(name="mailBody" type="string" string="ALERT\\r\\n%msg%")
if $msg contains "disk error" then {
    action(type="ommail" server="mail.example.com" port="25"
        mailfrom="rsyslog@example.com" mailto="admin@example.com"
        subject.text="Error Detected"
        body.enable="on" template="mailBody"
        action.execonlyonceeveryinterval="3600")
}

Keep in mind, rsyslog's SMTP implementation is fairly primitive, since it does not support either encryption or authentication. This means that the mail server you specify in the rsyslog configuration must be able to accept mail from rsyslog, even without encryption or authentication.

rsyslog can directly handle Linux kernel log messages. You simply need to enable the imklog input module:

module(load="imklog")

A separate klogd process is not required.

Documentation

For more information on rsyslogd logging, run the following commands:

• man 8 rsyslogd
• man 5 rsyslog.conf

Also, check out the rsyslog site and the official project repository.

syslog-ng

syslog-ng (Next Generation) is a compatible, but extended, re-implementation of a syslog daemon. The main advantages of syslog-ng compared to the traditional syslogd include:

• Filtering of messages based on their content (not just categories and priorities).
• Chaining of several filters.
• A more sophisticated input/output system, including forwarding by TCP and to subprocesses.

The logging program itself is called syslog-ng. For syslog clients, there is no difference, i.e., you can replace a syslogd with syslog-ng without any problems.

syslog-ng reads its configuration from a file, normally /etc/syslog-ng/syslog-ng.conf. Unlike syslogd, syslog-ng distinguishes various entry types in its configuration file.

Filters

Filters are Boolean expressions based on internal functions that can, for example, refer to the origin, category, priority, or textual content of a log message. Filters are also named.

They are used to sift through log messages or distribute them to various sinks. They rely on internal functions that consider specific aspects of messages. These functions can be joined using the logical operators and, or, and not.

These are various possible forms of filter syntax:

facility( ex_category... )

Matches messages with one of the listed categories. Multiple categories can be provided as a comma-separated list.

filter( ex_name )

Invokes another filtering rule and returns its value.

host( ex_regex )

Matches messages whose sending host matches ex_regex.

level( ex_priority... )

Matches messages with one of the listed priorities. Multiple priorities can be provided as a comma-separated list.

match( ex_regex )

Matches messages that match the ex_regex themselves.

netmask( ex_ip_address / ex_netmask )

Checks whether the IP address is in the given network.

priority( ex_priority... )

Same as level(). Multiple priorities can be provided as a comma-separated list.

program( ex_regex )

Matches messages where the name of the sending program matches ex_regex.

For example, you could define a filter that matches all messages from host europa containing the text error:

filter f_europa { host("europa") and match("error"); };

With the level() (or priority()) filter, you can specify either one or more priorities separated by commas, or else a range of priorities like warn..emerg.

Log Paths

A log path connects one or several message sources, filters, and sinks. If messages arrive from the sources and pass the filter (or filters), they will be forwarded to the specified sink(s). The configuration file consists of a number of such log paths.

Log paths bring sources, filters, and sinks together to evaluate messages. They always start with the log keyword.

The following rule causes all messages to do with authentication to be written to the /var/log.auth.log file:

# Prerequisites
source s_all { internal(); unix-stream("/dev/log"); };
filter f_auth { facility(auth, authpriv); };
destination df_auth { file("/var/log/auth.log"); };

# auth,authpriv.* /var/log/auth.log
log {
    source(s_all);
    filter(f_auth);
    destination(df_auth);
};

Below, every message passes through all log paths and is logged by all matching ones.

# kern.warn;*.err;authpriv.none /dev/tty10
filter f_nearly_all {
    (facility(kern) and priority(warn .. emerg))
    or (not facility(authpriv,kern));
};
destination df_tty { file("/dev/tty10"); };
log {
    source(s_all);
    filter(f_nearly_all);
    destination(df_tty);
};

If you want a message to not be further considered after it has passed a particular log path, you can add the flags(final) option to that path. flags(final) does not mean that the message is logged just once. It might have been logged by other paths before the path in question.

With flags(fallback), you can declare a path to be the default path. This path is only considered for log messages that did not match any paths that were not marked flags(fallback).

In the next example, whenever the amnesia user logs on or off, a message is displayed on the system admin's (root's) terminal, if they are logged in:

# We assume a suitable source definition.
filter login_amnesia {
    facility(authpriv)
        and (match("session opened") or match("session closed"))
        and match("user amnesia");
};
destination d_root { usertty("root"); };
log { source(...);
    filter(login_amnesia);
    destination(d_root);
};

Message Sinks

syslog-ng includes all logging methods of syslogd and more. Sinks consist of various drivers for logging methods. For example, you can write messages to a file:

destination d_file { file("/var/log/messages"); };

Also, you can specify a template that describes in which format the message should be written to the sink in question. When doing so, you can refer to macros that make various parts of the message accessible.

destination d_file {
    file("/var/log/$YEAR.$MONTH.$DAY/messages"
        template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG\n")
        template_escape(no)
        create_dirs(yes)
    );
};

The $YEAR, $MONTH, and comparable macros above are replaced by the obvious values. For example, $TZ is the current time zone, $LEVEL is the message priority, and $MSG is the message itself (including the sender's PID).

The template_escape() parameter controls whether quotes (' and ") should be escaped in the output. This is important if you want to feed the log messages to something like a SQL server.

Unlike syslogd, syslog-ng allows forwarding messages using TCP. This is more convenient when firewalls are involved and ensures that no log messages are lost (which may happen with UDP).

A TCP forwarding sink can be defined like so:

destination d_tcp { tcp("10.11.12.13" port(518); localport(518)); };

It is also useful to forward messages to programs using program(). syslog-ng starts the program when it is started itself, and keeps it running until it is stopped or it receives a SIGHUP signal.

This is not just to increase efficiency, but serves as a precaution against denial-of-service attacks, e.g., if a new process is started for every message, an attacker could shut off logging by sending large amounts of matching log messages (other messages that would point to these activities might then be dropped).

Message Sources

syslog-ng can read messages in various ways (e.g., from UNIX-domain sockets or UDP like syslogd, and from files, FIFOs, or TCP sockets). Every message source is assigned a name.

In syslog-ng, message sources are defined using the source keyword. A message source collects one or more drivers. To accomplish the equivalent of a normal syslogd, you would include the line source src { unix-stream("/dev/log"); internal(); }; in your configuration. This tells syslog-ng to listen to the UNIX-domain socket /dev/log. internal() refers to messages that syslog-ng creates by itself.

A syslog-ng message source corresponding to the -r option of syslogd might look like this:

source s_remote { udp(ip(0.0.0.0) port(518)); };

Since that is the default setting, source s_remote { udp(); }; would also be sufficient.

With ip(), you can let syslog-ng listen on specific local IP addresses only. This is not possible with syslogd.

The following source specification lets syslog-ng replace the klogd program:

source kmsg { file("/proc/kmsg" log_prefix("kernel: ")); };

All message sources support another parameter, log_msg_size(), which specifies the maximum message length in bytes.

Global Options

You can specify various global options that control syslog-ng's general behavior or determine default values for individual message sources or sinks (specific options for the sources or sinks take priority). The general options include various settings for handling DNS and the forwarding or rewriting of messages' sender hostnames.

If syslog-ng on host A receives a message from host B, it checks the keep_hostnames() option. If its value is yes, B is kept as the hostname for the log. If not, the outcome depends on the chain_hostnames() option.

If this is no, then A will be logged as the hostname. If it is yes, then syslog-ng will log B/A. This is particularly important if the log is then forwarded to yet another host.

Documentation

For more information on syslog-ng logging, run the following commands:

• man 8 syslog-ng
• man 5 syslog-ng.conf

Also, check out the syslog-ng site and the official project repository.

Log Rotation

Log files can quickly grow. To keep a system from being inundated with log data, you may want to put the relevant directories (e.g., /var/log/ or /var/) on their own partitions.

On the other hand, there is software that periodically checks the log files according to various criteria (e.g., the size), truncates them, and removes or archives old log files. This process is called rotation and a program that performs rotation is logrotate.

Log rotation is the process that renames a current log file (e.g., auth.log becomes auth.log.1) and sets up a new log file (auth.log) for new log entries. Older log files may be compressed.

logrotate is not a daemon, but is usually executed once a day using cron or a similar service. logrotate refuses to modify a log file more than once a day, except:

• If the decision depends on the size of the log file.
• You are using the hourly criterion.
• You are using logrotate's -f (--force) option.

/etc/logrotate.conf and /etc/logrotate.d/

According to convention, logrotate is configured using the /etc/logrotate.conf file and the files within the /etc/logrotate.d/ directory. The /etc/logrotate.conf file sets up general parameters, which can be overwritten by the files in /etc/logrotate.d/, if necessary.

The format of the /etc/logrotate.conf file is:

ex_path_to_log_file_glob {
    ex_instruction_1
    ex_instruction_2
}

In /etc/logrotate.conf, there is an include /etc/logrotate.d/ line, which causes the files from that directory to be read in that place, as if they were part of the /etc/logrotate.conf file itself.

In principle, logrotate reads all of the files named on the command line as configuration files, and the content of files mentioned later on overwrites that of files mentioned earlier. The /etc/logrotate.conf file is a convention that is put into action by means of a suitable invocation of logrotate in /etc/cron.daily/logrotate.

logrotate watches all files that it is told about by the aforementioned configuration files, not just those created by syslogd.

Typically, /etc/logrotate.conf contains directives that are outside of a brace-delimited block. These directives serve as defaults that apply to all log files in the configuration, unless something more specific is given in their own blocks of directives.

/etc/logrotate.conf directives include:

compress

Old versions of log files are compressed with gzip by default.

daily

Log files are rotated every day.

dateformat ex_format_string

Specify the extension for dateext using the notation similar to the strftime(3) function. Only %Y, %m, %d, %H, %M, %S, %V, and %s specifiers are allowed. The default value is -%Y%m%d except hourly, which uses -%Y%m%d%H as the default value.

delaycompress

Do not compress the file until it has already been rotated.

mail ex_address

When a log is rotated out of existence, it is mailed to ex_address. If no mail should be generated by a particular log, the nomail directive may be used.

maxsize ex_size

Log files are rotated when they grow bigger than ex_size, even before the additionally specified time interval (daily, weekly, monthly, or yearly).

minsize ex_size

Log files are rotated when they grow bigger than ex_size, but not before the additionally specified time interval (daily, weekly, monthly, or yearly).

missingok

If a log is missing, do not raise an error.

monthly

Log files are rotated the first time logrotate is run in a month (this is normally on the first day of the month).

notifempty

If empty, the log is not rotated.

postrotate/endscript

Run the commands that follow, up until the endscript keyword, after the logs are rotated.

rotate ex_num

Rotates a given log ex_num times before deleting it.

size ex_size

Log files are rotated only if they grow bigger than size bytes.

If size is followed by k, the size is assumed to be in kilobytes. If M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So, size 100, size 100k, size 100M, and size 100G are all valid.

sharedscripts

If the wildcard specified matches several files, run any scripts once for all files.

weekly

Rotates the log files once a week.

yearly

Log files are rotated if the current year is not the same as the last rotation.

Example Configuration

Below is an example configuration for logrotate:

/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        invoke-rc.d rsyslog rotate >/dev/null
    endscript
}

rotate 7

The first line specifies the files that this configuration applies to (e.g., /var/log/syslog). You may enumerate several files or specify shell search patterns. After that, inside curly braces, there is a block of directives that define how logrotate should deal with the given files.

rotate 7 means that, at most, seven old version of each log file will be kept. When this maximum is reached, the oldest version of the log file is deleted.

If you specify an address using mail, files will not be deleted, but instead be sent to the email address in question. rotate 0 deletes rotated log messages outright.

The rotated files are numbered in sequence, i.e., if the current version of the file is called /var/log/syslog, the immediately preceding version will be /var/log/syslog.1, and the version preceding that will be /var/log/syslog.2.

Instead of sequential numbers, you may use the date. For example, if today is July 19, 2021, and your logrotate run takes place daily in the morning, the immediately preceding version of the file will be /var/log/syslog-20210719, and the version preceding that will be called /var/log/syslog-20210718.

Using dateformat, you can control exactly how the date-based file extension should look. To do so, you need to specify a string that may contain the %Y, %m, %d, and %s keys. These stand for the four-digit year, calendar month, calendar day (in each case two digits and, if necessary, with a leading zero), and seconds since 1st January 1970, 12:00 AM UTC, the UNIX epoch. The default is -%Y%m%d.

When you use dateformat, note that logrotate does a lexicographic sort of filenames when rotating in order to find out which file is the oldest. This works with -%Y%m%d, but not with -%d%m%Y.

daily

daily means that log files should be rotated daily. Together with rotate 7, this implies that you always have access to last week's logs.

There are also weekly, monthly, and yearly values. With weekly, the file is rotated when the current day of the week is earlier than the day of the week of the last rotation, or more than one week has passed since the last rotation (i.e., rotation will take place on the first day of the week, which, according to United States custom, is Sunday).

With monthly, the file is rotated on the first logrotate run of the month (usually on the first of the month). With yearly, rotation takes place on the first logrotate run of the year. Theoretically, hourly rotates the log file every hour, but since logrotate is normally only run once a day, you will have to arrange for it to be run frequently enough.

An alternative criterion is size. This will rotate a log file when a certain size has been exceeded. The file size is given as a parameter. Without a unit, it will be taken to mean bytes, while the units k (or K), M, and G stand for kibibytes (2^10 bytes), mebibytes (2^20) bytes, or gibibytes (2^30 bytes), respectively.

size and the time-based criteria are mutually exclusive, i.e., if you specify a size criterion, rotation will solely depend on file size, no matter when the file was last rotated. File size and time can be used together by means of the maxsize and minsize criteria.

With maxsize, you can specify a size that will cause logrotate to rotate the file even if the next official date has not been reached. With minsize, the file will only be rotated at the specified point in time if it has exceeded the given size (small files will be skipped).

missingok, notifempty

missingok suppresses error messages if a log file could not be found (the default is nomissingok). notifempty does not rotate a file if it is empty (the default here is ifempty).

delaycompress, compress

The delaycompress directive ensures that a freshly rotated file is not immediately compressed after the rotation, but only on the next run. compress lets you specify that rotated versions of the log file should be compressed.

Usually, the sequence of files would look like:

/var/log/syslog /var/log/syslog.1.gz /var/log/syslog.2.gz ex_code

delaycompress would produce a sequence like:

/var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz ex_code

In other words, /var/log/syslog.1 remains uncompressed. You need this setting if there is a chance that the logging program (like rsyslog) might append data to the file after it has been renamed (i.e., rotated). This can happen because rsyslog keeps the log file open and renaming the file is irrelevant, as far as writing to it is concerned.

postrotate, endscript

You need to notify rsyslog that there is a new log file. This is what this directive is for:

postrotate
    invoke-rc.d rsyslog rotate >/dev/null
endscript

The shell commands between postrotate and endscript are executed by logrotate whenever the log file has been rotated. In the end, rsyslog's init script is invoked and it sends SIGHUP to the program. Then, SIGHUP causes rsyslog to re-read its configuration file, and close and reopen all log files.

Since /var/log/syslog was previously renamed, rsyslog opens a new log file under that name. At this point, logrotate could compress the /var/log/syslog.1 file, but it has no way of knowing when rsyslog is really done with the file. This is why this is postponed until the file gets rotated again.

Between postrotate and endscript, there may be several lines with commands. logrotate concatenates them all and passes them to the shell (/bin/sh) as a whole. The command is passed the name of the log file as a parameter and that is available there in the customary fashion as ${1}.

The postrotate commands are executed once for every log file enumerated at the start of the configuration block, i.e., the commands may be executed several times. You can use the sharedscripts directive to ensure that the commands are executed at most once for all files that match the search pattern (or not at all, if none of the files needed to be rotated).

You can use create to make sure that the log file is immediately recreated after the rotation and before the postrotate commands are executed. This uses the name of the old file. The file mode, owner, and group derive from the parameters to create.

The three possibilities are:

1. create 600 root adm File mode, user, and group.
2. create root adm Just user and group.
3. create Nothing at all.

Unspecified file properties are taken from the previous version of the file. Run man 8 logrotate for the full list of configuration parameters.

Documentation

You can find more information on the commands discussed above by examining the Linux User's Manual, either at the command line or online.