Logs are important to help us understand our GNU/Linux systems. They provide an avenue to track and audit system activity, which can be valuable for security and performance purposes.
There are several GNU/Linux logging implementations. The implementation that you encounter will depend on the GNU/Linux distribution that you are using.
Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.
systemd Logging
In a traditional GNU/Linux logging system, the syslog daemon accepts log messages on UDP port 514
or the /dev/log
socket. From there, it usually writes the messages to text files or forwards them to other hosts where they are written to files.
In a systemd system, background services can write log messages to their standard error output channel and systemd will arrange for them to be passed to the logging service. With systemd, log files are not text files. Instead, log messages are written to a binary database that can then be queried according to diverse criteria.
systemd-journald
, the systemd logging daemon, logs to a binary file and annotates each log entry with metadata. Anything that a daemon writes to the console is saved as a proper entry in the database.
Modern syslog implementations (e.g., Rsyslog or Syslog-NG) are capable of writing log messages to a database, but it is your responsibility to come up with a suitable database schema, to configure Rsyslog or Syslog-NG, and to develop software that allows you convenient access to the log messages. By default, systemd includes all of this functionality.
The systemd journal is not confined to textual log messages. For example, it is possible to store core dumps of crashed programs in the journal, as long as they are not oversized.
systemd's log system can interoperate with the traditional logging approach. If desired, it logs messages that arrive on UDP port 514
or /dev/log
, and can pass messages on to a traditional syslog daemon (or modern re-implementation).
systemd enables you to see the most recent log messages by a service using the systemctl
command:
systemctl status ex_service
Overlong lines are shortened, but to see them in full, use the -l
(--full
) option with the command (e.g., systemctl -l status ssh
).
The systemd Journal
There are some disadvantages of traditional logging systems' use of text files for log files:
- They can be difficult to search.
- Targeted evaluation is either tedious or requires additional non-standardized software.
- There is no type of cryptographic protection against the manipulation of log entries, and the amount of information that can be written to the log is limited.
The systemd journal is an integrated part of systemd. In the simplest case, systemd uses a limited-size ring buffer in /run/log/journal/
to store a certain number of log messages in RAM (which is sufficient if you want to pass the messages on to a traditional log service).
To take advantage of all journal features, you should ensure that the log messages are permanently stored on disk. This is done by creating the directory for storage (many distributions, like Debian and Fedora, are already configured to use /var/log/journal/
for a persistent journal).
# mkdir -p /var/log/journal/
# systemctl --signal=USR1 kill systemd-journald
The SIGUSR1
signal gets systemd to transfer the RAM-based journal (i.e., /run/log/journal/
) to the new file on disk (i.e., /var/log/journal/
).
systemd-journald and /etc/systemd/journald.conf
The journal is configured by means of the /etc/systemd/journald.conf
file. The [Journal]
section of this file contains the Storage
parameter, which can assume any of the following values:
auto
- Similar topersistent
, but the existence of the/var/log/journal/
directory determines whether a persistent journal is written. If the directory does not exist, the volatile journal is used.none
- No log messages are stored in the journal. You can still pass messages to a traditional syslog service.persistent
- Log messages are preferably stored on disk (/var/log/journal/
). The directory will be created if it does not exist. During early boot and if the disk is not writable, systemd falls back onto/run/log/journal/
.volatile
- Log messages are only stored in RAM, even if/var/log/journal/
exists.
Interesting parameters besides Storage
include:
Compress
- Specifies whether log files (or those exceeding a certain size) are transparently compressed. The default value is
yes
. RateLimitInterval
,RateLimitBurst
- These parameters are supposed to make it more difficult to flood the log with messages. If a service produces more than
RateLimitBurst
messages during a period of time given byRateLimitInterval
, then all further messages until that period of time is over are ignored (the log will contain only one message detailing the number of ignored messages). - By default, the limit is
10000
messages in30
seconds. If you set either of the parameters to zero, the limitation is lifted. Seal
- Lets you ensure that persistent journal files are protected against clandestine manipulation by means of a cryptographic signature. You will need to provide an encryption key.
SyncIntervalSec
- Specifies how often the journal will be synced to disk. The journal is always immediately saved after a message of priority
crit
(or above) has been logged. As long as no such message arrives, journald will wait for the interval specified bySyncIntervalSec
before saving it again. The default value is5
minutes.
systemd-journald
tries to make sensible use of available disk space, i.e., new messages are normally appended to the journal, but if a certain upper limit for the size of the journal is reached, it tries to remove old log messages.
The following /etc/systemd/journald.conf
parameters can be be used to set limits:
RuntimeMaxUse
- Describes how much space the journal may take up under
/run/log/journal/
. SystemMaxUse
- Describes how much space the journal may take up under
/var/log/journal/
. RuntimeKeepFree
,SystemKeepFree
- Determine how much space must be kept free on the file system in question.
systemd-journald
takes into account all of the above parameter values and confines itself to the minimum size dictated by both the MaxUse
and KeepFree
parameter values. The Runtime
values are used when the system is booting or no persistent journal is used. The System
values apply to the persistent journal if the system has been booted far enough. When determining the space used by the journal, only files whose names end in .journal
are considered.
Space amounts may be specified in bytes or via one of the following binary units:
K
M
G
T
P
E
If there is less space available when systemd-journald
starts than the KeepFree
values dictate, the limit is reduced even further such that space for other material remains.
Like the logrotate
command did in traditional logging systems, systemd rotates the journal to make room for new messages. To do so, the available space is subdivided into a number of files, so that the oldest can be continually discarded. This rotation is transparent to users because systemd-journald
does it of its own accord when required and journalctl
always evaluates the full journal, no matter how many files it consists of.
The subdivision is governed by the SystemMaxFileSize
and RuntimeMaxFileSize
parameters within the /etc/systemd/journald.conf
file. These parameters specify how large individual journal files may become.
Also, you may make the log file rotation depend on time via the following parameters:
MaxFileSec
- Determines the maximum time period before systemd starts a new log file (usually the size-based rotation is adequate). The default value is
1month
(0
means unlimited). MaxRetentionSec
- Specifies an upper limit for how long old log messages are kept around. Setting this to
0
disables the mechanism.
In /etc/systemd/journald.conf
, you can also configure log forwarding to a traditional syslog system. To do so, set ForwardToSyslog
to yes
:
[Journal]
ForwardToSyslog=yes
Access Rights
As the root user, you get to see the complete systemd journal log. As an ordinary user, you will only be able to peruse your own log, specifically the messages submitted by programs that you started yourself (or that the system started on your behalf).
If you want to have full access to the log, even as an ordinary user, you will need to ensure that you are part of the adm
or wheel
group (this will depend on the distribution that you are using).
For example, on Debian, you can add yourself to the adm
group like so:
# usermod -a -G adm ex_username
After you restart your system, this change will be effective.
journalctl
The journalctl
command is used to direct queries to the journal:
$ journalctl
-- Logs begin at Mo 2015-07-27 13:37:14 CEST, end at Mo 2015-07-27
22:20:47 CEST. --
Jul 27 13:37:14 tails systemd-journal[138]: Runtime journal is using 4.
Jul 27 13:37:14 tails systemd-journal[138]: Runtime journal is using 4.
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpuset
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpu
Jul 27 13:37:14 tails kernel: Initializing cgroup subsys cpuacct
Jul 27 13:37:14 tails kernel: Linux version 3.16.0-4-amd64 (debian-kern
Jul 27 13:37:14 tails kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.
The output will strongly resemble what would be found in /var/log/messages
, but includes various improvements:
-
The log is displayed using your favorite display program for text files. This is determined by the value of the
SYSTEMD_PAGER
environment variable, failing that the value ofPAGER
, and failing that,less
.Using the
SYSTEMD_LESS
environment variable, you can specify options forless
(if you do not use the system default, this variable is ignored). Alternatively, you can also put options directly intoSYSTEMD_PAGER
for any pager program, includingless
. -
If you invoke
journalctl
with the--no-pager
option, or setSYSTEMD_PAGER
tocat
or an empty string, the output will not be displayed page by page. - The output includes all accessible log files, including rotated ones.
- Time stamps are in the local (zone) time, not UTC.
- Log messages of priority
notice
orwarning
are displayed in bold face. - Log messages of priority
error
(or higher) appear in red.
Real-Time Journal Monitoring
To follow new messages being written to the journal, use journalctl
's -f
(--follow
) option:
journalctl -f
This will display ten lines of output before journalctl
waits for further messages to arrive.
As with the tail
command, you can set the initial number of lines shown with journalctl
using the -n
(--lines=
) option.
Services and Priorities
journalctl
's -u
(--unit=
) option can be used to restrict the output to those log messages written by a specific systemd unit:
$ journalctl -u ssh
-- Logs begin at Mo 2015-07-27 13:37:14 CEST, end at Di 2015-07-28
09:32:08 CEST. --
Jul 27 13:37:23 tails sshd[428]: Server listening on 0.0.0.0 port 22.
Jul 27 13:37:23 tails sshd[428]: Server listening on :: port 22.
Jul 27 13:56:50 tails sshd[912]: Accepted password for amnesia from 192.16
Jul 27 13:56:50 tails sshd[912]: pam_unix(sshd:session): session opened
Instead of a specific unit name, you can also provide journalctl
a shell search pattern to include several units, or use several -u
options.
To only display messages of certain priorities, use the -p
(--priority=
) option. This takes either a single numerical or textual priority and limits the output to messages of that priority or above. Alternatively, you can specify a range to only see those messages whose priority is in that range:
journalctl -p ex_priority_1..ex_priority_2
The -u
and -p
options can be combined. For example, journalctl -u nginx -p err
displays all error messages (or worse) from the NGINX web server.
The -k
(--dmesg
) option limits the output to messages logged by the operating system kernel. This considers only messages written since the last system boot.
Time
Log output can be filtered by time. The -S
(--since=
) and -U
(--until=
) options let you specify a date or time in the YYYY-MM-DD hh:mm:ss
format, and only messages written since or until that point in time will be output.
You can leave off the time value, in which case a time of 00:00:00
is assumed. If you just leave off the seconds, :00
is implied. If you leave off the date (in which case you will need to specify a time, with or without seconds), journalctl
will assume today
as the date.
The yesterday
, today
, and tomorrow
keywords stand for 00:00:00
yesterday, today, and tomorrow, respectively.
Relative time specifications are also allowed. For example, -30m
stands for half an hour ago and 1h
stands for in one hour.
Every system boot is assigned a unique identifier, and you can limit your search to the part of the journal between one boot and the next. In the simplest case, using journalctl
's -b
(--boot=
) option will only consider messages from the current boot:
journalctl -b -u nginx
With the --list-boots
option, journalctl
will output a list of boot identifiers to be found in the journal, together with the periods of time for which there are log entries:
$ journalctl --list-boots
-1 30eb83c06e054feba2716a1512f64cfc Mo 2015-07-27 22:45:08 CEST—
Di 2015-07-28 10:03:31 CEST
0 8533257004154353b45e99d916d66b20 Di 2015-07-28 10:04:22 CEST—
Di 2015-07-28 10:04:27 CEST
You may refer to specific boots by passing their index number to the -b
option (e.g., 1
stands for the chronologically first boot in the log and 2
for the second boot) or the negative offset in the first column of the output of journalctl --list-boots
(e.g., 0
refers to the current boot and -1
stands for the boot prior to the current boot).
The 320 character alphanumeric boot ID from the second column of journalctl --list-boots
can be specified to search the journal for that boot only. A positive or negative offset can be appended to the boot ID to identify boots before or after it, respectively.
For example, if 8533257004154353b45e99d916d66b20
represented the second boot ID in the log, the following would be another way of saying journalctl -b 1
:
journalctl -b 8533257004154353b45e99d916d66b20-1
Arbitrary Search Operations
If you specify a path name as an argument, journalctl
tries to act on it in a reasonable fashion:
- If it refers to an executable file,
journalctl
looks for journal entries made by that program. - If it refers to a device file,
journalctl
looks for entries concerning the device in question.
These search operations are special cases of a more general search mechanism offered by the journal. systemd does log more information than the traditional syslog mechanism. This can be demonstrated by invoking journalctl
with the -o verbose
(--output=verbose
) option:
$ journalctl -o verbose -u sshd
Mo 2015-07-27 13:37:23.580820 CEST [s=89256633e44649848747d32096fb42
68;i=1ca;b=30eb83c06e054feba2716a1512f64cfc;m=11a1309;t=51bd9c6f
8812e;x=f3d8849a4bcc3d87]
PRIORITY=6
_UID=0
_GID=0
_SYSTEMD_SLICE=system.slice
_BOOT_ID=30eb83c06e054feba2716a1512f64cfc
_MACHINE_ID=d2a0228dc98041409d7e68858cac6aba
_HOSTNAME=tails
_CAP_EFFECTIVE=3fffffffff
_TRANSPORT=syslog
SYSLOG_FACILITY=4
SYSLOG_IDENTIFIER=sshd
SYSLOG_PID=428
MESSAGE=Server listening on 0.0.0.0 port 22.
_PID=428
_COMM=sshd
_EXE=/usr/sbin/sshd
_CMDLINE=/usr/sbin/sshd -D
_SYSTEMD_CGROUP=/system.slice/ssh.service
_SYSTEMD_UNIT=ssh.service
_SOURCE_REALTIME_TIMESTAMP=1437997043580820
The first line in the output above is a time stamp for the message together with a cursor. The cursor identifies the message inside the journal and is needed, for example, to store log entries on remote computers.
Subsequent lines are journal fields that refer to the message in question. Field names without a leading underscore derive from information submitted by the logging program (i.e., the program filing the log entry) and, as such, are not necessarily reliable (the program could, for example, misrepresent its Process ID or name in SYSLOG_IDENTIFIER
). Field names with a leading underscore are supplied by systemd and cannot be manipulated by the logging program.
PRIORITY
, SYSLOG_FACILITY
, SYSLOG_IDENTIFIER
, SYSLOG_PID
, and MESSAGE
derive from the syslog protocol and are self-explanatory. _BOOT_ID
is the identifier of the current boot and _MACHINE_ID
identifies the logging system according to its entry in /etc/machine-id
.
_CAP_EFFECTIVE
specifies the special capabilities of the logging process and _TRANSPORT
describes how the message reached systemd (apart from syslog
, common sources are stdout
for messages that the program wrote to its standard output or standard error output, or kernel
for messages submitted by the operating system kernel).
_COMM
, _EXE
, and CMDLINE
all describe the command being executed. _SYSTEMD_SLICE
and _SYSTEMD_CGROUP
specify where in systemd's internal process management the logging process may be found.
More detailed information is available by running man 7 systemd.journal-fields
.
These fields can be searched by specifying them on journalctl
's command line:
journalctl _HOSTNAME=enceladus _SYSTEMD_UNIT=nginx.service
- Search terms using different fields are implicitly joined using
AND
. If the same field appears in several search terms, these are implicitly joined usingOR
. - There is also an explicit
OR
. For example,journalctl _HOSTNAME=enceladus _UID=80 + _HOSTNAME=deimos _UID=90
shows all processes with the User ID (UID)80
on the hostenceladus
, as well as all processes with the UID90
on the hostdeimos
(this will only work if you consolidate both of these journals on your system). - You can freely combine these search terms with options, e.g., to set up time limits or save typing,
journalctl -u nginx _HOSTNAME=enceladus
.
If you cannot remember which values a search term can assume, you can ask the journal using journalctl
's -F
(--field=
) option:
$ journalctl -F _SYSTEMD_UNIT
session-2.scope
udisks2.service
session-1.scope
polkitd.service
dbus.service
user@1000.service
Command line completion works for field names and values.
Commands
Useful systemd logging commands include:
ls -l /run/log/journal/
- View a long listing of the runtime journal directory.
ls -l /var/log/journal/
- View a long listing of the persistent journal directory.
less /etc/systemd/journald.conf
- View the contents of the journald configuration file. After any changes are made to this file, run
# systemctl restart systemd-journald
to make them effective.
journalctl Commands
The journalctl
command is used to query the systemd journal. If called without parameters, it will show the full contents of the journal, starting with the oldest entry collected.
To view journal entries for a specific binary, you can provide the binary's absolute path to journalctl
:
journalctl ex_binary_absolute_path
Helpful journalctl
options include:
-b
,--boot=
- Show messages for the current boot.
- This option can be provided a specific boot number relative to the current boot (e.g.,
-1
for the previous boot from the current boot, which is0
) or a boot ID (caf0524a1d394ce0bdbcff75b94444fe
) as an argument to only show messages for that boot. --disk-usage
- Show the current disk usage of all journal files.
-e
,--pager-end
- View the end of the journal.
-f
,--follow
- Show only the most recent journal entries and continuously print new entries as they are appended to the journal.
-k
,--dmesg
- Show only kernel messages. This is an alternative to
dmesg | less
. --list-boots
- Show a tabular list of boot numbers (relative to the current boot), their IDs, and the timestamps of the first and last message pertaining to the boot.
-n ex_number
,--lines=ex_number
- Show the most recent journal events and limit the number of events to
ex_number
. -o ex_formatting_option
,--output=ex_formatting_option
- Controls the formatting of the journal entries that are shown.
-o
takes one of the following options:-
cat
export
json
json-pretty
json-sse
json-seq
short
short-full
short-iso
short-iso-precise
short-precise
short-monotonic
short-unix
verbose
with-unit
-p ex_priority_level
,-p ex_priority_level..ex_priority_level
,--priority=ex_priority_level
,--priority=ex_priority_level..ex_priority_level
- View journal entries logged at a specific priority level/priority range or higher priority.
Show/Hide Log Priority Levels
Log Priority Levels Level Tag Meaning 0
emerg
Emergency: The system is unusable. 1
alert
Alert: Immediate action is needed to prevent a failure. 2
crit
Critical: The system has reached a critical level. 3
err
Error: Some part of the system encountered an error. 4
warn
Warning: Something happened that may have been processed incorrectly warning. 5
notice
Notice: Not an error condition, but may need special handling. 6
info
Informational: A normal log entry about a routine event. 7
debug
Debug: A message about internal state that is for debugging problems.
-r
,--reverse
- View the journal in reverse order (i.e., newest entries shown first).
-S 'YYYY-MM-DD hh:mm:ss'
,--since='YYYY-MM-DD hh:mm:ss'
- View journal entries since a specific time. Can be used with
-U
(--until=
). - If the date is omitted, the current date is assumed. If the time component is missing, midnight is assumed. If seconds is left off,
00
is used. - Relative values can be used, e.g.,
--since yesterday
,--since today
,--since '1 hour ago'
. -U 'YYYY-MM-DD hh:mm:ss'
,--until='YYYY-MM-DD hh:mm:ss
- View journal entries until a specific time. Can be used with
-S
(--since=
). - If the date is omitted, the current date is assumed. If the time component is missing, midnight is assumed. If seconds is left off,
00
is used. - Relative values can be used, e.g.,
--until yesterday
,--until today
,--until '1 hour ago'
. -u ex_unit
,--unit=ex_unit
- Show messages for the specified systemd unit. This parameter can be specified multiple times (e.g.,
-u ex_unit -u ex_unit -u ex_unit
). - Shell search patterns are supported.
- You can use
systemctl list-unit-files --type=ex_unit_type
to view all available units of a specific type andsystemctl list-units --type=ex_unit_type
to view all running units of a specific type. --utc
- View the journal in Coordinated Universal Time (UTC).
--vacuum-size=ex_size
,--vacuum-time=ex_time
,--vacuum-files=ex_files
- Remove the oldest archived journal files until the disk space they use falls below the specified size (specified with
K
,M
,G
andT
suffixes), or all archived journal files containing no data older than the specified time-span (specified withs
,m
,h
,days
,months
,weeks
andyears
suffixes), or no more than the specified number of separate journal files remain, respectively. - These three options may be combined into a single invocation to enforce any combination of a size, time, and number of files limit on the archived journal files.
Field Filters
journalctl
output can be filtered via the use of numerous field filters (you can also view all field filters by running man 7 systemd.journal-fields
).
journalctl ex_field=ex_value
- View messages using a specific field filter.
journalctl -F ex_field
,journalctl --field=ex_field
- View all available values for a given journal field.
journalctl _PID=ex_pid
- View messages by Process ID (PID).
journalctl _UID=ex_uid
- View messages by UID.
journalctl _GID=ex_gid
- View messages by Group ID (GID).
Documentation
For more information on systemd logging, run the following commands:
man 8 systemd-journald.service
man 5 journald.conf
man 7 systemd.journal-fields
man 1 journalctl
syslog Logging
Not all GNU/Linux distributions use the systemd initialization system and, therefore, do not use systemd logging. Alternative logging systems are usually based on syslog.
The Linux kernel and system/network services running in the background are not connected to user terminals. When such processes want to output a message, they might write it to the system console's screen. On a system using the X Window System (X11), such messages might show up in xconsole
windows (xconsole
displays system log messages in X windows).
In multi-user mode, writing a system message to the system console only is not sufficient. First, it is not clear that the message will actually be read by the system's administrator. Second, these screen messages cannot be saved and may easily get lost.
A solution to this problem is the syslog daemon, syslogd
. Instead of directly outputting messages, system messages with a specific meaning can be output using the syslog()
function, which is a part of the Linux C runtime library. These messages are accepted by syslogd
via the local socket /dev/log
.
Kernel message are handled by a different program, klogd
. This program preprocesses the messages and usually passes them along to syslogd
.
syslogd
logs the different system messages and is useful for debugging. Usually, it is started via an init script when the system is booted. When it receives messages, it can write them to a file or send them across the network to another computer, which manages a centralized log.
syslog Implementations
There are various implementations of syslog logging:
- syslogd
- rsyslogd Meant to be a rocket-fast implementation of a syslog daemon with support for plugins, alternate storage mechanisms, and more flexible rules processing. It can store logs in a database, filter the logs based on keywords, and keep statistics. Also, it is backwards-compatible with syslogd.
- syslog-ng The next generation syslog server. It has many of the same features of rsyslogd, but with a simpler filter syntax.
syslogd
The syslogd
configuration file is /etc/syslog.conf
and it specifies where log messages should go. /etc/syslog.conf
consists of two columns and looks something like this:
kern.warn;*.err;authpriv.none /dev/tty10
kern.warn;*.err;authpriv.none |/dev/xconsole
*.emerg *
*.=warn;*.=err -/var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none -/var/log/messages
Each rule consists of a selector and a destination, separated by white space.
The first column (i.e., the selector) of each line determines which messages will be selected and can be made up or one or more patterns. Each pattern is a facility and a priority (severity), separated by a period (.
). Each pattern is separated by a semicolon (;
). A wildcard (*
) can be used in the pattern.
The second column (i.e., the destination) determines where the messages go. A -
in front of a destination tells syslogd
that it should not commit each log entry to disk as it is logged, but to let the kernel write to disk when it has time. If a destination is just a wildcard (*
), the message is sent to the console of any logged in users.
Comments in this file start with a sharp (#
).
syslogd
supports logging to a remote host. Moving your log files from the local system to a different computer on the network can be a valuable administrative and security measure.
For example, you could add a line like this to /etc/syslog.conf
:
*.* @ex_IP_address_of_loghost
Then, you would restart syslogd
to activate the change:
# service syslog restart
Facilities and Priorities
The <facility>
denotes the system program or component creating the log message. If you specify a *
in place of a facility, this serves as a place holder for any facility.
It is not easily possible to define additional facilities. However, the local
facilities local0
to local7
serve as user-definable facilities and should be sufficient for most purposes.
Show/Hide Log Facilities
Tag | Type of Messages |
---|---|
auth |
Security logs that can be public. |
authpriv |
Security logs that need to be private. |
cron |
Scheduled jobs such as cron and at . |
daemon |
Other system daemons. |
kern |
Kernel messages. |
local0-7 |
Eight different user-definable facilities. |
lpr |
Printing. |
mail |
Email server. |
news |
news daemon messages. |
syslog |
Internal messages for syslog. |
user |
Random user level messages. |
uucp |
Log messages from the uucp (UNIX-to-UNIX Copy) daemon. |
The <priority>
specifies how serious the message is.
Show/Hide syslog Priorities
Level | Tag | Meaning |
---|---|---|
0 |
emerg |
Emergency: The system is unusable. |
1 |
alert |
Alert: Immediate action is needed to prevent a failure. |
2 |
crit |
Critical: The system has reached a critical level. |
3 |
err |
Error: Some part of the system encountered an error. |
4 |
warn |
Warning: Something happened that may have been processed incorrectly warning. |
5 |
notice |
Notice: Not an error condition, but may need special handling. |
6 |
info |
Informational: A normal log entry about a routine event. |
7 |
debug |
Debug: A message about internal state that is for debugging problems. |
No level | none |
No priority in the proper sense. Serves to exclude all messages from a certain facility. |
A message is logged with a facility and a priority. Whomever uses the syslog
function, namely the developer of the program in question, must assign a facility and priority to their code's messages. Many programs allow the administrator to at least redefine the message facility.
syslogd
takes care of processing the message. Different facilities can be logged at different priority levels.
When you log at a particular level, you are saying that you are only capturing logs at that or a higher priority (i.e., a lower priority level number) and discarding anything else. If you just want to capture messages of a single priority, you can do this by using the -
character (e.g., ex_facility.-ex_priority
).
The *
character can be used to target messages of any priority (e.g., ex_facility.*
. Also, you could just specify the debug
priority, ex_facility.debug
).
A preceding !
denotes negation (e.g., lpr.!info
deselects messages from the printing subsystem at a priority of info
and higher). !
and =
may be combined (e.g., lpr.!=info
deselects exactly those messages from the printing subsystem with priority info
).
You can also specify multiple facilities with the same priority (e.g., lpr,mail.info
, which selects messages of priority info
and higher that belong to the lpr
or mail
facilities.
Log Message Management
Log messages can be handled in different ways:
-
They can be written to a file. The filename must be specified as an absolute path. If there is a
-
in front of the path, the file will not immediately be written to disk (unlike normalsyslogd
operations).This means that in case of a system crash, you might lose pending log messages. The filename may also refer to a device file (e.g.,
/dev/tty3
). -
Log messages can be written to a named pipe (First In, First Out; FIFO). The FIFO name must be given as an absolute path with a preceding
|
. -
They can be passed across the network to another
syslogd
instance. This is specified as the name or IP address of the target system with a preceding@
character.This is especially useful if a critical system state occurs that renders the local log file inaccessible, as it deprives malicious actors from hiding their activity trail, and enables administrators to collect the log messages of all hosts in a network on a single computer for easier processing.
On the target host,
syslogd
must have been started using the-r
(remote) option in order to accept forwarded messages. -
They can be sent directly to users. The usernames in question must be given as a comma-separated list. The message will be displayed on the listed users' terminals if they are logged in when the message arrives.
- They can be sent to all logged-in users by specifying a
*
in place of a login name.
Log files are generally created below /var/log/
, but the specific filenames vary.
Some log files contain messages concerning users' privacy and should only be readable by the root user. Most distributions err towards caution and restrict access rights to all log files.
The log files created by syslogd
can be perused using the less
and tail
commands. Following a log with tail
's -f
(--follow
) option is particularly useful.
The messages written by syslogd
normally contain the date and time, the hostname, a hint about the process or component that created the message, and the message itself.
Typical messages may look like this:
Mar 31 09:56:09 tails modprobe: modprobe: Can't locate ...
Mar 31 11:10:08 tails su: (to root) user1 on /dev/pts/2
Mar 31 11:10:08 tails su: pam-unix2: session started for ...
An overly large log file can be removed using the rm
command (it is probably wise to save the log file first by renaming it with an extension like .old
). A new log file will be created when syslogd
is next restarted. There are also more convenient methods to remove large log files (e.g., logrotate
).
Conventional log locations include:
/var/log/boot.log
- Log entries from daemons as they were started during bootup.
/var/log/boot.msg
- All of the messages displayed on screen during system boot.
/var/log/cron
- Logs of scheduled job activity.
/var/log/dmesg
- Hardware detection information.
/var/log/exim4/
- All logs related to email (Debian).
/var/log/failog
- Failed authentication attempts.
/var/log/firewall
- Firewall log entries.
/var/log/maillog
- All logs related to email (Fedora).
/var/log/messages
- General purpose log messages that are not in one of the other logging files.
/var/log/rpmpkgs
- A list of installed RPM packages (Fedora).
/var/log/secure
- Security logs.
/var/log/warn
- Warning messages.
/var/log/wtmp
- Log entries from the
xinetd
daemon. /var/log/xferlog
- Logs of local File Transfer Protocol (FTP) server activity.
The Linux kernel does not send its log messages to syslogd
, but puts them into an internal ring buffer. They can be read from there in various ways, e.g., via a specialized system call or the /proc/kmsg
file. Traditionally, a program called klogd
is used to read /proc/kmsg
and pass the messages on to syslogd
.
During system startup, syslogd
(and possibly klogd
) are not immediately available. They must be started as programs and therefore cannot directly handle the kernel's start messages. The dmesg
command makes it possible to access the kernel log buffer retroactively and look at the system start log. With a command like # dmesg > boot.msg
, you can write these messages to a file for further perusal or debugging purposes.
You can use the dmesg
command to delete the kernel ring buffer and set a priority for direct notifications using dmesg
's -c
(--clear
) and -n
(--console-level ex_level
) options, respectively. Messages meeting or exceeding the priority that you set will be immediately sent to the console.
Kernel messages have priorities from 0
to 7
corresponding to the syslogd
priorities from emerg
down to debug
, respectively. For example, the # dmesg -n 1
command causes only alert
and emerg
messages to be directly written to the console.
All message will be written to /proc/kmsg
in every case. Here, it is the job of post-processing software, like syslogd
, to suppress unwanted messages.
dmseg
can tell you system information about the amount and usage of Random Access Memory (RAM), available Central Processing Units (CPUs), disks and other mass storage devices (e.g., IDE and SCSI), Universal Serial Bus (USB) devices, and network cards.
logger
The syslogd
mechanism can be tested using the logger
program, along with its -p ex_priority
(--priority ex_priority
) and -t ex_tag
(--tag ex_tag
) options:
logger -p ex_facility.ex_priority -t ex_tag ex_message
ex_priority
can be specified numerically or in the ex_facility.ex_priority
form shown above. For example, logger -p local0.err -t TEST 'Hello, world!'
produces a log message of the form Sep 20 17:27:45 tails TEST: Hello, world!
.
To include the PID in the log entry, add the -i
option.
If you want to log more messages than are already being collected by syslogd
, you can modify the /etc/syslog.conf
file and send syslogd
a SIGHUP
signal to get it to re-read its configuration file.
Documentation
For more information on syslogd
logging, run the following commands:
man 3 syslog
man 8 syslogd
man 5 syslog.conf
man 1 logger
Also, syslog protocol information can be found in RFC 5424.
rsyslogd
The rsyslogd
configuration file is /etc/rsyslog.conf
and it specifies where log messages should go. It is largely compatible with what syslogd
uses. rsyslogd
gets by without a separate klogd
program because it directly takes care of kernel log messages by itself.
Besides greater efficiency, rsyslog's goal is to support various sources and sinks for log messages. For example, it writes messages not just to text files and terminals, but also to a wide selection of databases.
The basic ideas behind rsyslog are:
- Source pass messages to rulesets. There is one standard built-in ruleset (
RSYSLOG_DefaultRuleset
) but, as the user, you get to define others. - Every ruleset may contain arbitrarily many rules.
- A rule consists of a filter and an action list. Filters make yes-no decisions about whether the corresponding action list will be executed.
- For each message, all of the rules in the ruleset will be executed in order from the first to the last (and no others). All rules will always be executed, no matter how the filter decisions go, although there is a stop processing action.
- An action list may contain many actions (at least one). Within an action list, no further filters are allowed. The actions determine what happens to matching log messages.
- The exact appearance of log messages in the output may be controlled through templates.
/etc/rsyslog.conf
In /etc/rsyslog.conf
, you may use three different styles of configuration settings in parallel:
- The traditional
/etc/syslog.conf
syntax (sysklogd
). - An obsolete rsyslog syntax (legacy rsyslog). This syntax can be recognized by commands that start with dollar signs (
$
). - The current rsyslog syntax (RainerScript). This syntax is best suited for complex situations.
sysklogd
syntax and legacy rsyslog syntax are line-based. In RainerScript, line breaks are irrelevant.
For simple applications, you can, and likely should, use the sysklogd
syntax. If you want to set configuration parameters or express complex control flows, RainerScript is probably more appropriate. It is likely best to avoid the obsolete rsyslog syntax, unless you need to use a rsyslog feature that is only accessible using that syntax.
Empty lines and comment lines in /etc/rsyslog.conf
are ignored. Comment lines include both lines (and parts of lines) that start with a #
(the comment then stops at the end of the line) and C-style comments that go from a /*
, disregarding line breaks, until a */
.
C-style comments may not be nested, but #
comments may occur inside C-style comments.
Extended Filter Expressions
rsyslog offers various features that surpass those of BSD syslogd
, e.g., you can use extended filter expressions for messages. Extended filter expressions always consist of a colon at the left margin, a property that rsyslog takes from the message, a filter operator, and a search term.
:ex_property, ex_filter_operator, ex_search_term ex_destination
For example, the following sends all log messages whose text contains the character sequence FOO
to the /var/log/foo.log
file:
:msg, contains, "FOO" /var/log/foo.log
Apart from msg
(the log message proper), the properties you may use include:
fromhost
- The name of the computer that forwarded the message to rsyslog.
hostname
- The name of the computer sending the message.
input-name
- The rsyslog module name of the source of the message.
pri
- The category and priority as a text string, with the number appended, as in
local0.err<133>
. syslogfacility
,syslogseverity
,syslogfacility-text
, andsyslogseverity-text
- For direct access to the category and priority.
timegenerated
- When the message was received.
The allowable comparison operators are:
contains
eregex
isequal
regex
startswith
regex
considers its parameters as a simple regular expression and eregex
as an extended regular expression according to POSIX. All comparison operators take upper and lower case into account.
The startswith
comparison is useful because it is more efficient than a regular expression that is anchored to the start of the message (as long as you are looking for a constant string).
However, you need to consider that the start of the message and what rsyslog thinks of that can be quite different. For example, if rsyslog receives a message via the syslog service, this will look something like <131>Jul 22 14:25:50 root: error found
.
As far as rsyslog is concerned, mgs
does not start at the e
of error
, but with a space character in front of it. So, if you are looking for messages that start with error
, you should say :msg, startswith, " error" /var/log/error.log
.
With traditional syslogd
, an entry like local0.* @enceladus.example.com
will forward log messages to a remote host via the (UDP-based) syslog protocol. With rsyslogd
, you may also write local0.* @@enceladus.example.com
to transmit log messages via TCP.
Potentially, this is more reliable, especially if firewalls are involved. At the other end of the TCP connection, there must be a suitably configured rsyslog listening for messages. You can ensure this in /etc/rsyslog.conf
like so:
module(load="imtcp" MaxSessions="500")
input(type="imtcp" port="514")
This does the same thing, but in the obsolete syntax:
$ModLoad imtcp
$InputTCPMaxSessions 500
$InputTCPServerRun 514
Consider that only the UDP port 514
is officially reserved for the syslog protocol. The TCP port 514
is used for a different purpose. You can specify a different port, e.g., local0.* @@enceladus.example.com:10518
(this works for UDP, too, if required). The changes required on the server will be easy to deduce.
Filters can be based on expressions that may contain arbitrary Boolean, arithmetic, or string operations. These always start with an if
at the beginning of a new line:
if $syslogfacility-text == "local0" and $msg startswith " FOO" and \
($msg contains "BAR" or $msg contains "BAZ") then /var/log/foo.log
With this rule, messages of category local0
will be written to the /var/log/foo.log
file, as long as they start with FOO
and also contain either BAR
or BAZ
(or both). Watch for dollar signs at the start of the property names.
Modules
rsyslog supports a large number of modules that determine what should happen to log messages. For example, you might forward important messages by email. To do so, you might put something like this into your /etc/rsyslog.conf
file:
module(load="ommail")
template(name="mailBody" type="string" string="ALERT\\r\\n%msg%")
if $msg contains "disk error" then {
action(type="ommail" server="mail.example.com" port="25"
mailfrom="rsyslog@example.com" mailto="admin@example.com"
subject.text="Error Detected"
body.enable="on" template="mailBody"
action.execonlyonceeveryinterval="3600")
}
Keep in mind, rsyslog's SMTP implementation is fairly primitive, since it does not support either encryption or authentication. This means that the mail server you specify in the rsyslog configuration must be able to accept mail from rsyslog, even without encryption or authentication.
rsyslog can directly handle Linux kernel log messages. You simply need to enable the imklog
input module:
module(load="imklog")
A separate klogd
process is not required.
Documentation
For more information on rsyslogd
logging, run the following commands:
man 8 rsyslogd
man 5 rsyslog.conf
Also, check out the rsyslog site and the official project repository.
syslog-ng
syslog-ng (Next Generation) is a compatible, but extended, re-implementation of a syslog daemon. The main advantages of syslog-ng compared to the traditional syslogd
include:
- Filtering of messages based on their content (not just categories and priorities).
- Chaining of several filters.
- A more sophisticated input/output system, including forwarding by TCP and to subprocesses.
The logging program itself is called syslog-ng
. For syslog clients, there is no difference, i.e., you can replace a syslogd
with syslog-ng without any problems.
syslog-ng reads its configuration from a file, normally /etc/syslog-ng/syslog-ng.conf
. Unlike syslogd
, syslog-ng distinguishes various entry types in its configuration file.
Filters
Filters are Boolean expressions based on internal functions that can, for example, refer to the origin, category, priority, or textual content of a log message. Filters are also named.
They are used to sift through log messages or distribute them to various sinks. They rely on internal functions that consider specific aspects of messages. These functions can be joined using the logical operators and
, or
, and not
.
These are various possible forms of filter syntax:
facility( ex_category... )
- Matches messages with one of the listed categories. Multiple categories can be provided as a comma-separated list.
filter( ex_name )
- Invokes another filtering rule and returns its value.
host( ex_regex )
- Matches messages whose sending host matches
ex_regex
. level( ex_priority... )
- Matches messages with one of the listed priorities. Multiple priorities can be provided as a comma-separated list.
match( ex_regex )
- Matches messages that match the
ex_regex
themselves. netmask( ex_ip_address / ex_netmask )
- Checks whether the IP address is in the given network.
priority( ex_priority... )
- Same as
level()
. Multiple priorities can be provided as a comma-separated list. program( ex_regex )
- Matches messages where the name of the sending program matches
ex_regex
.
For example, you could define a filter that matches all messages from host europa
containing the text error
:
filter f_europa { host("europa") and match("error"); };
With the level()
(or priority()
) filter, you can specify either one or more priorities separated by commas, or else a range of priorities like warn..emerg
.
Log Paths
A log path connects one or several message sources, filters, and sinks. If messages arrive from the sources and pass the filter (or filters), they will be forwarded to the specified sink(s). The configuration file consists of a number of such log paths.
Log paths bring sources, filters, and sinks together to evaluate messages. They always start with the log
keyword.
The following rule causes all messages to do with authentication to be written to the /var/log.auth.log
file:
# Prerequisites
source s_all { internal(); unix-stream("/dev/log"); };
filter f_auth { facility(auth, authpriv); };
destination df_auth { file("/var/log/auth.log"); };
# auth,authpriv.* /var/log/auth.log
log {
source(s_all);
filter(f_auth);
destination(df_auth);
};
Below, every message passes through all log paths and is logged by all matching ones.
# kern.warn;*.err;authpriv.none /dev/tty10
filter f_nearly_all {
(facility(kern) and priority(warn .. emerg))
or (not facility(authpriv,kern));
};
destination df_tty { file("/dev/tty10"); };
log {
source(s_all);
filter(f_nearly_all);
destination(df_tty);
};
If you want a message to not be further considered after it has passed a particular log path, you can add the flags(final)
option to that path. flags(final)
does not mean that the message is logged just once. It might have been logged by other paths before the path in question.
With flags(fallback)
, you can declare a path to be the default path. This path is only considered for log messages that did not match any paths that were not marked flags(fallback)
.
In the next example, whenever the amnesia
user logs on or off, a message is displayed on the system admin's (root
's) terminal, if they are logged in:
# We assume a suitable source definition.
filter login_amnesia {
facility(authpriv)
and (match("session opened") or match("session closed"))
and match("user amnesia");
};
destination d_root { usertty("root"); };
log { source(...);
filter(login_amnesia);
destination(d_root);
};
Message Sinks
syslog-ng includes all logging methods of syslogd
and more. Sinks consist of various drivers for logging methods. For example, you can write messages to a file:
destination d_file { file("/var/log/messages"); };
Also, you can specify a template that describes in which format the message should be written to the sink in question. When doing so, you can refer to macros that make various parts of the message accessible.
destination d_file {
file("/var/log/$YEAR.$MONTH.$DAY/messages"
template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG\n")
template_escape(no)
create_dirs(yes)
);
};
The $YEAR
, $MONTH
, and comparable macros above are replaced by the obvious values. For example, $TZ
is the current time zone, $LEVEL
is the message priority, and $MSG
is the message itself (including the sender's PID).
The template_escape()
parameter controls whether quotes ('
and "
) should be escaped in the output. This is important if you want to feed the log messages to something like a SQL server.
Unlike syslogd
, syslog-ng allows forwarding messages using TCP. This is more convenient when firewalls are involved and ensures that no log messages are lost (which may happen with UDP).
A TCP forwarding sink can be defined like so:
destination d_tcp { tcp("10.11.12.13" port(518); localport(518)); };
It is also useful to forward messages to programs using program()
. syslog-ng starts the program when it is started itself, and keeps it running until it is stopped or it receives a SIGHUP
signal.
This is not just to increase efficiency, but serves as a precaution against denial-of-service attacks, e.g., if a new process is started for every message, an attacker could shut off logging by sending large amounts of matching log messages (other messages that would point to these activities might then be dropped).
Message Sources
syslog-ng can read messages in various ways (e.g., from UNIX-domain sockets or UDP like syslogd
, and from files, FIFOs, or TCP sockets). Every message source is assigned a name.
In syslog-ng, message sources are defined using the source
keyword. A message source collects one or more drivers. To accomplish the equivalent of a normal syslogd
, you would include the line source src { unix-stream("/dev/log"); internal(); };
in your configuration. This tells syslog-ng to listen to the UNIX-domain socket /dev/log
. internal()
refers to messages that syslog-ng creates by itself.
A syslog-ng message source corresponding to the -r
option of syslogd
might look like this:
source s_remote { udp(ip(0.0.0.0) port(518)); };
Since that is the default setting, source s_remote { udp(); };
would also be sufficient.
With ip()
, you can let syslog-ng listen on specific local IP addresses only. This is not possible with syslogd
.
The following source specification lets syslog-ng replace the klogd
program:
source kmsg { file("/proc/kmsg" log_prefix("kernel: ")); };
All message sources support another parameter, log_msg_size()
, which specifies the maximum message length in bytes.
Global Options
You can specify various global options that control syslog-ng's general behavior or determine default values for individual message sources or sinks (specific options for the sources or sinks take priority). The general options include various settings for handling DNS and the forwarding or rewriting of messages' sender hostnames.
If syslog-ng on host A
receives a message from host B
, it checks the keep_hostnames()
option. If its value is yes
, B
is kept as the hostname for the log. If not, the outcome depends on the chain_hostnames()
option.
If this is no, then A
will be logged as the hostname. If it is yes
, then syslog-ng will log B/A
. This is particularly important if the log is then forwarded to yet another host.
Documentation
For more information on syslog-ng logging, run the following commands:
man 8 syslog-ng
man 5 syslog-ng.conf
Also, check out the syslog-ng site and the official project repository.
Log Rotation
Log files can quickly grow. To keep a system from being inundated with log data, you may want to put the relevant directories (e.g., /var/log/
or /var/
) on their own partitions.
On the other hand, there is software that periodically checks the log files according to various criteria (e.g., the size), truncates them, and removes or archives old log files. This process is called rotation and a program that performs rotation is logrotate
.
Log rotation is the process that renames a current log file (e.g., auth.log
becomes auth.log.1
) and sets up a new log file (auth.log
) for new log entries. Older log files may be compressed.
logrotate
is not a daemon, but is usually executed once a day using cron
or a similar service. logrotate
refuses to modify a log file more than once a day, except:
- If the decision depends on the size of the log file.
- You are using the
hourly
criterion. - You are using
logrotate
's-f
(--force
) option.
/etc/logrotate.conf and /etc/logrotate.d/
According to convention, logrotate
is configured using the /etc/logrotate.conf
file and the files within the /etc/logrotate.d/
directory. The /etc/logrotate.conf
file sets up general parameters, which can be overwritten by the files in /etc/logrotate.d/
, if necessary.
The format of the /etc/logrotate.conf
file is:
ex_path_to_log_file_glob {
ex_instruction_1
ex_instruction_2
}
In /etc/logrotate.conf
, there is an include /etc/logrotate.d/
line, which causes the files from that directory to be read in that place, as if they were part of the /etc/logrotate.conf
file itself.
In principle, logrotate
reads all of the files named on the command line as configuration files, and the content of files mentioned later on overwrites that of files mentioned earlier. The /etc/logrotate.conf
file is a convention that is put into action by means of a suitable invocation of logrotate
in /etc/cron.daily/logrotate
.
logrotate
watches all files that it is told about by the aforementioned configuration files, not just those created by syslogd
.
Typically, /etc/logrotate.conf
contains directives that are outside of a brace-delimited block. These directives serve as defaults that apply to all log files in the configuration, unless something more specific is given in their own blocks of directives.
/etc/logrotate.conf
directives include:
compress
- Old versions of log files are compressed with
gzip
by default. daily
- Log files are rotated every day.
dateformat ex_format_string
- Specify the extension for
dateext
using the notation similar to thestrftime(3)
function. Only%Y
,%m
,%d
,%H
,%M
,%S
,%V
, and%s
specifiers are allowed. The default value is-%Y%m%d
except hourly, which uses-%Y%m%d%H
as the default value. delaycompress
- Do not compress the file until it has already been rotated.
mail ex_address
- When a log is rotated out of existence, it is mailed to
ex_address
. If no mail should be generated by a particular log, thenomail
directive may be used. maxsize ex_size
- Log files are rotated when they grow bigger than
ex_size
, even before the additionally specified time interval (daily
,weekly
,monthly
, oryearly
). minsize ex_size
- Log files are rotated when they grow bigger than
ex_size
, but not before the additionally specified time interval (daily
,weekly
,monthly
, oryearly
). missingok
- If a log is missing, do not raise an error.
monthly
- Log files are rotated the first time
logrotate
is run in a month (this is normally on the first day of the month). notifempty
- If empty, the log is not rotated.
postrotate/endscript
- Run the commands that follow, up until the
endscript
keyword, after the logs are rotated. rotate ex_num
- Rotates a given log
ex_num
times before deleting it. size ex_size
- Log files are rotated only if they grow bigger than size bytes.
- If size is followed by
k
, the size is assumed to be in kilobytes. IfM
is used, the size is in megabytes, and ifG
is used, the size is in gigabytes. So, size100
, size100k
, size100M
, and size100G
are all valid. sharedscripts
- If the wildcard specified matches several files, run any scripts once for all files.
weekly
- Rotates the log files once a week.
yearly
- Log files are rotated if the current year is not the same as the last rotation.
Example Configuration
Below is an example configuration for logrotate
:
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
invoke-rc.d rsyslog rotate >/dev/null
endscript
}
rotate 7
The first line specifies the files that this configuration applies to (e.g., /var/log/syslog
). You may enumerate several files or specify shell search patterns. After that, inside curly braces, there is a block of directives that define how logrotate
should deal with the given files.
rotate 7
means that, at most, seven old version of each log file will be kept. When this maximum is reached, the oldest version of the log file is deleted.
If you specify an address using mail
, files will not be deleted, but instead be sent to the email address in question. rotate 0
deletes rotated log messages outright.
The rotated files are numbered in sequence, i.e., if the current version of the file is called /var/log/syslog
, the immediately preceding version will be /var/log/syslog.1
, and the version preceding that will be /var/log/syslog.2
.
Instead of sequential numbers, you may use the date. For example, if today is July 19, 2021, and your logrotate
run takes place daily in the morning, the immediately preceding version of the file will be /var/log/syslog-20210719
, and the version preceding that will be called /var/log/syslog-20210718
.
Using dateformat
, you can control exactly how the date-based file extension should look. To do so, you need to specify a string that may contain the %Y
, %m
, %d
, and %s
keys. These stand for the four-digit year, calendar month, calendar day (in each case two digits and, if necessary, with a leading zero), and seconds since 1st January 1970, 12:00 AM UTC, the UNIX epoch. The default is -%Y%m%d
.
When you use dateformat
, note that logrotate
does a lexicographic sort of filenames when rotating in order to find out which file is the oldest. This works with -%Y%m%d
, but not with -%d%m%Y
.
daily
daily
means that log files should be rotated daily. Together with rotate 7
, this implies that you always have access to last week's logs.
There are also weekly
, monthly
, and yearly
values. With weekly
, the file is rotated when the current day of the week is earlier than the day of the week of the last rotation, or more than one week has passed since the last rotation (i.e., rotation will take place on the first day of the week, which, according to United States custom, is Sunday).
With monthly
, the file is rotated on the first logrotate
run of the month (usually on the first of the month). With yearly
, rotation takes place on the first logrotate
run of the year. Theoretically, hourly
rotates the log file every hour, but since logrotate
is normally only run once a day, you will have to arrange for it to be run frequently enough.
An alternative criterion is size
. This will rotate a log file when a certain size has been exceeded. The file size is given as a parameter. Without a unit, it will be taken to mean bytes, while the units k
(or K
), M
, and G
stand for kibibytes (2^10
bytes), mebibytes (2^20
) bytes, or gibibytes (2^30
bytes), respectively.
size
and the time-based criteria are mutually exclusive, i.e., if you specify a size
criterion, rotation will solely depend on file size, no matter when the file was last rotated. File size and time can be used together by means of the maxsize
and minsize
criteria.
With maxsize
, you can specify a size that will cause logrotate
to rotate the file even if the next official date has not been reached. With minsize
, the file will only be rotated at the specified point in time if it has exceeded the given size (small files will be skipped).
missingok, notifempty
missingok
suppresses error messages if a log file could not be found (the default is nomissingok
). notifempty
does not rotate a file if it is empty (the default here is ifempty
).
delaycompress, compress
The delaycompress
directive ensures that a freshly rotated file is not immediately compressed after the rotation, but only on the next run. compress
lets you specify that rotated versions of the log file should be compressed.
Usually, the sequence of files would look like:
/var/log/syslog /var/log/syslog.1.gz /var/log/syslog.2.gz ex_code
delaycompress
would produce a sequence like:
/var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz ex_code
In other words, /var/log/syslog.1
remains uncompressed. You need this setting if there is a chance that the logging program (like rsyslog) might append data to the file after it has been renamed (i.e., rotated). This can happen because rsyslog keeps the log file open and renaming the file is irrelevant, as far as writing to it is concerned.
postrotate, endscript
You need to notify rsyslog that there is a new log file. This is what this directive is for:
postrotate
invoke-rc.d rsyslog rotate >/dev/null
endscript
The shell commands between postrotate
and endscript
are executed by logrotate
whenever the log file has been rotated. In the end, rsyslog's init script is invoked and it sends SIGHUP
to the program. Then, SIGHUP
causes rsyslog to re-read its configuration file, and close and reopen all log files.
Since /var/log/syslog
was previously renamed, rsyslog opens a new log file under that name. At this point, logrotate
could compress the /var/log/syslog.1
file, but it has no way of knowing when rsyslog is really done with the file. This is why this is postponed until the file gets rotated again.
Between postrotate
and endscript
, there may be several lines with commands. logrotate
concatenates them all and passes them to the shell (/bin/sh
) as a whole. The command is passed the name of the log file as a parameter and that is available there in the customary fashion as ${1}
.
The postrotate
commands are executed once for every log file enumerated at the start of the configuration block, i.e., the commands may be executed several times. You can use the sharedscripts
directive to ensure that the commands are executed at most once for all files that match the search pattern (or not at all, if none of the files needed to be rotated).
You can use create
to make sure that the log file is immediately recreated after the rotation and before the postrotate
commands are executed. This uses the name of the old file. The file mode, owner, and group derive from the parameters to create
.
The three possibilities are:
create 600 root adm
File mode, user, and group.create root adm
Just user and group.create
Nothing at all.
Unspecified file properties are taken from the previous version of the file. Run man 8 logrotate
for the full list of configuration parameters.
Documentation
You can find more information on the commands discussed above by examining the Linux User's Manual, either at the command line or online.