Systemd

From Bauman National Library
This page was last modified on 20 December 2015, at 16:19.
systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.

Basic systemctl usage

The main command used to introspect and control systemd is systemctl. Some of its uses are examining the system state and managing the system and services.

Analyzing the system state

List running units:

$ systemctl

or:

$ systemctl list-units

List failed units:

$ systemctl --failed

List installed' unit files with:

$ systemctl list-unit-files

Using units

Units can be, for example, services (.service), mount points (.mount), devices (.device) or sockets (.socket).

When using systemctl, you generally have to specify the complete name of the unit file, including its suffix. There are however a few short forms when specifying the unit in the following systemctl commands:

  • If you do not specify the suffix, systemctl will assume .service.
  • Mount points will automatically be translated into the appropriate .mount unit.
  • Similar to mount points, devices are automatically translated into the appropriate .device unit, therefore specifying /dev/sda2 is equivalent to dev-sda2.device.

Start a unit immediately:

# systemctl start unit

Stop a unit immediately:

# systemctl stop unit

Restart a unit:

# systemctl restart unit

Ask a unit to reload its configuration:

# systemctl reload unit

Show the status of a unit, including whether it is running or not:

$ systemctl status unit

Check whether a unit is already enabled or not:

$ systemctl is-enabled unit

Enable a unit to be started on bootup:

# systemctl enable unit

Disable a unit to not start during bootup:

# systemctl disable unit

Mask a unit to make it impossible to start it:

# systemctl mask unit

Unmask a unit:

# systemctl unmask unit

Show the manual page associated with a unit (this has to be supported by the unit file):

$ systemctl help unit

Reload systemd, scanning for new or changed units:

# systemctl daemon-reload

Power management

polkit is necessary for power management as an unprivileged user. If you are in a local systemd-logind user session and no other session is active, the following commands will work without root privileges. If not (for example, because another user is logged into a tty), systemd will automatically ask you for the root password.

Shut down and reboot the system:

$ systemctl reboot

Shut down and power-off the system:

$ systemctl poweroff

Suspend the system:

$ systemctl suspend

Put the system into hibernation:

$ systemctl hibernate

Put the system into hybrid-sleep state (or suspend-to-both):

$ systemctl hybrid-sleep

Writing unit files

The syntax of systemd's unit files is inspired by XDG Desktop Entry Specification .desktop files, which are in turn inspired by Microsoft Windows .ini files. Unit files are loaded from two locations. From lowest to highest precedence they are:

  • /usr/lib/systemd/system/: units provided by installed packages
  • /etc/systemd/system/: units installed by the system administrator

Handling dependencies

With systemd, dependencies can be resolved by designing the unit files correctly. The most typical case is that the unit A requires the unit B to be running before A is started. In that case add Requires=B and After=B to the [Unit] section of A. If the dependency is optional, add Wants=B and After=B instead. Note that Wants= and Requires= do not imply After=, meaning that if After= is not specified, the two units will be started in parallel.

Dependencies are typically placed on services and not on targets. For example, network.target is pulled in by whatever service configures your network interfaces, therefore ordering your custom unit after it is sufficient since network.target is started anyway.

Service types

There are several different start-up types to consider when writing a custom service file. This is set with the Type= parameter in the [Service] section:

  • Type=simple (default): systemd considers the service to be started up immediately. The process must not fork. Do not use this type if other services need to be ordered on this service, unless it is socket activated.
  • Type=forking: systemd considers the service started up once the process forks and the parent has exited. For classic daemons use this type unless you know that it is not necessary. You should specify PIDFile= as well so systemd can keep track of the main process.
  • Type=oneshot: this is useful for scripts that do a single job and then exit. You may want to set RemainAfterExit=yes as well so that systemd still considers the service as active after the process has exited.
  • Type=notify: identical to Type=simple, but with the stipulation that the daemon will send a signal to systemd when it is ready. The reference implementation for this notification is provided by libsystemd-daemon.so.
  • Type=dbus: the service is considered ready when the specified BusName appears on DBus's system bus.
  • Type=idle: systemd will delay execution of the service binary until all jobs are dispatched. Other than that behavior is very similar to Type=simple.

See the systemd.service(5) man page for a more detailed explanation of the Type values.

Editing provided units

To avoid conflicts with pacman, unit files provided by packages should not be directly edited. There are two safe ways to modify a unit without touching the original file: create a new unit file which overwrites the original unit or create drop-in snippets which are applied on top of the original unit. For both methods, you must reload the unit afterwards to apply your changes. This can be done either by editing the unit with systemctl edit (which reloads the unit automatically) or by reloading all units with:

# systemctl daemon-reload

Replacement unit files

To replace the unit file /usr/lib/systemd/system/unit, create the file /etc/systemd/system/unit and reenable the unit to update the symlinks:

# systemctl reenable unit

Alternatively, run:

# systemctl edit --full unit

This opens /etc/systemd/system/unit in your editor (copying the installed version if it does not exist yet) and automatically reloads it when you finish editing.

Drop-in snippets

To create drop-in snippets for the unit file /usr/lib/systemd/system/unit, create the directory /etc/systemd/system/unit.d/ and place .conf files there to override or add new options. systemd will parse these .conf files and apply them on top of the original unit.

The easiest way to do this is to run:

# systemctl edit unit

This opens the file /etc/systemd/system/unit.d/override.conf in your text editor (creating it if necessary) and automatically reloads the unit when you are done editing.

Targets

systemd uses targets which serve a similar purpose as runlevels but act a little different. Each target is named instead of numbered and is intended to serve a specific purpose with the possibility of having multiple ones active at the same time. Some target are implemented by inheriting all of the services of another target and adding additional services to it. There are systemd targets that mimic the common SystemVinit runlevels so you can still switch targets using the familiar telinit RUNLEVEL command.

Get current targets

The following should be used under systemd instead of running runlevel:

$ systemctl list-units --type=target

Create custom target

The runlevels that held a defined meaning under sysvinit (i.e., 0, 1, 3, 5, and 6); have a 1:1 mapping with a specific systemd target. Unfortunately, there is no good way to do the same for the user-defined runlevels like 2 and 4. If you make use of those it is suggested that you make a new named systemd target as /etc/systemd/system/your target that takes one of the existing runlevels as a base (you can look at /usr/lib/systemd/system/graphical.target as an example), make a directory /etc/systemd/system/your target.wants, and then symlink the additional services from /usr/lib/systemd/system/ that you wish to enable.

Targets table

SysV Runlevel systemd Target Notes
0 runlevel0.target, poweroff.target Halt the system.
1, s, single runlevel1.target, rescue.target Single user mode.
2, 4 runlevel2.target, runlevel4.target, multi-user.target User-defined/Site-specific runlevels. By default, identical to 3.
3 runlevel3.target, multi-user.target Multi-user, non-graphical. Users can usually login via multiple consoles or via the network.
5 runlevel5.target, graphical.target Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login.
6 runlevel6.target, reboot.target Reboot
emergency emergency.target Emergency shell

Change current target

In systemd targets are exposed via target units. You can change them like this:

# systemctl isolate graphical.target

This will only change the current target, and has no effect on the next boot. This is equivalent to commands such as telinit 3 or telinit 5 in Sysvinit.

Change default target to boot into

The standard target is default.target, which is aliased by default to graphical.target (which roughly corresponds to the old runlevel 5). To change the default target at boot-time, append one of the following kernel parameters to your bootloader:

  • systemd.unit=multi-user.target (which roughly corresponds to the old runlevel 3),
  • systemd.unit=rescue.target (which roughly corresponds to the old runlevel 1).

Alternatively, you may leave the bootloader alone and change default.target. This can be done using systemctl:

# systemctl set-default multi-user.target

To be able to override the previously set default.target, use the force option:

# systemctl set-default -f multi-user.target

The effect of this command is output by systemctl; a symlink to the new default target is made at /etc/systemd/system/default.target.

Journal

systemd has its own logging system called the journal; therefore, running a syslog daemon is no longer required. To read the log, use:

# journalctl

In Arch Linux, the directory /var/log/journal/ is a part of the systemd package, and the journal (when Storage= is set to auto in /etc/systemd/journald.conf) will write to /var/log/journal/. If you or some program delete that directory, systemd will not recreate it automatically and instead will write its logs to /run/systemd/journal in a nonpersistent way. However, the folder will be recreated when you set Storage=persistent and run systemctl restart systemd-journald (or reboot).

Filtering output

journalctl allows you to filter the output by specific fields. Be aware that if there are many messages to display or filtering of large time span has to be done, the output of this command can be delayed for quite some time.

While the journal is stored in a binary format, the content of stored messages is not modified. This means it is viewable with strings, for example for recovery in an environment which does not have systemd installed. Example command:

$ strings /mnt/arch/var/log/journal/af4967d77fba44c6b093d0e9862f6ddd/system.journal | grep -i message

Examples:

  • Show all messages from this boot:
# journalctl -b

However, often one is interested in messages not from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the -b flag: journalctl -b -0 shows messages from the current boot, journalctl -b -1 from the previous boot, journalctl -b -2 from the second previous and so on. See man 1 journalctl for full description, the semantics is much more powerful.

  • Show all messages from date (and optional time):
# journalctl --since="2012-10-30 18:17:16"
  • Show all messages since 20 minutes ago:
# journalctl --since "20 min ago"
  • Follow new messages:
# journalctl -f
  • Show all messages by a specific executable:
# journalctl /usr/lib/systemd/systemd
  • Show all messages by a specific process:
# journalctl _PID=1
  • Show all messages by a specific unit:
# journalctl -u netcfg
  • Show kernel ring buffer:
# journalctl -k
  • Show auth.log equivalent by filtering on syslog facility:
# journalctl -f -l SYSLOG_FACILITY=10

See man 1 journalctl, man 7 systemd.journal-fields, or Lennart's blog post for details.

Journal size limit

If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the respective file system. For example, with /var/log/journal located on a 50 GiB root partition this would lead to 5 GiB of journal data. The maximum size of the persistent journal can be controlled by uncommenting and changing the following:

/etc/systemd/journald.conf
SystemMaxUse=50M

Refer to man journald.conf for more info.

Clean journal files manually

The journal files are located under /var/log/journal, rm will do the work. Or, use journalctl,

Examples:

  • Remove archived journal files until the disk space they use falls below 100M:
# journalctl --vacuum-size=100M
  • Make all journal files contain no data older than 2 weeks.
# journalctl --vacuum-time=2weeks

Refer to man journalctl for more info.

Forward journald to /dev/tty12

Create a drop-in directory /etc/systemd/journald.conf.d and create a fw-tty12.conf file in it:

/etc/systemd/journald.conf.d/fw-tty12.conf
[Journal]
ForwardToConsole=yes
TTYPath=/dev/tty12
MaxLevelConsole=info

Then restart systemd-journald.

Specify a different journal to view

There may be a need to check the logs of another system that is dead in the water, like booting from a live system to recover a production system. In such case, one can mount the disk in e.g. /mnt, and specify the journal path via -D/--directory, like so:

$ journalctl -D /mnt/var/log/journal -xe

Troubleshooting

Short lived processes do not seem to log any output

If journalctl -u foounit does not show any output for a short lived service, look at the PID instead. For example, if systemd-modules-load.service fails, and systemctl status systemd-modules-load shows that it ran as PID 123, then you might be able to see output in the journal for that PID, i.e. journalctl -b _PID=123. Metadata fields for the journal such as _SYSTEMD_UNIT and _COMM are collected asynchronously and rely on the /proc directory for the process existing. Fixing this requires fixing the kernel to provide this data via a socket connection, similar to SCM_CREDENTIALS.

Disabling application crash dumps journaling

Edit the file /etc/systemd/coredump.conf by adding this line:

 Storage=none

and run:

# systemctl daemon-reload

to reload the configuration.

Boot time increasing over time

After using systemd-analyze a number of users have noticed that their boot time has increased significantly in comparison with what it used to be. After using systemd-analyze blame NetworkManager is being reported as taking an unusually large amount of time to start.

The problem for some users has been due to /var/log/journal becoming too large. This may have other impacts on performance, such as for systemctl status or journalctl. As such the solution is to remove every file within the folder (ideally making a backup of it somewhere, at least temporarily) and then setting a journal file size limit as described in Journal size limit.

systemd-tmpfiles-setup.service fails to start at boot

Starting with systemd 219, /usr/lib/tmpfiles.d/systemd.conf specifies ACL attributes for directories under /var/log/journal and, therefore, requires ACL support to be enabled for the filesystem the journal resides on.

systemctl enable fails for symlinks in /etc/systemd/system

If /etc/systemd/system/foo.service is a symlink and systemctl enable foo.service is run, it will fail with this error:

Failed to issue method call: No such file or directory

This is a design choice of systemd. As a workaround, enabling by absolute path works:

# systemctl enable /absolute/path/foo.service

See also