RED HAT ENTERPRISE LINUX

systemd Services

Control and Monitor System Services and Daemons

College-Level Course Module | RHEL System Administration

Learning Objectives

1
Understand systemd architecture

Units, services, daemons, and the systemd ecosystem

2
Control services with systemctl

Start, stop, restart, and reload services

3
Monitor service status

Check status, view logs, and troubleshoot failures

4
Configure boot behavior

Enable and disable services for automatic startup

5
Work with systemd targets

Understand boot targets and system states

What is systemd?

systemd is the system and service manager for Linux. It is the first process started at boot (PID 1) and is responsible for starting all other services and managing them throughout the system's operation.

systemd Responsibilities:
  • Boot the system (init process)
  • Start/stop services on demand
  • Track service dependencies
  • Manage system logging (journald)
  • Mount filesystems
  • Manage network, time, login
Key Components:
  • systemd - Main daemon (PID 1)
  • systemctl - Control command
  • journald - Logging system
  • journalctl - Log viewer
  • Unit files - Service definitions
History: systemd replaced SysVinit in RHEL 7. It provides parallel startup, on-demand activation, and unified configuration.

Services and Daemons

A daemon is a background process that runs continuously. A service in systemd context is a unit that manages a daemon or other background task.

# Common system services (daemons)
sshd.service      - OpenSSH server daemon
httpd.service     - Apache HTTP Server
mariadb.service   - MariaDB database server
crond.service     - Command scheduler
firewalld.service - Dynamic firewall daemon
NetworkManager.service - Network management
chronyd.service   - NTP time synchronization

# List running services
[root@host ~]# systemctl list-units --type=service --state=running
UNIT                      LOAD   ACTIVE SUB     DESCRIPTION
chronyd.service           loaded active running NTP client/server
crond.service             loaded active running Command Scheduler
firewalld.service         loaded active running firewalld
NetworkManager.service    loaded active running Network Manager
sshd.service              loaded active running OpenSSH server daemon

Understanding Units

Units are resources that systemd manages. Services are the most common, but systemd also manages mounts, timers, sockets, and more.

Unit TypeExtensionPurpose
Service.serviceManage daemons and processes
Target.targetGroup units, define system states
Socket.socketIPC/network socket for on-demand activation
Mount.mountFilesystem mount points
Timer.timerScheduled activation (like cron)
Path.pathFile/directory watching
Device.deviceHardware devices
# List all unit types
[root@host ~]# systemctl list-units --type=help

# List all loaded units
[root@host ~]# systemctl list-units

# List units of specific type
[root@host ~]# systemctl list-units --type=timer

systemctl Basics

systemctl is the primary command for interacting with systemd. It controls services, checks status, and configures boot behavior.

# Basic syntax
systemctl [command] [unit]

# Common service control commands
[root@host ~]# systemctl start httpd.service     # Start now
[root@host ~]# systemctl stop httpd.service      # Stop now
[root@host ~]# systemctl restart httpd.service   # Stop then start
[root@host ~]# systemctl reload httpd.service    # Reload config

# The .service extension is optional for services
[root@host ~]# systemctl start httpd
[root@host ~]# systemctl stop httpd

# Check if service is active
[user@host ~]$ systemctl is-active httpd
active

# Check if service is enabled (starts at boot)
[user@host ~]$ systemctl is-enabled httpd
enabled

Starting and Stopping

# Start a service
[root@host ~]# systemctl start httpd
# No output means success

# Stop a service
[root@host ~]# systemctl stop httpd

# Restart (stop + start)
[root@host ~]# systemctl restart httpd
# Brief interruption while service restarts

# Reload configuration (no restart)
[root@host ~]# systemctl reload httpd
# Service stays running, just rereads config

# Reload if supported, otherwise restart
[root@host ~]# systemctl reload-or-restart httpd

# Conditional restart (only if already running)
[root@host ~]# systemctl try-restart httpd
# Does nothing if service is stopped

# Kill all processes in service (forceful)
[root@host ~]# systemctl kill httpd
reload vs restart: reload keeps the service running (no downtime), but requires service support. restart guarantees config changes take effect.

Checking Status

systemctl status provides detailed information about a service including its state, recent log entries, and process information.

[root@host ~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-01-20 09:00:00 EST; 2h ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1234 (sshd)
      Tasks: 1 (limit: 23456)
     Memory: 5.2M
        CPU: 125ms
     CGroup: /system.slice/sshd.service
             └─1234 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Jan 20 09:00:00 host systemd[1]: Starting OpenSSH server daemon...
Jan 20 09:00:00 host sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 20 09:00:00 host systemd[1]: Started OpenSSH server daemon.
Key fields: Loaded (unit file status), Active (running state), Main PID, Memory usage, and recent journal entries.

Service States

StateDescription
active (running) Service is running normally
active (exited) One-shot service completed successfully
active (waiting) Running but waiting for an event
inactive (dead) Service is not running
failed Service attempted to start but failed
activating Service is starting up
deactivating Service is shutting down
# Quick state checks (useful for scripting)
[user@host ~]$ systemctl is-active httpd
active

[user@host ~]$ systemctl is-active stopped-service
inactive

[user@host ~]$ systemctl is-failed httpd
active        # "active" means not failed

# Exit codes: 0 = yes (active/enabled/failed), non-zero = no

Enabling and Disabling

Enable configures a service to start automatically at boot. Disable prevents automatic startup. Neither affects current running state.

# Enable service to start at boot
[root@host ~]# systemctl enable httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service 
  → /usr/lib/systemd/system/httpd.service

# Disable service from starting at boot
[root@host ~]# systemctl disable httpd
Removed /etc/systemd/system/multi-user.target.wants/httpd.service

# Enable does NOT start the service now!
# To enable AND start in one command:
[root@host ~]# systemctl enable --now httpd

# Disable AND stop in one command:
[root@host ~]# systemctl disable --now httpd

# Check enabled status
[user@host ~]$ systemctl is-enabled httpd
enabled
Important: enable/disable control boot behavior only. Use start/stop to control current state. Use --now to do both.

Masking Services

Masking completely disables a service - it cannot be started even manually. Use this to prevent a service from running under any circumstances.

# Mask a service (completely disable)
[root@host ~]# systemctl mask httpd
Created symlink /etc/systemd/system/httpd.service → /dev/null

# Try to start masked service - fails
[root@host ~]# systemctl start httpd
Failed to start httpd.service: Unit httpd.service is masked.

# Check status shows masked
[root@host ~]# systemctl status httpd
○ httpd.service
     Loaded: masked (Reason: Unit httpd.service is masked.)
     Active: inactive (dead)

# Unmask to allow starting again
[root@host ~]# systemctl unmask httpd
Removed /etc/systemd/system/httpd.service
Use case: Mask services that conflict with others, or services you want to ensure never run on this system.

Listing Services

# List all loaded units
[root@host ~]# systemctl list-units

# List only services
[root@host ~]# systemctl list-units --type=service

# List running services
[root@host ~]# systemctl list-units --type=service --state=running

# List failed services
[root@host ~]# systemctl list-units --type=service --state=failed
UNIT           LOAD   ACTIVE SUB    DESCRIPTION
httpd.service  loaded failed failed Apache HTTP Server

# List all installed unit files (not just loaded)
[root@host ~]# systemctl list-unit-files --type=service
UNIT FILE                  STATE    PRESET
httpd.service              disabled disabled
sshd.service               enabled  enabled
chronyd.service            enabled  enabled

# Show enabled services
[root@host ~]# systemctl list-unit-files --type=service --state=enabled

Viewing Service Logs

journalctl views logs from systemd's journal. Filter by unit to see logs for a specific service.

# View logs for a specific service
[root@host ~]# journalctl -u sshd
Jan 20 09:00:00 host systemd[1]: Starting OpenSSH server daemon...
Jan 20 09:00:00 host sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 20 09:00:01 host sshd[1234]: Server listening on :: port 22.
Jan 20 09:00:01 host systemd[1]: Started OpenSSH server daemon.

# Follow logs in real-time (like tail -f)
[root@host ~]# journalctl -u httpd -f

# Show logs since last boot
[root@host ~]# journalctl -u httpd -b

# Show recent logs (last 50 lines)
[root@host ~]# journalctl -u httpd -n 50

# Show logs from specific time period
[root@host ~]# journalctl -u httpd --since "2024-01-20 09:00" --until "2024-01-20 12:00"

# Show only errors and above
[root@host ~]# journalctl -u httpd -p err

Troubleshooting Failures

# Check for failed services
[root@host ~]# systemctl --failed
UNIT           LOAD   ACTIVE SUB    DESCRIPTION
httpd.service  loaded failed failed Apache HTTP Server

# Get detailed status
[root@host ~]# systemctl status httpd
× httpd.service - Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
     Active: failed (Result: exit-code) since Mon 2024-01-20 10:30:00
    Process: 5678 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
   Main PID: 5678 (code=exited, status=1/FAILURE)

Jan 20 10:30:00 host httpd[5678]: AH00558: httpd: Could not determine server's FQDN
Jan 20 10:30:00 host httpd[5678]: (98)Address already in use: AH00072: make_sock: could not bind

# View detailed logs
[root@host ~]# journalctl -xeu httpd
# -x adds explanatory messages, -e jumps to end

# Reset failed state after fixing
[root@host ~]# systemctl reset-failed httpd
Common causes: Configuration errors, port conflicts, missing dependencies, permission issues, resource exhaustion.

Understanding Targets

Targets are groups of units that represent system states. They replace the runlevels from SysVinit and define what services run at each stage of boot.

TargetOld RunlevelDescription
poweroff.target0System shutdown
rescue.target1Single-user rescue mode
multi-user.target3Multi-user, no GUI (servers)
graphical.target5Multi-user with GUI (workstations)
reboot.target6System reboot
emergency.target-Minimal emergency shell
# View current default target
[root@host ~]# systemctl get-default
multi-user.target

# Set default target
[root@host ~]# systemctl set-default graphical.target
Removed /etc/systemd/system/default.target
Created symlink /etc/systemd/system/default.target → /usr/lib/.../graphical.target

Changing System State

# Switch to different target immediately
[root@host ~]# systemctl isolate multi-user.target
# Stops graphical services, switches to text mode

[root@host ~]# systemctl isolate graphical.target
# Starts graphical services

# Rescue mode (single-user, for maintenance)
[root@host ~]# systemctl isolate rescue.target

# Emergency mode (minimal, for serious recovery)
[root@host ~]# systemctl isolate emergency.target

# Equivalent convenient commands
[root@host ~]# systemctl rescue      # Same as isolate rescue.target
[root@host ~]# systemctl emergency   # Same as isolate emergency.target

# Reboot and shutdown
[root@host ~]# systemctl reboot
[root@host ~]# systemctl poweroff
[root@host ~]# systemctl halt
Note: isolate switches to a target, stopping services not needed by the new target. This can disrupt users!

Service Dependencies

Services can depend on other services. systemd tracks these dependencies and starts services in the correct order.

# Show what a service requires
[root@host ~]# systemctl list-dependencies httpd
httpd.service
● ├─system.slice
● ├─httpd-init.service
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ...

# Show reverse dependencies (what requires this service)
[root@host ~]# systemctl list-dependencies httpd --reverse

# Show dependencies of a target
[root@host ~]# systemctl list-dependencies multi-user.target

# View unit file to see dependency configuration
[root@host ~]# systemctl cat httpd
[Unit]
Description=Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Wants=httpd-init.service
...

Unit Files

Unit files define how systemd manages a service. They contain configuration for startup, dependencies, and behavior.

# Unit file locations (in priority order)
/etc/systemd/system/    - Local administrator configuration (highest priority)
/run/systemd/system/    - Runtime generated units
/usr/lib/systemd/system/ - Package-installed units (vendor)

# View a unit file
[root@host ~]# systemctl cat sshd.service
[Unit]
Description=OpenSSH server daemon
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target
Never edit files in /usr/lib/systemd/system/ - they are overwritten on updates. Copy to /etc/systemd/system/ first.

Modifying Unit Files

# Edit unit file (creates drop-in override automatically)
[root@host ~]# systemctl edit httpd
# Opens editor for /etc/systemd/system/httpd.service.d/override.conf

# Example: Add environment variable
[Service]
Environment="OPTIONS=-DFOREGROUND"

# Edit full unit file (copy original to /etc/systemd/system/)
[root@host ~]# systemctl edit --full httpd

# After editing, reload systemd configuration
[root@host ~]# systemctl daemon-reload
# Then restart the service for changes to take effect
[root@host ~]# systemctl restart httpd

# View all files affecting a unit
[root@host ~]# systemctl cat httpd
# Shows base file plus any drop-in overrides

# Revert to vendor unit file (remove local changes)
[root@host ~]# systemctl revert httpd
Always run: systemctl daemon-reload after modifying unit files, then restart the service.

Best Practices

Do

  • Check status before and after changes
  • Use enable --now to enable and start
  • Use systemctl edit for customizations
  • Run daemon-reload after unit changes
  • Check logs when services fail
  • Test configuration before restart
  • Document service customizations
  • Understand dependencies before stopping

Do Not

  • Edit files in /usr/lib/systemd/system/
  • Forget daemon-reload after changes
  • Ignore failed services
  • Stop critical services without planning
  • Assume enable starts the service
  • Skip status checks in scripts
  • Mask services without documentation
  • Restart when reload would work
Troubleshooting workflow: status → logs (journalctl) → dependencies → unit file → fix → daemon-reload → restart → status

Key Takeaways

1

systemd: Init system managing services, mounts, targets. PID 1 on modern Linux. Controls the entire system lifecycle.

2

Control: systemctl start|stop|restart|reload controls running state. enable|disable controls boot behavior.

3

Monitor: systemctl status shows state and logs. journalctl -u shows full service logs.

4

Targets: System states like multi-user.target, graphical.target. set-default configures boot target.

LAB EXERCISES

  • Install and start a web server service (httpd)
  • Enable the service to start at boot
  • Check service status and view recent logs
  • Stop, restart, and reload the service
  • Intentionally break the config, observe failure, fix it
  • Create a drop-in override to customize the service
  • List all enabled services on the system

Next: Introduction to Networking