RED HAT ENTERPRISE LINUX

systemd Services

Control and Monitor System Services and Daemons

CIS126RH | RHEL System Administration 1
Mesa Community College

Learning Objectives

Understand systemd architecture

Units, services, daemons, and the systemd ecosystem

Control services with systemctl

Start, stop, restart, reload, enable, and disable services

Monitor service status and logs

Use systemctl status and journalctl to diagnose services

Manage targets and unit files

Understand boot targets and customize service configuration

What is systemd?

systemd is the system and service manager for Linux. It is the first process started at boot (PID 1) and is responsible for starting all other services and managing them throughout the system lifecycle.

systemd Responsibilities:

Boot the system (init process)
Start and stop services on demand
Track service dependencies
Manage system logging (journald)
Mount filesystems
Manage network, time, login sessions

Key Tools:

systemctl — control services and system state
journalctl — view system and service logs
systemd-analyze — analyze boot performance
loginctl — manage user sessions

Services and Daemons

A daemon is a background process that runs continuously, waiting for requests. In systemd, daemons are managed as service units (files ending in .service).

# Common system services
sshd.service          - OpenSSH server daemon
httpd.service         - Apache HTTP Server
mariadb.service       - MariaDB database server
crond.service         - Command scheduler (cron jobs)
firewalld.service     - Dynamic firewall daemon
NetworkManager.service - Network management
chronyd.service       - NTP time synchronization

Naming convention: The .service suffix can be omitted with systemctl. systemctl start sshd and systemctl start sshd.service are equivalent.

Listing Running Services

# List running service units
[root@host ~]# systemctl list-units --type=service --state=running
UNIT                      LOAD   ACTIVE SUB     DESCRIPTION
chronyd.service           loaded active running NTP client/server
crond.service             loaded active running Command Scheduler
firewalld.service         loaded active running firewalld
NetworkManager.service    loaded active running Network Manager
sshd.service              loaded active running OpenSSH server daemon

Column Meanings

`LOAD`	Unit file found and parsed
`ACTIVE`	High-level state (active/inactive/failed)
`SUB`	Low-level state (running/exited/dead)

Useful Filters

`--state=running`	Active services
`--state=failed`	Failed services
`--type=timer`	Scheduled timer units

Understanding Units

Unit Type	Extension	Purpose
Service	`.service`	System services and daemons
Target	`.target`	Group of units (like runlevels)
Socket	`.socket`	Network or IPC socket activation
Path	`.path`	Monitor filesystem paths
Timer	`.timer`	Scheduled activation (like cron)
Mount	`.mount`	Filesystem mount points
Device	`.device`	Hardware device units

# List units of a specific type
[root@host ~]# systemctl list-units --type=service
[root@host ~]# systemctl list-units --type=timer
[root@host ~]# systemctl list-units --type=help    # All types

systemctl Basics

# Basic syntax: systemctl [command] [unit]

# Service control — .service extension is optional
[root@host ~]# systemctl start   httpd    # Start now
[root@host ~]# systemctl stop    httpd    # Stop now
[root@host ~]# systemctl restart httpd    # Stop + start
[root@host ~]# systemctl reload  httpd    # Reload config (no restart)

# Status checks (any user can run these)
[user@host ~]$ systemctl is-active  httpd
active
[user@host ~]$ systemctl is-enabled httpd
enabled
[user@host ~]$ systemctl is-failed  httpd
active   # "active" means NOT failed

active vs enabled: active = running right now. enabled = will start at next boot. These are independent — a service can be active but not enabled, or enabled but not currently running.

Starting, Stopping, Restarting

# Start a service (no output = success)
[root@host ~]# systemctl start httpd

# Stop a service
[root@host ~]# systemctl stop httpd

# Restart (stop + start — brief interruption)
[root@host ~]# systemctl restart httpd

# Reload configuration only (service keeps running)
[root@host ~]# systemctl reload httpd

# Reload if supported, otherwise restart
[root@host ~]# systemctl reload-or-restart httpd

reload vs restart: reload sends a signal to re-read configuration without stopping. restart stops and starts — causes a brief outage. Always prefer reload when supported (check with systemctl cat for ExecReload=).

Conditional and Forceful Operations

# try-restart: restart only if already running
[root@host ~]# systemctl try-restart httpd
# Does nothing if service is stopped — safe in scripts

# try-reload-or-restart: reload if possible, restart if needed
[root@host ~]# systemctl try-reload-or-restart httpd

# kill: send signal to all processes in a service
[root@host ~]# systemctl kill httpd
# Default sends SIGTERM — use --signal= for other signals
[root@host ~]# systemctl kill --signal=SIGKILL httpd

kill vs stop: stop sends SIGTERM and waits for orderly shutdown. kill sends a signal directly to the processes — use it when stop hangs. Use --signal=SIGKILL only as a last resort.

Checking Status

[root@host ~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-01-20 09:00:00 EST; 2h ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1234 (sshd)
      Tasks: 1 (limit: 23456)
     Memory: 5.2M
        CPU: 125ms
     CGroup: /system.slice/sshd.service
             └─1234 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Jan 20 09:00:00 host systemd[1]: Starting OpenSSH server daemon...
Jan 20 09:00:00 host sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 20 09:00:00 host systemd[1]: Started OpenSSH server daemon.

Service States

Active State	Sub-state	Meaning
`active`	`running`	Service is running normally
`active`	`exited`	One-shot service completed successfully
`inactive`	`dead`	Service is stopped (not an error)
`failed`	`failed`	Service exited with error or crashed
`activating`	`start`	Service is in the process of starting
`deactivating`	`stop`	Service is in the process of stopping

# Quick state checks — useful for scripting
[user@host ~]$ systemctl is-active httpd      # active or inactive
[user@host ~]$ systemctl is-failed httpd      # active (not failed) or failed
# Exit codes: 0 = condition is true, non-zero = false

Enabling and Disabling Services

# Enable service to start at boot
[root@host ~]# systemctl enable httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service
  → /usr/lib/systemd/system/httpd.service

# Disable service from starting at boot
[root@host ~]# systemctl disable httpd
Removed /etc/systemd/system/multi-user.target.wants/httpd.service

# Enable AND start immediately (most common pattern)
[root@host ~]# systemctl enable --now httpd

# Disable AND stop immediately
[root@host ~]# systemctl disable --now httpd

# Check boot enablement
[user@host ~]$ systemctl is-enabled httpd
enabled

Important: enable does NOT start the service immediately. Use enable --now to enable and start in one step.

Masking Services

# Mask a service — links unit file to /dev/null
[root@host ~]# systemctl mask httpd
Created symlink /etc/systemd/system/httpd.service → /dev/null

# Masked service CANNOT be started — not even manually
[root@host ~]# systemctl start httpd
Failed to start httpd.service: Unit httpd.service is masked.

# Status shows masked state
[root@host ~]# systemctl status httpd
○ httpd.service
     Loaded: masked (Reason: Unit httpd.service is masked.)
     Active: inactive (dead)

# Unmask to allow starting again
[root@host ~]# systemctl unmask httpd
Removed /etc/systemd/system/httpd.service

disable vs mask: disable prevents auto-start at boot but allows manual start. mask completely prevents starting by any means — even manually or as a dependency.

Listing Services

# List all loaded service units
[root@host ~]# systemctl list-units --type=service

# List failed services
[root@host ~]# systemctl --failed
UNIT           LOAD   ACTIVE SUB    DESCRIPTION
httpd.service  loaded failed failed Apache HTTP Server

# List ALL installed unit files (including not-loaded)
[root@host ~]# systemctl list-unit-files --type=service
UNIT FILE                  STATE    PRESET
httpd.service              disabled disabled
sshd.service               enabled  enabled
chronyd.service            enabled  enabled

# Show only enabled services
[root@host ~]# systemctl list-unit-files --type=service --state=enabled

list-units vs list-unit-files: list-units shows what is loaded in memory. list-unit-files shows all installed unit files and their enabled/disabled state — even services not currently loaded.

Viewing Service Logs

# View all logs for a service
[root@host ~]# journalctl -u sshd
Jan 20 09:00:00 host systemd[1]: Starting OpenSSH server daemon...
Jan 20 09:00:00 host sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 20 09:00:01 host sshd[1234]: Server listening on :: port 22.
Jan 20 09:00:01 host systemd[1]: Started OpenSSH server daemon.

# Follow logs in real time (like tail -f)
[root@host ~]# journalctl -u httpd -f

# Logs since last boot
[root@host ~]# journalctl -u httpd -b

# Show last 50 lines
[root@host ~]# journalctl -u httpd -n 50

# Show only errors and above
[root@host ~]# journalctl -u httpd -p err

# Logs from a time range
[root@host ~]# journalctl -u httpd --since "2024-01-20 09:00"

Troubleshooting Failures

# Step 1: Find failed services
[root@host ~]# systemctl --failed
UNIT           LOAD   ACTIVE SUB    DESCRIPTION
httpd.service  loaded failed failed Apache HTTP Server

# Step 2: Check detailed status
[root@host ~]# systemctl status httpd
× httpd.service - Apache HTTP Server
     Active: failed (Result: exit-code) since Mon 2024-01-20 10:30:00
    Process: 5678 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
Jan 20 10:30:00 host httpd[5678]: (98)Address already in use: AH00072: make_sock: failed to listen on [::]:80

# Step 3: View full logs with explanations (-xeu)
[root@host ~]# journalctl -xeu httpd
# -x adds explanatory text, -e jumps to end, -u filters by unit

# Step 4: After fixing, reset failed state and restart
[root@host ~]# systemctl reset-failed httpd
[root@host ~]# systemctl start httpd

Understanding Targets

Targets are groups of units that define a system state. They replace the runlevels concept from SysV init.

Target	Old Runlevel	Description
`poweroff.target`	0	Shut down
`rescue.target`	1	Single user, minimal services
`multi-user.target`	3	Multi-user text mode (servers)
`graphical.target`	5	Multi-user with GUI
`reboot.target`	6	Reboot
`emergency.target`	—	Minimal recovery (read-only)

# View and set the default target
[root@host ~]# systemctl get-default
multi-user.target
[root@host ~]# systemctl set-default graphical.target

Changing System State

# Switch to a target immediately (isolate stops unneeded units)
[root@host ~]# systemctl isolate multi-user.target
# Stops graphical services, switches to text mode

[root@host ~]# systemctl isolate graphical.target
# Starts display manager and GUI services

# Rescue and emergency modes
[root@host ~]# systemctl isolate rescue.target
[root@host ~]# systemctl rescue        # Shorthand

[root@host ~]# systemctl isolate emergency.target
[root@host ~]# systemctl emergency     # Shorthand

# Shutdown and reboot
[root@host ~]# systemctl reboot
[root@host ~]# systemctl poweroff
[root@host ~]# systemctl halt          # Stop but don't power off

isolate note: Not all targets are isolatable. Only targets with AllowIsolate=yes in their unit file can be used with isolate.

Service Dependencies

# Show what a service requires to start
[root@host ~]# systemctl list-dependencies httpd
httpd.service
● ├─system.slice
● ├─httpd-init.service
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   └─...

# Show what depends on this service (reverse)
[root@host ~]# systemctl list-dependencies httpd --reverse

# View the unit file to understand dependency directives
[root@host ~]# systemctl cat httpd
[Unit]
Description=Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Wants=httpd-init.service

Key dependency directives:
Requires= — must start; if it fails, this unit fails
Wants= — should start; failure is tolerated
After= — ordering only; start after listed units

Unit Files

# Unit file locations (highest priority first)
/etc/systemd/system/     - Admin overrides (highest priority)
/run/systemd/system/     - Runtime generated units
/usr/lib/systemd/system/ - Package-installed units (vendor default)

# View a unit file in full
[root@host ~]# systemctl cat sshd.service
[Unit]
Description=OpenSSH server daemon
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

Modifying Unit Files

# Create a drop-in override (PREFERRED — preserves original)
[root@host ~]# systemctl edit httpd
# Opens editor for /etc/systemd/system/httpd.service.d/override.conf
# Example: add environment variable
[Service]
Environment="OPTIONS=-DFOREGROUND"

# Edit the full unit file (copy to /etc/systemd/system/)
[root@host ~]# systemctl edit --full httpd

# REQUIRED after any unit file change
[root@host ~]# systemctl daemon-reload
[root@host ~]# systemctl restart httpd

# View all files affecting a unit (base + overrides)
[root@host ~]# systemctl cat httpd

# Revert all local changes to vendor default
[root@host ~]# systemctl revert httpd

Best practice: Always use systemctl edit (drop-in) rather than editing vendor unit files directly. Overrides survive package updates; direct edits to /usr/lib/systemd/system/ are overwritten.

Best Practices

Do

Check systemctl status before and after changes
Use enable --now to enable and start at once
Use systemctl edit for drop-in overrides
Run daemon-reload after unit file changes
Check journalctl -xeu unit when services fail
Test configuration syntax before restarting
Understand dependencies before stopping services

Do Not

Edit unit files in /usr/lib/systemd/system/
Forget daemon-reload after unit changes
Use systemctl kill --signal=SIGKILL as first approach
Mask services without understanding dependencies
Stop critical services (sshd, NetworkManager) remotely without a backup plan

Key Takeaways

systemd: Init system managing services, mounts, targets. PID 1 on modern RHEL. Controls the entire system lifecycle.

Control: systemctl start|stop|restart|reload for runtime state. enable|disable for boot behavior. These are independent.

Monitor: systemctl status shows state and recent logs. journalctl -xeu unit gives full logs with hints.

Customize: systemctl edit creates safe drop-in overrides. Always run daemon-reload after unit changes.

Graded Lab

Install and start a web server service (httpd)
Enable the service to start at boot with enable --now
Check service status and view recent logs with journalctl
Stop, restart, and reload the service; observe behavior differences
Intentionally break the configuration, observe the failure, and fix it
Create a drop-in override with systemctl edit
List all enabled services on the system

Next: Introduction to Networking