RED HAT ENTERPRISE LINUX

Preserve
System Journals

Preserve system journals

CIS126RH | RHEL System Administration 1
Mesa Community College

By default the systemd journal is stored in volatile memory and is lost at every reboot. For servers where log history must survive reboots — for troubleshooting crashes, compliance auditing, or security investigation — persistent journal storage must be explicitly configured. This module covers the storage modes available to systemd-journald, how to enable persistent storage, and how to control the disk space the journal consumes. These skills are tested on the RHCSA exam.

Learning Objectives

1. Explain the journal storage modes — Describe volatile, persistent, auto, and none storage modes and when each is appropriate
2. Configure persistent journal storage — Enable persistent storage by creating /var/log/journal/ or by setting Storage=persistent in /etc/systemd/journald.conf
3. Control journal size and retention — Configure SystemMaxUse, SystemKeepFree, and MaxRetentionSec to limit disk usage
4. Manage and maintain the journal — Use journalctl --vacuum-size and --vacuum-time to reclaim disk space from old journal data

How the Journal Stores Data

The systemd journal stores log entries as structured binary files. The location of these files determines whether they survive a reboot.

• Journal files are named *.journal and are organised by machine ID
• Each file contains a rotating ring of log entries with a hash chain for integrity
• The journal daemon (systemd-journald) manages reading and writing
• Multiple processes can write simultaneously — the journal is process-safe

The Four Storage Modes

The Storage= directive in /etc/systemd/journald.conf controls where and how the journal stores data.

The journald.conf Configuration File

All journal behaviour is controlled by /etc/systemd/journald.conf. The file ships with every option commented out — the comments show defaults.

# View the configuration file
$ cat /etc/systemd/journald.conf
# This file is part of systemd.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# To override individual settings, you may also create files
# in /etc/systemd/journald.conf.d/ with .conf suffix.
#
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month

Enabling Persistent Storage

There are two methods to enable persistent journal storage. Both work — choose the one appropriate for the situation.

Method 1 — Create the directory (uses Storage=auto default)

# Create the persistent journal directory
$ sudo mkdir -p /var/log/journal

# Apply correct ownership and permissions
$ sudo systemd-tmpfiles --create --prefix /var/log/journal

# Signal journald to re-evaluate storage
$ sudo systemctl restart systemd-journald

Method 2 — Set Storage=persistent in journald.conf

# Edit the configuration file
$ sudo vim /etc/systemd/journald.conf
# Change: #Storage=auto  to: Storage=persistent

# Restart journald to apply the change
$ sudo systemctl restart systemd-journald

# Confirm the journal is now writing to /var/log/journal/
$ ls /var/log/journal/
b3e7c...   # directory named after the machine ID

Verifying Persistent Storage

After enabling persistent storage, confirm the change took effect before rebooting.

# Confirm the directory exists and has the right structure
$ ls -la /var/log/journal/
total 8
drwxr-sr-x+ 3 root systemd-journal 4096 May 25 10:00 .
drwxr-xr-x. 1 root root            4096 May 25 10:00 ..
drwxr-sr-x. 2 root systemd-journal 4096 May 25 10:00 b3e7c2...

# Check current journal disk usage
$ journalctl --disk-usage
Archived and active journals take up 48.0M in the file system.

# Confirm journald is reading the config correctly
$ sudo journalctl --list-boots
 0 b3e7... Mon 2026-05-25 07:00 — present
# After a reboot, -1 should also appear here

# Check systemd-journald configuration in effect
$ sudo systemd-analyze cat-config systemd/journald.conf

Journal Size and Retention Controls

Without size limits, a persistent journal can grow to fill an entire disk. These settings in journald.conf prevent that.

Configuring Size Limits

Size values use standard unit suffixes. Time values use human-readable strings.

# Edit journald.conf to add size limits
$ sudo vim /etc/systemd/journald.conf

[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month

# Restart journald to apply
$ sudo systemctl restart systemd-journald

# Valid size suffixes: K, M, G, T (kilobytes, megabytes, gigabytes, terabytes)
# Valid time values: s, min, h, days, months, weeks, years
# Example: MaxRetentionSec=2weeks  SystemMaxUse=2G

# Using a drop-in file instead (preferred for maintainability)
$ sudo mkdir -p /etc/systemd/journald.conf.d
$ sudo tee /etc/systemd/journald.conf.d/size.conf <<EOF
[Journal]
SystemMaxUse=500M
MaxRetentionSec=1month
EOF
$ sudo systemctl restart systemd-journald

Vacuuming the Journal

journalctl --vacuum-* removes old journal files immediately to reclaim disk space — without waiting for automatic rotation.

# Check current journal disk usage first
$ journalctl --disk-usage
Archived and active journals take up 2.3G in the file system.

# Remove old entries until total usage is under 500M
$ sudo journalctl --vacuum-size=500M
Vacuuming done, freed 1.8G of disk space.

# Remove entries older than 2 weeks
$ sudo journalctl --vacuum-time=2weeks
Vacuuming done, freed 850M of disk space.

# Keep only the most recent N files
$ sudo journalctl --vacuum-files=5

# Verify the result
$ journalctl --disk-usage
Archived and active journals take up 450M in the file system.

Journal File Structure and Integrity

The journal's binary file format includes cryptographic integrity features that detect tampering or corruption.

# List the actual journal files on disk
$ ls -lh /var/log/journal/$(cat /etc/machine-id)/
-rw-r-----+ 1 root systemd-journal  96M May 25 10:00 system.journal
-rw-r-----+ 1 root systemd-journal  32M May 20 23:59 system@abc123.journal
# system.journal = current active file
# system@*.journal = archived rotated files

# Verify integrity of all journal files
$ sudo journalctl --verify
PASS: /var/log/journal/.../system.journal
PASS: /var/log/journal/.../system@abc123.journal

# Get the machine ID (directory name in /var/log/journal/)
$ cat /etc/machine-id
b3e7c2a1...

Forwarding to rsyslog and Remote Syslog

journald can forward entries to rsyslog (for text file logging) and to a remote syslog server for centralised log management.

Forward to rsyslog (default on RHEL)

# journald.conf — ForwardToSyslog enables rsyslog forwarding
[Journal]
Storage=persistent
ForwardToSyslog=yes    # default is yes on RHEL

Forward to syslog-compatible remote server

# /etc/rsyslog.conf — forward to a remote syslog server
*.* @192.168.1.10:514        # UDP (single @)
*.* @@192.168.1.10:514       # TCP (double @@, more reliable)

# Restart rsyslog after changes
$ sudo systemctl restart rsyslog

Recommended Configuration for a Server

A production server journal configuration balances retention history, disk usage, and performance.

# /etc/systemd/journald.conf — recommended server configuration
[Journal]
# Store journal persistently across reboots
Storage=persistent

# Limit total journal size to 1 GB
SystemMaxUse=1G

# Keep at least 500 MB free on the filesystem
SystemKeepFree=500M

# Rotate individual files at 100 MB
SystemMaxFileSize=100M

# Retain at most 30 days of history
MaxRetentionSec=1month

# Keep at most 10 archived files
SystemMaxFiles=10

# Forward to rsyslog for text file compatibility
ForwardToSyslog=yes

Volatile vs Persistent: Side by Side

Journal Preservation Quick Reference

Common Mistakes

Knowledge Check

Answer these before moving to the next slide.

1. By default, where does the systemd journal store its data, and why is this a problem for production servers?
2. Write the two steps required to enable persistent journal storage by editing journald.conf.
3. After configuring persistent storage, how do you confirm it is working after a reboot?
4. What is the difference between SystemMaxUse=500M in journald.conf and running journalctl --vacuum-size=500M?
5. What does MaxRetentionSec=2weeks do?
6. What are the four possible values for Storage= in journald.conf and what does each one do?

Knowledge Check — Answers

1. By default, the journal stores data in /run/log/journal/ — a tmpfs filesystem that exists in RAM. This is cleared at every reboot or power loss. For production servers, this means no log history to investigate crashes, unexpected reboots, or security events after the system restarts.
2. Step 1: Edit /etc/systemd/journald.conf and set Storage=persistent (uncomment or change the Storage line).
   Step 2: Restart the journal daemon: sudo systemctl restart systemd-journald
3. Reboot the system, then run journalctl --list-boots. If persistence is working, the output will show at least two entries — entry 0 for the current boot and entry -1 for the previous boot.
4. SystemMaxUse=500M in journald.conf sets an ongoing automatic limit — journald continuously enforces this ceiling and automatically removes old files when usage approaches 500 MB.
   journalctl --vacuum-size=500M is a one-time command that removes old archived files right now until usage drops below 500 MB. It does not set any future limit.
5. MaxRetentionSec=2weeks instructs journald to automatically delete journal entries older than two weeks. Old archived journal files are removed during rotation or vacuum operations when they contain only entries older than the specified retention period.
6. volatile — always store in /run/log/journal/ (RAM, lost on reboot).
   persistent — always store in /var/log/journal/ (disk, survives reboot).
   auto — use /var/log/journal/ if it exists, /run/log/journal/ otherwise. This is the default.
   none — discard all journal data immediately — nothing is stored.

Key Takeaways

1. The journal is volatile by default — it is lost at every reboot. The default storage is /run/log/journal/ in RAM. Production servers must have persistent storage configured to retain log history.
2. Enable persistence by setting Storage=persistent and restarting journald. Edit /etc/systemd/journald.conf, set Storage=persistent, then run sudo systemctl restart systemd-journald. Verify with a reboot and journalctl --list-boots.
3. Control disk usage with SystemMaxUse and MaxRetentionSec. SystemMaxUse=1G caps the total journal size. MaxRetentionSec=1month automatically removes old entries. Use journalctl --vacuum-size or --vacuum-time for immediate one-time cleanup.
4. Vacuum removes archived files only; active files are never deleted. journalctl --disk-usage shows current space consumption. journalctl --verify checks file integrity. The configuration file is /etc/systemd/journald.conf.

Graded Lab

• Run journalctl --list-boots to see the current boot sessions. Note whether only boot 0 is shown (volatile storage) or multiple boots (already persistent).
• Run journalctl --disk-usage to record the current journal disk footprint.
• Configure persistent journal storage: set Storage=persistent in /etc/systemd/journald.conf and restart systemd-journald.
• Confirm the /var/log/journal/ directory now contains journal files. Reboot the system. After reboot, run journalctl --list-boots to confirm both boot 0 and boot -1 are listed.
• Add size limits to the configuration: SystemMaxUse=500M and MaxRetentionSec=1month. Restart journald and verify the settings are active with systemd-analyze cat-config systemd/journald.conf.
• Run sudo journalctl --vacuum-size=100M to clean down to 100M and observe how much space is freed. Run journalctl --disk-usage before and after to compare.

Next: Start, stop, and check the status of network services