RED HAT ENTERPRISE LINUX

Image-Based Management

Building, Deploying, and Upgrading with bootc

CIS238RH | RHEL System Administration
Mesa Community College

Learning Objectives

Understand image-based deployment concepts

Bootable containers, OSTree, atomic updates, and immutable infrastructure

Create bootable container images

Build custom OS images using Containerfiles and podman

Deploy image-based systems

Install servers from bootable container images

Manage upgrades and rollbacks

Perform atomic system updates with easy rollback capability

Traditional vs Image-Based

Traditional (Package-Based)

Install base OS
Add packages with dnf
Configure individually
Update packages incrementally
Each system potentially unique
Rollback is complex

Image-Based (bootc)

Build complete OS image
Deploy identical copies
Configuration in image
Update entire image atomically
All systems identical
Instant rollback available

Image-Based Workflow

Build Image

→

Test Image

→

Push to Registry

→

Deploy to Servers

Key insight: Image-based treats the OS like a container — build once, deploy many, update atomically, roll back instantly.

Updates mean building a new image and deploying it atomically — the entire system switches to the new version at once. If problems occur, you roll back to the previous image instantly.

What is bootc?

bootc (bootable containers) is technology that enables deploying and updating Linux systems using OCI container images. Build your OS as a container, deploy it as a server.

Key Components

OCI container images (standard format)
OSTree for atomic filesystem updates
bootc CLI for system management
Standard container tools (podman, buildah)

Benefits

Reproducible deployments
Atomic updates
Instant rollbacks
Familiar container workflow
Infrastructure as code

`bootc status` Overview

# bootc manages bootable container images on a running system
[root@server ~]# bootc status
Current booted image: registry.redhat.io/rhel9/rhel-bootc:9.3
    Image version: 9.3.0 (2024-01-15)
    Image digest: sha256:a1b2c3d4...

Staged for next boot: none

Rollback available: registry.redhat.io/rhel9/rhel-bootc:9.2
    Image version: 9.2.0 (2023-10-01)

# Key components:
# - OCI container images (standard format)
# - OSTree for atomic filesystem updates
# - bootc CLI for system management
# - Standard container tools (podman, buildah)

bootc status shows what image is currently running, what is staged for the next boot, and what rollback options are available — giving full visibility into every system's state.

OSTree Foundation

OSTree is a system for managing bootable filesystem trees. It enables atomic updates by deploying complete filesystem snapshots rather than updating files in place.

Current Boot: rhel-bootc:9.3

Staged: rhel-bootc:9.4 (pending reboot)

Rollback: rhel-bootc:9.2

OSTree Repository (stores all versions)

Viewing OSTree Deployments

# OSTree stores filesystem trees efficiently:
# - Content-addressed storage (deduplication)
# - Atomic switching between versions
# - Boot loader integration for rollback

# View OSTree deployments
[root@server ~]# rpm-ostree status
State: idle
Deployments:
● ostree-image-signed:docker://registry.redhat.io/rhel9/rhel-bootc:9.3
               Digest: sha256:a1b2c3d4...
  ostree-image-signed:docker://registry.redhat.io/rhel9/rhel-bootc:9.2
               Digest: sha256:e5f6g7h8...

The ● bullet marks the currently booted deployment. You typically have two or three deployments available: current, staged (if an update is pending), and previous (for rollback).

RHEL Bootc Base Images

Red Hat provides base bootc images that serve as foundations for custom images. These include the kernel, systemd, and core OS components needed to boot.

Image	Description	Use Case
rhel-bootc	Full RHEL bootable base image	General purpose servers
rhel-bootc-minimal	Minimal bootable image	Appliances, single-purpose systems

Important: Always start FROM a bootc base image. Regular container images (like UBI) lack the kernel and boot components needed for a bootable system.

Pulling Bootc Base Images

# Pull RHEL bootc base image
[user@workstation ~]$ podman pull registry.redhat.io/rhel9/rhel-bootc:9.3

# Inspect base image
[user@workstation ~]$ podman inspect registry.redhat.io/rhel9/rhel-bootc:9.3

# Base images include:
# - Linux kernel and firmware
# - systemd and core services
# - SELinux policies
# - Basic system utilities
# - bootc and rpm-ostree tools

Pull bootc base images from registry.redhat.io (requires Red Hat credentials). These images are updated regularly with security fixes and should be the foundation for all your custom images.

Creating Custom Images

# Containerfile for custom web server bootc image
FROM registry.redhat.io/rhel9/rhel-bootc:9.3

# Install packages
RUN dnf install -y httpd mod_ssl php php-mysqlnd && \
    dnf clean all && \
    systemctl enable httpd

# Add configuration files
COPY httpd.conf /etc/httpd/conf/httpd.conf
COPY ssl.conf /etc/httpd/conf.d/ssl.conf

# Add application files
COPY webapp/ /var/www/html/

# Set SELinux file contexts (important for bootc)
RUN setfiles -F /etc/selinux/targeted/contexts/files/file_contexts /var/www/html

# Labels for identification
LABEL description="Custom RHEL web server"
LABEL version="1.0"

Containerfile Key Points

bootc-Specific Requirements

Start FROM a bootc base image
Use systemctl enable to start services at boot
Run setfiles after adding files to set SELinux contexts
Always run dnf clean all to reduce size

Standard Containerfile Features

FROM — specify base image
RUN — execute commands
COPY — add files and configs
LABEL — add version metadata

SELinux: Without setfiles, SELinux may block access to your added files at runtime. Always set correct contexts for custom files.

Building Images

# Build the bootc image
[user@workstation ~]$ podman build -t mywebserver:1.0 -f Containerfile .
STEP 1/8: FROM registry.redhat.io/rhel9/rhel-bootc:9.3
STEP 2/8: RUN dnf install -y httpd mod_ssl php...
...
STEP 8/8: LABEL version="1.0"
Successfully tagged localhost/mywebserver:1.0

# Verify the image
[user@workstation ~]$ podman images
REPOSITORY                           TAG    IMAGE ID      SIZE
localhost/mywebserver                1.0    a1b2c3d4e5f6  2.1 GB
registry.redhat.io/rhel9/rhel-bootc  9.3    f6e5d4c3b2a1  1.8 GB

# Test the image (limited - cannot fully boot in container)
[user@workstation ~]$ podman run --rm -it mywebserver:1.0 rpm -q httpd
httpd-2.4.57-5.el9.x86_64

Bootc images are larger than typical application containers (several GB) because they include the full OS. Testing in a container is limited — full validation requires deploying to a VM or staging system.

Tagging & Pushing Images

# Tag for registry
[user@workstation ~]$ podman tag mywebserver:1.0 registry.example.com/mywebserver:1.0

# Push to registry
[user@workstation ~]$ podman push registry.example.com/mywebserver:1.0

Use semantic versioning tags (1.0, 1.1, 2.0) rather than just :latest for production deployments — so you always know exactly what version each system is running and can target specific rollbacks.

Registry Workflow

Build Image

→

Test

→

Push to Registry

→

Deploy / Update

Registry Options

Red Hat Quay (quay.io or self-hosted)
Docker Hub (docker.io)
Private registries (Harbor, Artifactory)
registry.redhat.io (Red Hat images)

Enterprise Tip

Run a private registry for control over image distribution and to avoid external dependencies in production environments.

Registry Commands

# Login to registry
[user@workstation ~]$ podman login registry.example.com
Username: admin
Password: 
Login Succeeded!

# Push image to registry
[user@workstation ~]$ podman push registry.example.com/rhel-webserver:1.0

# Systems pull updates from registry
[root@server ~]# bootc upgrade
Pulling manifest for registry.example.com/rhel-webserver:latest
Importing ostree commit...
Staging deployment...
Queued for next boot: registry.example.com/rhel-webserver:1.1

Systems pull images from the registry using bootc upgrade. The registry URL is stored in system configuration, so running the command automatically checks for newer versions of the current image.

Installing from Images

Deploy bootc images to new systems using bootc-image-builder to create installable media, or use bootc install on existing systems.

# Method 1: Create installation ISO with bootc-image-builder
[user@workstation ~]$ sudo podman run --rm -it --privileged \
    --pull=newer \
    -v ./output:/output \
    registry.redhat.io/rhel9/bootc-image-builder:latest \
    --type iso \
    registry.example.com/mywebserver:1.0
Building ISO image...
Output: /output/bootiso/install.iso

# Boot the ISO on target hardware/VM and install

bootc-image-builder can produce ISOs for bare-metal installation, disk images for cloud deployment (QCOW2, VMDK, AMI), or other formats from a single container image source.

Converting Existing Systems

# Method 2: Convert existing system to bootc
[root@existing ~]# podman run --rm --privileged \
    -v /:/target \
    -v /var/lib/containers:/var/lib/containers \
    registry.example.com/mywebserver:1.0 \
    bootc install to-existing-root /target

# Reboot into new image-based system
[root@existing ~]# systemctl reboot

bootc install to-existing-root transforms a traditional RHEL system to image-based management in place — useful for migrating existing infrastructure without reinstalling from scratch.

`bootc status` Detail

# Check current system status
[root@server ~]# bootc status
Current booted image: registry.example.com/rhel-webserver:1.0
    Image version: 1.0.0
    Image digest: sha256:a1b2c3d4e5f6...
    Timestamp: 2024-01-15T10:30:00Z

Staged for next boot: none

Rollback available: registry.example.com/rhel-webserver:0.9
    Image version: 0.9.0
    Image digest: sha256:e5f6g7h8i9j0...
    Timestamp: 2024-01-01T08:00:00Z

# Show detailed image information
[root@server ~]# bootc status --format=json | jq .

`bootc status` for Scripting

# Verify system is running expected image
[root@server ~]# bootc status --booted
registry.example.com/rhel-webserver:1.0

# Check if updates are available (without applying)
[root@server ~]# bootc upgrade --check
Update available: registry.example.com/rhel-webserver:1.1

The --booted flag returns just the current image reference — ideal for scripts verifying that servers are running expected versions. bootc upgrade --check queries the registry without staging anything, useful for monitoring pipelines.

Upgrading Systems

Atomic updates: bootc downloads the new image, stages it, and activates on reboot. The running system is unchanged until reboot — no partial updates.

# Check for and apply updates
[root@server ~]# bootc upgrade
Pulling manifest for registry.example.com/rhel-webserver:latest
Layers already present: 45/52
Pulling new layers: 7
Importing to ostree...
Staging deployment...
Upgrade staged. Reboot to apply.

# Reboot to activate new image
[root@server ~]# systemctl reboot

After Upgrade: Verify & Switch

# After reboot, verify new version
[root@server ~]# bootc status
Current booted image: registry.example.com/rhel-webserver:1.1
Rollback available: registry.example.com/rhel-webserver:1.0

# Switch to specific image (not just latest)
[root@server ~]# bootc switch registry.example.com/rhel-webserver:2.0

Safe updates: The running system continues operating normally while the update is staged. Only reboot activates the change. If you need a specific version rather than latest, use bootc switch with the full image reference.

Rolling Back

Instant rollback: If an upgrade causes problems, roll back to the previous working image immediately. No complex recovery procedures needed.

# Scenario: Upgrade caused problems
[root@server ~]# bootc status
Current booted image: registry.example.com/rhel-webserver:1.1
Rollback available: registry.example.com/rhel-webserver:1.0

# Rollback to previous image
[root@server ~]# bootc rollback
Rollback staged. Reboot to apply.

# Reboot to activate rollback
[root@server ~]# systemctl reboot

After Rollback

# Alternative: Select previous deployment at boot menu
# GRUB shows multiple deployments - select older one

# After rollback
[root@server ~]# bootc status
Current booted image: registry.example.com/rhel-webserver:1.0
Rollback available: registry.example.com/rhel-webserver:1.1

Recovery time: Rollback is just a reboot — no reinstallation, no package downgrading, no complex recovery. System returns to a known-good state instantly.

After rollback, the newer (problematic) version is still available as the rollback option. You can also select previous deployments from the GRUB boot menu — useful if the new image fails to boot at all.

Managing Local Changes

Image-based systems are intended to be immutable, but some local state must persist. These live outside the image in designated locations.

Persistent (survives upgrades)

/etc — Configuration (merged with image)
/var — Variable data, logs, databases
/home — User home directories

Immutable (from image)

/usr — System binaries and libraries
/lib — Libraries (symlink to /usr/lib)
/bin — Binaries (symlink to /usr/bin)

Local Changes: Best Practice

# Configuration changes in /etc persist across upgrades
[root@server ~]# vi /etc/myapp/config.yaml
# This change survives upgrades

# To add packages locally (temporary, testing only):
[root@server ~]# rpm-ostree install vim-enhanced
Staging deployment... Reboot to apply.

# Better approach: Add packages to the image
# Edit Containerfile, rebuild, upgrade

Avoid drift: rpm-ostree install adds packages locally but creates drift from the image and complicates updates. The correct approach is adding packages to the Containerfile, rebuilding, and upgrading.

Containerfile Best Practices

FROM registry.redhat.io/rhel9/rhel-bootc:9.3

# Combine RUN commands to reduce layers
RUN dnf install -y \
        httpd \
        mod_ssl \
        php \
        php-mysqlnd \
    && dnf clean all \
    && systemctl enable httpd

# Copy configs before application files (better caching)
COPY etc/ /etc/

# Set correct SELinux contexts
RUN setfiles -F /etc/selinux/targeted/contexts/files/file_contexts /etc

# Application files last (most likely to change)
COPY webapp/ /var/www/html/
RUN setfiles -F /etc/selinux/targeted/contexts/files/file_contexts /var/www/html

Image Labels & Caching

# Always include OCI metadata labels
LABEL org.opencontainers.image.version="1.0"
LABEL org.opencontainers.image.description="Production web server"
LABEL org.opencontainers.image.created="2024-01-15"

Caching tip: Put things that change frequently (app code) last in the Containerfile. Earlier layers are cached, making rebuilds faster. Put stable things like package installation first.

Consider using build arguments (ARG) for version numbers so they can be injected from CI pipelines — e.g., ARG VERSION → LABEL version="${VERSION}".

Multi-Stage Builds

# Stage 1: Build application
FROM registry.redhat.io/ubi9/go-toolset:latest AS builder
WORKDIR /app
COPY src/ .
RUN go build -o /app/myservice

# Stage 2: Create bootable image
FROM registry.redhat.io/rhel9/rhel-bootc:9.3

# Install runtime dependencies only (not build tools)
RUN dnf install -y minimal-runtime-deps && dnf clean all

# Copy built application from builder stage
COPY --from=builder /app/myservice /usr/local/bin/

# Add systemd service file
COPY myservice.service /etc/systemd/system/
RUN systemctl enable myservice

LABEL version="1.0"

Multi-Stage Build Benefits

Smaller images: Build tools are only in the builder stage, not the final image. The bootable image contains only what is needed to run — no compiler, no source code, no build artifacts.

This pattern works for any compiled language: Go, Rust, C/C++. For interpreted languages like Python, use a build stage to install dependencies and copy only the installed packages to the final image.

Smaller images mean: faster pulls, less disk usage per deployment, and a reduced attack surface in production — all critical benefits for OS-level images.

CI/CD Integration

Integrate bootc image building into CI/CD pipelines for automated, tested deployments. Build on commit, test automatically, deploy to staging, then production.

# Example CI pipeline (GitLab CI / GitHub Actions concept)
stages:
  - build
  - test
  - push
  - deploy-staging
  - deploy-production

build-image:
  script:
    - podman build -t myserver:${CI_COMMIT_SHA} .

test-image:
  script:
    - podman run myserver:${CI_COMMIT_SHA} /tests/run-tests.sh

CI/CD: Push & Deploy

push-image:
  script:
    - podman push registry.example.com/myserver:${CI_COMMIT_SHA}
    - podman push registry.example.com/myserver:latest

deploy-staging:
  script:
    - ssh staging "bootc upgrade && systemctl reboot"

deploy-production:
  script:
    - ssh prod-servers "bootc upgrade && systemctl reboot"
  when: manual

Traceability: Using the commit SHA as an image tag means every deployed image can be traced back to the exact source code that built it. Manual approval gates protect production.

Troubleshooting bootc

# Check bootc service status
[root@server ~]# systemctl status bootc-fetch-apply-updates.timer

# View bootc logs
[root@server ~]# journalctl -u bootc-fetch-apply-updates

# Check OSTree deployments
[root@server ~]# rpm-ostree status
[root@server ~]# ostree admin status

# Verify image signature (if using signed images)
[root@server ~]# bootc status --format=json | jq '.booted.image.signature'

# Debug upgrade issues
[root@server ~]# bootc upgrade --verbose 2>&1 | tee upgrade.log

Common Issues

# Registry authentication
[user@workstation ~]$ podman login registry.example.com

# Network connectivity check
[user@workstation ~]$ curl registry.example.com/v2/

# SELinux denials
[root@server ~]# ausearch -m avc -ts recent

# Disk space
[root@server ~]# df -h /var/lib/containers

# Emergency: Boot previous deployment from GRUB menu

Emergency escape: If a system will not boot after upgrade, select the previous deployment from the GRUB menu. The previous deployment is always available as a boot option.

Security Considerations

Security Benefits

Immutable base reduces attack surface
Atomic updates prevent partial/broken states
Easy rollback from compromised images
Reproducible builds enable auditing
Image signing verifies authenticity
Consistent state across all systems

Security Practices

Use signed images in production
Scan images for vulnerabilities
Keep base images updated
Minimize installed packages
Apply SELinux contexts correctly
Secure registry access

Image Signing & Scanning

# Sign images with cosign
[user@workstation ~]$ cosign sign registry.example.com/myserver:1.0

# Verify signature before deployment
[root@server ~]# cosign verify registry.example.com/myserver:1.0

# Scan for vulnerabilities
[user@workstation ~]$ podman image scan myserver:1.0

Best practices: sign all production images with cosign, scan images before deployment, keep base images updated with security fixes, and secure registry access with authentication and TLS.

Best Practices

Do

Start FROM official bootc base images
Use specific version tags, not :latest
Set SELinux contexts for added files
Enable services with systemctl enable
Clean dnf cache to reduce size
Test images before production deployment
Use CI/CD for automated building
Keep rollback images available

Do Not

Use non-bootc base images (UBI, etc.)
Modify /usr on running systems
Skip SELinux context configuration
Install packages locally with rpm-ostree
Deploy untested images to production
Ignore image signing in production
Let systems drift from the image
Delete all rollback deployments

The Philosophy

Treat the OS like a container — build it once, test it, deploy identical copies everywhere, update atomically, roll back instantly.

Build Once

→

Test It

→

Deploy Everywhere

→

Update Atomically

Key Takeaways

Image-Based: bootc deploys complete OS images atomically. Build with Containerfiles, deploy with bootc, update and rollback instantly.

Building: Start FROM bootc base images. Add packages, configs, enable services. Set SELinux contexts. Push to registry.

Deploying: Use bootc-image-builder for install media, or bootc install to-existing-root for conversions.

Managing: bootc status/upgrade/rollback. Atomic updates with one reboot. Instant rollback to previous image.

Graded Lab

HANDS-ON EXERCISES

Build a custom bootc image with web server and application
Push image to registry and deploy to VM
Update the image and perform atomic upgrade
Practice rollback after simulated problematic upgrade
Examine OSTree deployments with rpm-ostree status
Create CI pipeline for automated image building

Image-based management represents the future of Linux system administration