AU294 · Red Hat Enterprise Linux Automation with Ansible

Troubleshooting

Chapter 8 of 8

Objective: Troubleshoot source-to-image (S2I) builds and deployments, and troubleshoot containerized applications running on OpenShift.

Overview

Every earlier chapter builds toward this one: when an image won't build, a pod won't start, or a running application misbehaves, a systematic approach to gathering evidence is faster and more reliable than guessing.

Key Concepts

Diagnosing a failed or stuck build

When an S2I or Containerfile build fails, the build logs are the first place to look. On OpenShift, oc logs build/<build-name> (or oc logs -f bc/<name> to follow the most recent build) shows exactly where the build process stopped, whether that's a failed dependency install, a missing file in the source repository, or a build resource quota being exceeded.

Diagnosing a pod that won't start

A pod that never reaches Running has a small number of likely causes, and oc describe pod <name> is the fastest way to narrow them down: check the Events section at the bottom of the output first.

Symptom Likely cause
ImagePullBackOff Wrong image name/tag, or missing registry credentials
CrashLoopBackOff The container's main process exits immediately — check oc logs
Pending Insufficient cluster resources, or an unsatisfiable scheduling constraint
ContainerCreating (stuck) Often a volume mount problem, e.g. an unbound PVC

Reading pod and container logs

oc logs <pod> (add -c <container> when a pod has more than one container, and -p to see logs from a previous, crashed instance of the container) is the primary tool for understanding what an application actually did before it failed.

Interactive investigation

For problems logs alone don't explain, oc exec -it <pod> -- /bin/bash opens a shell inside a running container to check configuration files, environment variables, and network connectivity directly — the same technique covered for Podman in an earlier chapter, applied through the OpenShift CLI.

Checking configuration and connectivity

Many application failures trace back to configuration, not the application code: oc describe configmap/secret confirms expected keys exist, oc get events --sort-by=.lastTimestamp surfaces recent cluster-level problems, and oc rsh combined with tools like curl from inside the cluster network confirms whether a Service is actually reachable.

Working with It

Start with describe, Then logs

oc describe pod shows scheduling and startup problems (Events); oc logs shows what happened once the container's process actually started — they answer different questions.

--previous Is Essential for Crash Loops

When a pod is stuck in CrashLoopBackOff, the current container instance may not have logs yet; oc logs --previous retrieves logs from the last instance before it crashed.

Don't Assume It's the Application

A surprising number of container failures trace back to a missing environment variable, an unmounted volume, or a misnamed Service — rule out configuration and connectivity before assuming the application code itself is at fault.

A typical troubleshooting sequence

$ oc get pods
$ oc describe pod mysite-7f8d9c-abcde
$ oc logs mysite-7f8d9c-abcde --previous
$ oc exec -it mysite-7f8d9c-abcde -- /bin/bash
[pod]$ env | grep DB_
[pod]$ curl -sv http://inventory-db:5432

Watch Out For

Common Pitfall

A surprising number of container failures trace back to a missing environment variable, an unmounted volume, or a misnamed Service — rule out configuration and connectivity before assuming the application code itself is at fault.

Key Terms

ImagePullBackOff
A pod status indicating the container image could not be pulled.
CrashLoopBackOff
A pod status indicating the container repeatedly starts and then exits/crashes.
oc describe
A CLI command showing detailed configuration and recent Events for a resource.
oc logs --previous
Retrieves logs from a container's last instance before it most recently crashed.
oc exec
Runs a command, or opens a shell, inside a running container via the OpenShift CLI.

Review Questions

  1. What are two of the most common reasons a pod ends up in ImagePullBackOff?
  2. Why would you use oc logs --previous instead of oc logs when debugging a CrashLoopBackOff?
  3. Which command would you check first to find out why a pod has been stuck in Pending for ten minutes?
  4. What might a stuck ContainerCreating state often indicate, based on this chapter?
  5. Describe a troubleshooting sequence you'd follow for an application pod that starts successfully but can't reach its database.