RED HAT ENTERPRISE LINUX

Password Recovery

Gaining Administrative Access When Locked Out

College-Level Course Module | RHEL System Administration

Learning Objectives

1
Interrupt the boot process at GRUB

Edit boot parameters to add rd.break for initramfs access

2
Access the system without authentication

Use the initramfs shell to access the root filesystem

3
Reset or unlock the root password

Change password and handle locked accounts

4
Fix SELinux context and complete recovery

Ensure proper relabeling so login works after reboot

When You Need Recovery

Password recovery is needed when you cannot authenticate as root through normal means - the password is unknown, forgotten, or the account is locked.

Forgotten Password

Administrator forgot the root password. No one else knows it. Most common scenario.

Inherited System

Took over a system from someone who left. Credentials weren't documented or handed over.

Locked Account

Root account locked due to failed attempts or administrative action. Password correct but can't login.

Emergency Access

No sudo users available. Need root for emergency repairs. Only option is console recovery.

Requirement: Physical or console access to the system. This procedure cannot be done remotely - you must be able to interact with GRUB.

Recovery Method Overview

1
Interrupt Boot at GRUB Press a key to stop automatic boot, select kernel entry
2
Edit Boot Parameters Press 'e' to edit, add rd.break to the linux line
3
Boot to initramfs Shell Press Ctrl+x to boot, land at switch_root:/# prompt
4
Mount Root Filesystem Read-Write Remount /sysroot with write permissions
5
Change Root and Reset Password chroot /sysroot, run passwd root
6
Fix SELinux Context Create /.autorelabel to trigger filesystem relabeling
7
Exit and Reboot Exit shells, system reboots, relabels, then normal boot

Step 1: Interrupt GRUB

When the system starts, GRUB displays a boot menu briefly (default 5 seconds). Press any key to stop the countdown and interact with the menu.

GNU GRUB version 2.06
Red Hat Enterprise Linux (5.14.0-362.13.1.el9_3.x86_64) 9.3
Red Hat Enterprise Linux (5.14.0-362.8.1.el9_3.x86_64) 9.3
Red Hat Enterprise Linux (0-rescue-abc123def456) 9.3

Use ↑ and ↓ keys to change selection.
Press 'e' to edit the selected entry.
Press 'c' for a command-line.
Be quick! Default timeout is 5 seconds. Watch the screen carefully during POST and press a key as soon as you see the GRUB menu appear.

Step 2: Edit Boot Parameters

# After pressing 'e', you see the boot entry configuration:
setparams 'Red Hat Enterprise Linux (5.14.0-362.13.1.el9_3.x86_64) 9.3'
        
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,gpt2' abc123...
        else
          search --no-floppy --fs-uuid --set=root abc123...
        fi
        linux   ($root)/vmlinuz-5.14.0-362.13.1.el9_3.x86_64 root=/dev/mapper/rhel-root 
                ro crashkernel=1G-4G:192M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root 
                rd.lvm.lv=rhel/swap rhgb quiet
        initrd  ($root)/initramfs-5.14.0-362.13.1.el9_3.x86_64.img

# Find the line starting with 'linux' (highlighted above)
# Navigate to the END of that line
# Add: rd.break
Navigation: Use arrow keys to move. The linux line may wrap across multiple screen lines. Go to the very end after all existing parameters.

Adding rd.break

rd.break interrupts the boot process in the initramfs, before the real root filesystem is fully mounted and before systemd starts. No authentication is required.

# Before (end of linux line):
... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet

# After adding rd.break:
... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet rd.break

# Optionally, remove 'rhgb quiet' to see boot messages:
... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rd.break

✓ Checkpoint

The linux line should now end with rd.break. Double-check before proceeding. Press Ctrl+x to boot with these parameters.

Why rd.break works: The initramfs runs before systemd, before login, before authentication is configured. It's a minimal environment designed to prepare the real root - and it trusts whoever has console access.

Step 3: Boot to initramfs

# Press Ctrl+x to boot with your modified parameters
# System boots, then stops with:

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

switch_root:/#

# You now have an initramfs shell!
# The prompt "switch_root:/#" indicates you're in initramfs

# Check what's mounted
switch_root:/# mount | grep sysroot
/dev/mapper/rhel-root on /sysroot type xfs (ro,relatime,seclabel,attr2,...)

# Notice: /sysroot is mounted READ-ONLY (ro)
# The real root filesystem is at /sysroot, not /
You're in! The switch_root:/# prompt means you're in the initramfs shell with access to the system. No password was required.

Step 4: Mount Root Read-Write

# The root filesystem at /sysroot is read-only
# We need to remount it with write permissions

switch_root:/# mount -o remount,rw /sysroot

# Verify it's now read-write
switch_root:/# mount | grep sysroot
/dev/mapper/rhel-root on /sysroot type xfs (rw,relatime,seclabel,attr2,...)

# Notice: now shows 'rw' instead of 'ro'

✓ Checkpoint

Verify mount shows rw (read-write) for /sysroot. If it still shows ro, the remount didn't work - check for typos and try again.

Why remount? The initramfs mounts your root read-only as a safety measure during the boot preparation phase. We override this to make changes.

Step 5: Reset Password

# Change root into the real filesystem
switch_root:/# chroot /sysroot

# Your prompt changes - you're now "in" the real system
sh-5.1# 

# Reset the root password
sh-5.1# passwd root
Changing password for user root.
New password:         # Type new password (not displayed)
Retype new password:  # Type again to confirm
passwd: all authentication tokens updated successfully.

# If the account was locked, unlock it:
sh-5.1# passwd -u root
Unlocking password for user root.
passwd: Success

# Alternatively, check account status:
sh-5.1# passwd -S root
root PS 2024-01-20 0 99999 7 -1 (Password set, SHA512 crypt.)
Don't exit yet! The password is changed, but we still need to fix SELinux context. Continue to the next step.

Locked vs Forgotten Accounts

Forgotten Password

  • Password unknown but account active
  • Solution: passwd root
  • Sets new password, account works

Locked Account

  • Account administratively locked
  • Password might be known but can't login
  • Solution: passwd -u root to unlock
  • May also need new password
# Check if account is locked
sh-5.1# passwd -S root
root LK 2024-01-20 0 99999 7 -1 (Password locked.)  # LK = Locked!

# Or check /etc/shadow directly
sh-5.1# grep ^root /etc/shadow
root:!!$6$abc123...:19742:0:99999:7:::  # !! means locked

# Unlock the account
sh-5.1# passwd -u root

# For both problems - set new password and ensure unlocked:
sh-5.1# passwd root       # Set new password
sh-5.1# passwd -u root    # Ensure unlocked

Step 6: Fix SELinux Context

Critical step! When we changed /etc/shadow outside the normal boot process, its SELinux security context became incorrect. Without fixing this, login will still fail.

# Still in chroot, create the autorelabel trigger file
sh-5.1# touch /.autorelabel

# Verify it was created
sh-5.1# ls -la /.autorelabel
-rw-r--r--. 1 root root 0 Jan 20 10:00 /.autorelabel

# This file tells SELinux to relabel the entire filesystem on next boot
⚠ Most common failure! Forgetting this step is the #1 reason password reset doesn't work. You changed the password correctly, but SELinux blocks login because /etc/shadow has the wrong context.
What happens: On next boot, before login, SELinux examines every file and restores correct labels. This can take several minutes on large filesystems. Be patient.

Alternative: Targeted Relabel

# Instead of relabeling entire filesystem (slow),
# you can fix just the shadow file's context

# Method 1: Load SELinux policy and fix context manually
sh-5.1# load_policy -i
sh-5.1# restorecon -v /etc/shadow
Relabeled /etc/shadow from system_u:object_r:unlabeled_t:s0 
          to system_u:object_r:shadow_t:s0

# Method 2: If load_policy isn't available, use /.autorelabel
sh-5.1# touch /.autorelabel

# Full relabel takes time but is more reliable
# Targeted restorecon is faster but requires working SELinux tools
Exam tip: The touch /.autorelabel method is simpler and always works. Use it unless you have a specific reason for targeted relabeling. Full relabel might take 5-10 minutes but is foolproof.

Step 7: Exit and Reboot

# Exit the chroot environment
sh-5.1# exit

# Back at initramfs prompt
switch_root:/# 

# Exit initramfs to continue boot (which triggers reboot)
switch_root:/# exit

# System reboots automatically
# During boot, you'll see SELinux relabeling messages:
*** Warning -- SELinux targeted policy relabel is required.
*** Relabeling could take a very long time, depending on file
*** system size and speed of hard drives.
****

# Wait for relabeling to complete (may take several minutes)
# System reboots again after relabeling
# Normal boot proceeds, login with new password!

✓ Final Checkpoint

After relabeling completes and system boots, log in as root with your new password. If it fails, you likely forgot /.autorelabel.

Complete Procedure

1 Reboot system, press any key to interrupt GRUB countdown
2 Select kernel entry, press 'e' to edit
3 Find linux line, add rd.break at the end
4 Press Ctrl+x to boot
5 mount -o remount,rw /sysroot
6 chroot /sysroot
7 passwd root (enter new password twice)
8 touch /.autorelabelDON'T FORGET!
9 exit, exit (exit chroot, then exit initramfs)
10 Wait for relabeling, login with new password

Security Implications

Physical access = Root access. The rd.break technique requires no authentication. Anyone with console access can reset the root password and take control of the system.

Physical Security

Servers must be in locked rooms. Limit who has physical access to machines.

GRUB Password

Set a GRUB password to prevent unauthorized boot parameter editing.

BIOS/UEFI Password

Prevent booting from external media. Lock BIOS configuration.

Disk Encryption

LUKS encryption protects data even if attacker boots their own media.

Defense in depth: Combine multiple protections. No single measure is sufficient - physical security, GRUB password, BIOS password, and encryption together provide robust protection.

Setting GRUB Password

# Generate password hash
[root@server ~]# grub2-setpassword
Enter password: 
Confirm password: 

# This creates /boot/grub2/user.cfg with hashed password
[root@server ~]# cat /boot/grub2/user.cfg
GRUB2_PASSWORD=grub.pbkdf2.sha512.10000.ABC123...

# Now GRUB requires password to edit entries or access command line
# Normal booting still works without password

# To remove GRUB password
[root@server ~]# rm /boot/grub2/user.cfg

# For UEFI systems, file location may differ:
[root@server ~]# cat /boot/efi/EFI/redhat/user.cfg
Don't lock yourself out! If you set a GRUB password and forget it, legitimate recovery becomes very difficult. Document the password securely.

Troubleshooting

ProblemCauseSolution
Can't stop GRUB countdownTimeout too fastKeep pressing keys, try arrow keys
No GRUB menu appearsHidden menuHold Shift (BIOS) or press Esc repeatedly
Can't find linux lineWrong entryLook for line with vmlinuz and root=
rd.break doesn't workTypo or wrong locationMust be on linux line, space before rd.break
No switch_root promptBoot continuedrd.break not added correctly, try again
mount remount failsFilesystem errorMay need fsck, or disk hardware issue
passwd command failsNot in chrootMust chroot /sysroot first
Login fails after rebootForgot /.autorelabelRepeat process, add autorelabel file
Relabeling takes foreverLarge filesystemNormal - wait 10-15+ minutes
Most common issue: Login fails because /.autorelabel was forgotten. If this happens, go through the rd.break process again - you don't need to change the password again, just create /.autorelabel and reboot.

Alternative Methods

init=/bin/bash

  • Add init=/bin/bash to linux line
  • Kernel starts bash instead of systemd
  • Even more minimal than rd.break
  • Root filesystem may be read-only
  • Same remount and chroot needed

Rescue Media

  • Boot from RHEL installation ISO
  • Select "Troubleshooting" → "Rescue"
  • System mounts at /mnt/sysimage
  • chroot /mnt/sysimage
  • Works when GRUB is broken
# init=/bin/bash method (at GRUB, add to linux line):
linux ... rhgb quiet init=/bin/bash

# After boot:
bash-5.1# mount -o remount,rw /
bash-5.1# passwd root
bash-5.1# touch /.autorelabel
bash-5.1# exec /sbin/init    # Or: /sbin/reboot -f
When to use alternatives: init=/bin/bash works when rd.break doesn't. Rescue media works when GRUB is broken or you can't modify boot parameters.

Best Practices

✓ Do

  • Practice the procedure on test systems
  • Always remember /.autorelabel
  • Document root passwords securely
  • Use password managers/vaults
  • Implement physical security
  • Consider GRUB passwords for sensitive systems
  • Verify login works after recovery
  • Audit who performs recovery

✗ Don't

  • Skip SELinux relabeling step
  • Use this on systems you don't own
  • Leave recovery procedure documented publicly
  • Rely solely on root password knowledge
  • Forget to test the new password
  • Interrupt the relabeling process
  • Ignore the security implications
  • Perform on production without approval
Ethics reminder: Only perform password recovery on systems you're authorized to administer. Unauthorized access is illegal regardless of technical ability.

Key Takeaways

1

GRUB Access: Interrupt boot, press 'e' to edit, add rd.break to linux line, Ctrl+x to boot.

2

initramfs Shell: Land at switch_root:/# prompt. Remount: mount -o remount,rw /sysroot

3

Reset Password: chroot /sysroot, then passwd root. Use passwd -u root if locked.

4

SELinux Fix: touch /.autorelabelCRITICAL! Then exit twice to reboot.

LAB EXERCISES

  • Practice the complete password reset procedure on a test VM
  • Intentionally forget /.autorelabel to see SELinux denial
  • Lock root account with passwd -l root, then recover
  • Set a GRUB password and verify it blocks editing
  • Time yourself - aim for under 5 minutes total
  • Try the init=/bin/bash alternative method

Next: Configuring Time Synchronization