College-Level Course Module | RHEL System Administration
Learning Objectives
1
Interrupt the boot process at GRUB
Edit boot parameters to add rd.break for initramfs access
2
Access the system without authentication
Use the initramfs shell to access the root filesystem
3
Reset or unlock the root password
Change password and handle locked accounts
4
Fix SELinux context and complete recovery
Ensure proper relabeling so login works after reboot
When You Need Recovery
Password recovery is needed when you cannot authenticate as root through normal means - the password is unknown, forgotten, or the account is locked.
Forgotten Password
Administrator forgot the root password. No one else knows it. Most common scenario.
Inherited System
Took over a system from someone who left. Credentials weren't documented or handed over.
Locked Account
Root account locked due to failed attempts or administrative action. Password correct but can't login.
Emergency Access
No sudo users available. Need root for emergency repairs. Only option is console recovery.
Requirement: Physical or console access to the system. This procedure cannot be done remotely - you must be able to interact with GRUB.
Recovery Method Overview
1
Interrupt Boot at GRUB
Press a key to stop automatic boot, select kernel entry
2
Edit Boot Parameters
Press 'e' to edit, add rd.break to the linux line
3
Boot to initramfs Shell
Press Ctrl+x to boot, land at switch_root:/# prompt
4
Mount Root Filesystem Read-Write
Remount /sysroot with write permissions
5
Change Root and Reset Password
chroot /sysroot, run passwd root
6
Fix SELinux Context
Create /.autorelabel to trigger filesystem relabeling
7
Exit and Reboot
Exit shells, system reboots, relabels, then normal boot
Step 1: Interrupt GRUB
When the system starts, GRUB displays a boot menu briefly (default 5 seconds). Press any key to stop the countdown and interact with the menu.
GNU GRUB version 2.06
Red Hat Enterprise Linux (5.14.0-362.13.1.el9_3.x86_64) 9.3
Red Hat Enterprise Linux (5.14.0-362.8.1.el9_3.x86_64) 9.3
Red Hat Enterprise Linux (0-rescue-abc123def456) 9.3
Use ↑ and ↓ keys to change selection.
Press 'e' to edit the selected entry.
Press 'c' for a command-line.
Be quick! Default timeout is 5 seconds. Watch the screen carefully during POST and press a key as soon as you see the GRUB menu appear.
Step 2: Edit Boot Parameters
# After pressing 'e', you see the boot entry configuration:setparams 'Red Hat Enterprise Linux (5.14.0-362.13.1.el9_3.x86_64) 9.3'
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod xfs
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='hd0,gpt2' abc123...
else
search --no-floppy --fs-uuid --set=root abc123...
fi
linux ($root)/vmlinuz-5.14.0-362.13.1.el9_3.x86_64 root=/dev/mapper/rhel-root
ro crashkernel=1G-4G:192M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root
rd.lvm.lv=rhel/swap rhgb quiet
initrd ($root)/initramfs-5.14.0-362.13.1.el9_3.x86_64.img# Find the line starting with 'linux' (highlighted above)# Navigate to the END of that line# Add: rd.break
Navigation: Use arrow keys to move. The linux line may wrap across multiple screen lines. Go to the very end after all existing parameters.
Adding rd.break
rd.break interrupts the boot process in the initramfs, before the real root filesystem is fully mounted and before systemd starts. No authentication is required.
# Before (end of linux line):... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet# After adding rd.break:... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet rd.break# Optionally, remove 'rhgb quiet' to see boot messages:... rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rd.break
✓ Checkpoint
The linux line should now end with rd.break. Double-check before proceeding. Press Ctrl+x to boot with these parameters.
Why rd.break works: The initramfs runs before systemd, before login, before authentication is configured. It's a minimal environment designed to prepare the real root - and it trusts whoever has console access.
Step 3: Boot to initramfs
# Press Ctrl+x to boot with your modified parameters# System boots, then stops with:Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
switch_root:/## You now have an initramfs shell!# The prompt "switch_root:/#" indicates you're in initramfs# Check what's mountedswitch_root:/# mount | grep sysroot
/dev/mapper/rhel-root on /sysroot type xfs (ro,relatime,seclabel,attr2,...)# Notice: /sysroot is mounted READ-ONLY (ro)# The real root filesystem is at /sysroot, not /
You're in! The switch_root:/# prompt means you're in the initramfs shell with access to the system. No password was required.
Step 4: Mount Root Read-Write
# The root filesystem at /sysroot is read-only# We need to remount it with write permissionsswitch_root:/# mount -o remount,rw /sysroot
# Verify it's now read-writeswitch_root:/# mount | grep sysroot
/dev/mapper/rhel-root on /sysroot type xfs (rw,relatime,seclabel,attr2,...)# Notice: now shows 'rw' instead of 'ro'
✓ Checkpoint
Verify mount shows rw (read-write) for /sysroot. If it still shows ro, the remount didn't work - check for typos and try again.
Why remount? The initramfs mounts your root read-only as a safety measure during the boot preparation phase. We override this to make changes.
Step 5: Reset Password
# Change root into the real filesystemswitch_root:/# chroot /sysroot
# Your prompt changes - you're now "in" the real systemsh-5.1# # Reset the root passwordsh-5.1# passwd root
Changing password for user root.
New password: # Type new password (not displayed)Retype new password: # Type again to confirmpasswd: all authentication tokens updated successfully.# If the account was locked, unlock it:sh-5.1# passwd -u root
Unlocking password for user root.
passwd: Success# Alternatively, check account status:sh-5.1# passwd -S root
root PS 2024-01-20 0 99999 7 -1 (Password set, SHA512 crypt.)
Don't exit yet! The password is changed, but we still need to fix SELinux context. Continue to the next step.
Locked vs Forgotten Accounts
Forgotten Password
Password unknown but account active
Solution: passwd root
Sets new password, account works
Locked Account
Account administratively locked
Password might be known but can't login
Solution: passwd -u root to unlock
May also need new password
# Check if account is lockedsh-5.1# passwd -S root
root LK 2024-01-20 0 99999 7 -1 (Password locked.)# LK = Locked!# Or check /etc/shadow directlysh-5.1# grep ^root /etc/shadow
root:!!$6$abc123...:19742:0:99999:7:::# !! means locked# Unlock the accountsh-5.1# passwd -u root
# For both problems - set new password and ensure unlocked:sh-5.1# passwd root # Set new passwordsh-5.1# passwd -u root # Ensure unlocked
Step 6: Fix SELinux Context
Critical step! When we changed /etc/shadow outside the normal boot process, its SELinux security context became incorrect. Without fixing this, login will still fail.
# Still in chroot, create the autorelabel trigger filesh-5.1# touch /.autorelabel
# Verify it was createdsh-5.1# ls -la /.autorelabel
-rw-r--r--. 1 root root 0 Jan 20 10:00 /.autorelabel# This file tells SELinux to relabel the entire filesystem on next boot
⚠ Most common failure! Forgetting this step is the #1 reason password reset doesn't work. You changed the password correctly, but SELinux blocks login because /etc/shadow has the wrong context.
What happens: On next boot, before login, SELinux examines every file and restores correct labels. This can take several minutes on large filesystems. Be patient.
Alternative: Targeted Relabel
# Instead of relabeling entire filesystem (slow),# you can fix just the shadow file's context# Method 1: Load SELinux policy and fix context manuallysh-5.1# load_policy -i
sh-5.1# restorecon -v /etc/shadow
Relabeled /etc/shadow from system_u:object_r:unlabeled_t:s0
to system_u:object_r:shadow_t:s0# Method 2: If load_policy isn't available, use /.autorelabelsh-5.1# touch /.autorelabel
# Full relabel takes time but is more reliable# Targeted restorecon is faster but requires working SELinux tools
Exam tip: The touch /.autorelabel method is simpler and always works. Use it unless you have a specific reason for targeted relabeling. Full relabel might take 5-10 minutes but is foolproof.
Step 7: Exit and Reboot
# Exit the chroot environmentsh-5.1# exit
# Back at initramfs promptswitch_root:/# # Exit initramfs to continue boot (which triggers reboot)switch_root:/# exit
# System reboots automatically# During boot, you'll see SELinux relabeling messages:*** Warning -- SELinux targeted policy relabel is required.
*** Relabeling could take a very long time, depending on file
*** system size and speed of hard drives.
****# Wait for relabeling to complete (may take several minutes)# System reboots again after relabeling# Normal boot proceeds, login with new password!
✓ Final Checkpoint
After relabeling completes and system boots, log in as root with your new password. If it fails, you likely forgot /.autorelabel.
Complete Procedure
1Reboot system, press any key to interrupt GRUB countdown
2Select kernel entry, press 'e' to edit
3Find linux line, add rd.break at the end
4Press Ctrl+x to boot
5mount -o remount,rw /sysroot
6chroot /sysroot
7passwd root (enter new password twice)
8touch /.autorelabel ← DON'T FORGET!
9exit, exit (exit chroot, then exit initramfs)
10Wait for relabeling, login with new password
Security Implications
Physical access = Root access. The rd.break technique requires no authentication. Anyone with console access can reset the root password and take control of the system.
Physical Security
Servers must be in locked rooms. Limit who has physical access to machines.
GRUB Password
Set a GRUB password to prevent unauthorized boot parameter editing.
BIOS/UEFI Password
Prevent booting from external media. Lock BIOS configuration.
Disk Encryption
LUKS encryption protects data even if attacker boots their own media.
Defense in depth: Combine multiple protections. No single measure is sufficient - physical security, GRUB password, BIOS password, and encryption together provide robust protection.
Setting GRUB Password
# Generate password hash[root@server ~]# grub2-setpassword
Enter password: Confirm password: # This creates /boot/grub2/user.cfg with hashed password[root@server ~]# cat /boot/grub2/user.cfg
GRUB2_PASSWORD=grub.pbkdf2.sha512.10000.ABC123...# Now GRUB requires password to edit entries or access command line# Normal booting still works without password# To remove GRUB password[root@server ~]# rm /boot/grub2/user.cfg
# For UEFI systems, file location may differ:[root@server ~]# cat /boot/efi/EFI/redhat/user.cfg
Don't lock yourself out! If you set a GRUB password and forget it, legitimate recovery becomes very difficult. Document the password securely.
Troubleshooting
Problem
Cause
Solution
Can't stop GRUB countdown
Timeout too fast
Keep pressing keys, try arrow keys
No GRUB menu appears
Hidden menu
Hold Shift (BIOS) or press Esc repeatedly
Can't find linux line
Wrong entry
Look for line with vmlinuz and root=
rd.break doesn't work
Typo or wrong location
Must be on linux line, space before rd.break
No switch_root prompt
Boot continued
rd.break not added correctly, try again
mount remount fails
Filesystem error
May need fsck, or disk hardware issue
passwd command fails
Not in chroot
Must chroot /sysroot first
Login fails after reboot
Forgot /.autorelabel
Repeat process, add autorelabel file
Relabeling takes forever
Large filesystem
Normal - wait 10-15+ minutes
Most common issue: Login fails because /.autorelabel was forgotten. If this happens, go through the rd.break process again - you don't need to change the password again, just create /.autorelabel and reboot.
Alternative Methods
init=/bin/bash
Add init=/bin/bash to linux line
Kernel starts bash instead of systemd
Even more minimal than rd.break
Root filesystem may be read-only
Same remount and chroot needed
Rescue Media
Boot from RHEL installation ISO
Select "Troubleshooting" → "Rescue"
System mounts at /mnt/sysimage
chroot /mnt/sysimage
Works when GRUB is broken
# init=/bin/bash method (at GRUB, add to linux line):linux ... rhgb quiet init=/bin/bash# After boot:bash-5.1# mount -o remount,rw /
bash-5.1# passwd root
bash-5.1# touch /.autorelabel
bash-5.1# exec /sbin/init # Or: /sbin/reboot -f
When to use alternatives: init=/bin/bash works when rd.break doesn't. Rescue media works when GRUB is broken or you can't modify boot parameters.
Best Practices
✓ Do
Practice the procedure on test systems
Always remember /.autorelabel
Document root passwords securely
Use password managers/vaults
Implement physical security
Consider GRUB passwords for sensitive systems
Verify login works after recovery
Audit who performs recovery
✗ Don't
Skip SELinux relabeling step
Use this on systems you don't own
Leave recovery procedure documented publicly
Rely solely on root password knowledge
Forget to test the new password
Interrupt the relabeling process
Ignore the security implications
Perform on production without approval
Ethics reminder: Only perform password recovery on systems you're authorized to administer. Unauthorized access is illegal regardless of technical ability.
Key Takeaways
1
GRUB Access: Interrupt boot, press 'e' to edit, add rd.break to linux line, Ctrl+x to boot.
2
initramfs Shell: Land at switch_root:/# prompt. Remount: mount -o remount,rw /sysroot
3
Reset Password:chroot /sysroot, then passwd root. Use passwd -u root if locked.
4
SELinux Fix:touch /.autorelabel — CRITICAL! Then exit twice to reboot.
LAB EXERCISES
Practice the complete password reset procedure on a test VM
Intentionally forget /.autorelabel to see SELinux denial
Lock root account with passwd -l root, then recover