Module 12: Boot Process, Kernel and Troubleshooting - Diagnostics & Recovery

12.1 The Boot Process

Starting a Linux computer involves a multi-stage handoff across motherboard hardware, firmware, bootloader binaries, compressed kernel images, and the initial userspace init system. Understanding each transition enables rapid diagnosis when systems fail to boot.

12.1.1 The Four Stages of Linux Boot

Every Linux machine passes through four distinct execution phases on initial startup:

Stage	Core Component	Execution Location	Operational Responsibility
1. Firmware	UEFI / BIOS	Motherboard ROM / NVRAM	Executes POST hardware verification; locates EFI System Partition (ESP) or MBR
2. Bootloader	GRUB2	/boot/efi & /boot/grub/	Displays boot menu, loads compressed kernel (vmlinuz) and initramfs into RAM
3. Kernel	vmlinuz + initramfs	System RAM (Kernel Space)	Initializes device drivers, mounts temporary root in RAM, mounts real storage root
4. Init System	systemd (PID 1)	Userspace Memory	Starts services, mounts non-root filesystems, and brings host to target runlevel

12.1.2 GRUB Configuration and Recovery

GRUB (Grand Unified Bootloader) version 2 is the default bootloader for major Linux distributions:

`GRUB Maintenance Rules`

/etc/default/grub:The user-editable template file where default kernel command-line arguments, timeouts, and terminal console settings are configured.
Do Not Edit /boot/grub/grub.cfg Directly:This configuration file is generated automatically. Manual edits are discarded when a new kernel package installs.
Debian / Ubuntu Regeneration:Execute sudo update-grub to recompile the active boot configuration.
RHEL / Rocky Linux Regeneration:Execute sudo grub2-mkconfig -o /boot/grub2/grub.cfg (or /boot/efi/EFI/rocky/grub.cfg on UEFI).

12.1.3 Rescue and Emergency Modes

`Maintenance Targets`

rescue.target (Single-User Mode):Mounts all local storage partitions and launches a maintenance root shell with basic services running (no networking). Trigger from a live system via sudo systemctl isolate rescue.target.
emergency.target (Minimal Shell):Mounts only the root partition in read-only mode without starting any background daemons. Used when filesystem errors prevent standard boot.
GRUB Boot Parameter Overrides:Press e on the GRUB menu screen, locate the kernel line (starting with linux), and append systemd.unit=rescue.target to boot directly into recovery.

12.2 The Kernel and Modules

The Linux kernel is a monolithic core that dynamically extends its hardware and protocol support using Loadable Kernel Modules (LKMs). LKMs load and unload on demand without requiring system reboots.

12.2.1 Managing Modules: uname, lsmod, and modprobe

Use dedicated utilities to inspect the active kernel release, query loaded modules, and resolve module dependencies:

Command	Function	Practical Example
uname -r / uname -a	Prints the active kernel release, architecture, and hostname	uname -r
lsmod	Formats and lists all kernel modules loaded in memory	lsmod \| grep -i kvm
modinfo <module>	Displays author, license, dependencies, and parameter details for a module	modinfo overlay
sudo modprobe <module>	Loads a module into the kernel with automatic dependency resolution	sudo modprobe br_netfilter
sudo modprobe -r <module>	Safely unloads an unused module and its unused dependencies	sudo modprobe -r overlay

Blacklist a kernel module

Write the blacklist file, then rebuild initramfs so it applies at boot.

# Blacklist a module to prevent automatic loading at boot
# Creates /etc/modprobe.d/blacklist.conf
printf 'blacklist nouveau\noptions nouveau modeset=0\n' | sudo tee /etc/modprobe.d/blacklist.conf

# Rebuild initramfs to apply blacklist changes on boot
sudo update-initramfs -u

12.2.2 Hardware and Kernel Ring Buffer (dmesg)

`Hardware Discovery Tools`

dmesg -T --level=err,warn:Dumps kernel ring buffer messages with human-readable timestamps filtered strictly to warning and error events.
lscpu:Displays CPU sockets, core counts, threads per core, virtualization support, and L1/L2/L3 cache architectures.
lspci -nnk:Lists all connected PCI devices alongside numeric vendor IDs and active kernel driver bindings.
lsusb:Enumerates connected USB controllers and external peripherals.

12.2.3 Kernel Parameters via sysctl

The sysctl utility modifies kernel parameters at runtime by updating the virtual filesystem representation inside /proc/sys/:

`sysctl Runtime and Persistent Tuning`

Read Active Value: sysctl parameter.name:Queries current setting: sysctl net.ipv4.ip_forward.
Temporary Modification: sysctl -w:Applies immediate change without reboot: sudo sysctl -w net.ipv4.ip_forward=1.
Persistent Configuration: /etc/sysctl.d/*.conf:Save settings into dedicated drop-in files such as /etc/sysctl.d/99-network.conf.
Reload All Parameters: sudo sysctl --system:Parses all drop-in files across /etc/sysctl.d/, /run/sysctl.d/, and /usr/lib/sysctl.d/.

12.3 Performance Monitoring

Effective performance troubleshooting begins by measuring CPU saturation, memory allocation pressure, disk I/O wait latency, and network throughput before altering configurations.

12.3.1 CPU and Memory Metrics

Inspect processor queue lengths and physical RAM distribution using standard core utilities:

`Core Metric Interpreters`

uptime (Load Averages):Displays 1, 5, and 15-minute load averages. On an 8-core CPU, a load average below 8.0 indicates nominal capacity; above 8.0 indicates queued processes.
free -h (RAM & Swap):Highlights total, used, free, and available memory. The available column is the true measure of memory remaining before swapping occurs.
vmstat 1 5 (Virtual Memory Statistics):Samples system state every 1 second. The wa column tracks percentage of CPU time spent waiting on disk I/O; r tracks runnable processes.

Virtual memory statistics

Sample vmstat every second and read the key columns.

# Sample virtual memory statistics every 1 second (5 intervals)
vmstat 1 5

# Key vmstat columns to monitor:
# r  : Number of processes waiting for CPU run time
# b  : Number of processes in uninterruptible sleep (waiting on disk/network I/O)
# si : Memory swapped in from disk per second (paging in)
# so : Memory swapped out to disk per second (paging out - high values indicate RAM exhaustion)
# wa : Percentage of CPU time spent waiting on pending I/O operations

12.3.2 Disk and Network I/O

`Storage and Network Profilers`

iostat -xz 1 (sysstat):Reports extended disk metrics per block device. Monitor await (average I/O response time in milliseconds) and %util (percentage of time the device was busy).
sudo iotop -o:Interactive top-like interface showing only processes currently producing disk read or write operations.
sudo iftop -i eth0:Real-time network bandwidth monitor displaying throughput per host socket pair.

12.3.3 Isolating the Constrained Subsystem

Use this diagnostic matrix to match observed performance symptoms to the root hardware bottleneck:

Observed Metric Profile	Primary Bottleneck	Primary Tool	Immediate Action
High load avg + high %us / %sy	CPU Saturation	top / htop	Identify runaway processes; adjust priority with `renice`
High load avg + high %wa + high await	Storage I/O Bottleneck	iostat -xz 1 / iotop	Throttle rogue writing jobs; migrate write loads to faster storage
High si / so rates + low available RAM	Memory Exhaustion	free -m / vmstat	Reduce application heap pools; add swap space or expand RAM
High dropped packets + bandwidth cap	Network Saturation	iftop / ip -s link	Inspect connections with `ss -ta`; enable interface bonding

12.4 Troubleshooting Methodology

Approaching complex system failures with an organized diagnostic sequence prevents wasted time, minimizes downtime, and avoids unintended secondary outages.

12.4.1 The 5-Step Incident Response Sequence

`The 5-Step Incident Response Sequence`

1. Reproduce:Observe exact error messages and symptoms
2. Isolate:Narrow scope down to component or layer
3. Review Logs:Trace journalctl, dmesg, and service files
4. Apply Fix:Implement single minimal verified change
5. Verify:Confirm recovery and update runbooks

12.4.2 Resolving Common Failure Scenarios

Standard operational procedures for frequent Linux server failures:

Failure Scenario	Underlying Cause	Diagnostic Command	Resolution Action
No space left on device	Inode exhaustion or deleted files held open by processes	df -i / sudo lsof +L1	Delete unused files or restart process holding unlinked file
Process terminated abruptly	Kernel OOM Killer invoked due to RAM exhaustion	dmesg -T \| grep -i oom	Review process memory consumption; allocate swap partition
Service won't start	Configuration syntax error or port collision	journalctl -xeu <service>	Check port bindings with `ss -tulnp`; run daemon syntax check
Permission denied (Root)	Filesystem mounted read-only or blocked by MAC (SELinux/AppArmor)	mount \| grep '(ro,' / ausearch	Remount read-write or restore security contexts via `restorecon`

12.4.3 Recovering a Lost Root Password

If administrative credentials are lost on physical servers or virtual consoles, regain access through the bootloader:

GRUB root password recovery

Boot-menu steps are comments; the commands run from the emergency root shell.

# Step 1: Reboot host and press 'e' on the GRUB boot menu selection
# Step 2: Locate the line starting with 'linux' (or 'linux16')
# Step 3: Append 'init=/bin/bash' (or 'rd.break' on RHEL/Rocky) to the end of that line
# Step 4: Press Ctrl+X or F10 to boot directly into single-user root shell

# Step 5: Remount root filesystem in read-write mode
mount -o remount,rw /

# Step 6: Reset the root password
passwd root

# Step 7 (SELinux systems only): Trigger full filesystem context relabeling on next boot
touch /.autorelabel

# Step 8: Resume standard boot process
exec /sbin/init

12.5 Interactive Lab: Kernel and Resource Inspection

Practice checking kernel versions, inspecting uptime and load averages, viewing memory allocations, and querying system information with the commands below. Press Load in Terminal to open them in the Linux terminal:

Kernel and Resource Inspection

Load these in the terminal, then press Enter to run them.

$ uname -a
$ uptime
$ free -h
$ cat /proc/version

Knowledge Check: Module 12 Boot Process, Kernel and Troubleshooting

1. What is the first userspace process with PID 1 launched by the Linux kernel?

2. Which command dynamically loads a kernel module along with its required dependencies?

3. Which command reloads all persistent kernel parameter files located in /etc/sysctl.d/?

4. In vmstat output, what metric indicates that the CPU is waiting on disk I/O operations?

5. Which command reveals if a disk is out of metadata storage rather than block storage?

6. During root password recovery on SELinux systems, which file must be created to restore file labels?

Previous Next