Linux: Capstone and Certification Prep
Synthesize the entire Linux curriculum into a production-grade infrastructure deployment. Master industry certification objectives across LFCS, RHCSA, and CompTIA Linux+, work through timed hands-on exam scenarios, and prepare for systems engineering interviews.
15.1 Capstone Project: Deploy a Secure Web Server
This capstone project transforms an unconfigured Linux machine into an audited, hardened, monitored production web host. Follow each step in sequence to construct an operational environment incorporating identity management, boundary network defense, web services, encrypted transport, scheduled maintenance, and disaster recovery.
15.1.1 Provision a Server and Create Users
Begin by establishing basic system identity, refreshing repository indexes, and creating dedicated unprivileged administrative accounts. Never perform day-to-day operations or deploy application code directly under the root account.
Provision the host and deployer account
Set the hostname, create the user, and install the SSH key.
# 1. Set static system hostname and update package repositories sudo hostnamectl set-hostname web-prod-01.internal sudo apt update && sudo apt -y upgrade # 2. Create unprivileged administrator account with custom UID sudo useradd -m -u 1200 -s /bin/bash -c "Primary Deployer" deployer # 3. Add deployer to administrative group (sudo on Debian/Ubuntu, wheel on RHEL) sudo usermod -aG sudo deployer # 4. Configure password aging policy (expire password every 90 days) sudo chage -M 90 -W 7 deployer # 5. Provision public key SSH authentication for deployer sudo mkdir -p /home/deployer/.ssh sudo cp ~/.ssh/authorized_keys /home/deployer/.ssh/ sudo chown -R deployer:deployer /home/deployer/.ssh sudo chmod 700 /home/deployer/.ssh sudo chmod 600 /home/deployer/.ssh/authorized_keys
User Provisioning Principles
- Dedicated Service Accounts:Create system users with no login shell (/usr/sbin/nologin) for daemons like web servers and databases.
- Granular Sudo Grants:Use drop-in files in /etc/sudoers.d/ rather than editing /etc/sudoers directly to preserve vendor package defaults.
- Strict Permissions:SSH directories must remain 0700 and authorized_keys 0600, or sshd will reject incoming key handshakes.
15.1.2 Harden SSH and Configure the Firewall
Lock down the remote management plane before placing any services online. Disable root login, enforce public key cryptography, and establish a default-deny packet filtering baseline.
Harden SSH and enable the firewall
Write a drop-in SSH config, reload sshd, and configure UFW.
# 1. Create drop-in hardened SSH configuration sudo tee /etc/ssh/sshd_config.d/99-hardened.conf << 'EOF' PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes MaxAuthTries 3 ClientAliveInterval 300 ClientAliveCountMax 2 X11Forwarding no EOF # 2. Test configuration syntax before reloading the daemon sudo sshd -t sudo systemctl reload sshd # 3. Configure packet filtering firewall with UFW sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow 22/tcp comment "SSH Remote Access" sudo ufw allow 80/tcp comment "HTTP Web Traffic" sudo ufw allow 443/tcp comment "HTTPS Encrypted Web Traffic" sudo ufw --force enable sudo ufw status verbose
Firewall Defense in Depth
- Default Deny Rule:Always block inbound ports by default and explicitly whitelist required service ports one by one.
- Daemon Pre-flight Validation:Run sshd -t to verify syntax before reloading to avoid getting locked out of remote servers.
- State Tracking:Both UFW and firewalld preserve established connections while rejecting unsolicited ingress packets.
15.1.3 Install Nginx, Deploy a Site, Add HTTPS
Install Nginx, isolate the application document root under appropriate group permissions, configure a virtual host, and obtain a trusted TLS certificate via Let's Encrypt.
Nginx site with HTTPS
Install Nginx, write the site, enable it, and request a certificate.
# 1. Install Nginx and Let's Encrypt Certbot
sudo apt install -y nginx certbot python3-certbot-nginx
# 2. Create web root directory and demo landing page
sudo mkdir -p /var/www/capstone/html
sudo chown -R deployer:www-data /var/www/capstone
sudo chmod -R 755 /var/www/capstone
cat << 'EOF' | sudo tee /var/www/capstone/html/index.html
<!DOCTYPE html>
<html lang="en">
<head><title>Production Host</title></head>
<body><h1>Production Capstone Host Active</h1></body>
</html>
EOF
# 3. Define Nginx server block
sudo tee /etc/nginx/sites-available/capstone.conf << 'EOF'
server {
listen 80;
server_name capstone.example.com;
root /var/www/capstone/html;
index index.html;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
location / {
try_files $uri $uri/ =404;
}
}
EOF
# 4. Enable configuration and test syntax
sudo ln -sf /etc/nginx/sites-available/capstone.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
# 5. Acquire automated SSL/TLS certificate with Certbot
sudo certbot --nginx -d capstone.example.com --non-interactive --agree-tos -m admin@example.com15.1.4 Add Monitoring, Backups and a Cron Job
A production system requires automated backups and status health checks. Implement a backup script with tar compression, SHA256 integrity verification, retention pruning, and scheduled execution via crontab.
Automated backup script and cron job
Write the script, set permissions, and schedule it daily.
# 1. Create backup script at /usr/local/bin/system-backup.sh
sudo tee /usr/local/bin/system-backup.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/var/backups/web-capstone"
TIMESTAMP="$(date +%Y%m%d_%H%M%S)"
ARCHIVE_NAME="capstone_${TIMESTAMP}.tar.gz"
mkdir -p "${BACKUP_DIR}"
# Create compressed archive of web files and configs
tar -czf "${BACKUP_DIR}/${ARCHIVE_NAME}" -C /var/www/capstone html
# Generate checksum for archive integrity
sha256sum "${BACKUP_DIR}/${ARCHIVE_NAME}" > "${BACKUP_DIR}/${ARCHIVE_NAME}.sha256"
# Remove archives older than 7 days
find "${BACKUP_DIR}" -type f -name "capstone_*.tar.gz*" -mtime +7 -delete
logger -t capstone-backup "Backup completed: ${ARCHIVE_NAME}"
EOF
# 2. Assign execution permissions
sudo chmod 750 /usr/local/bin/system-backup.sh
sudo chown root:root /usr/local/bin/system-backup.sh
# 3. Install scheduled daily backup cron job
sudo tee /etc/cron.d/capstone-backup << 'EOF'
# Run daily at 02:30 AM as root
30 2 * * * root /usr/local/bin/system-backup.sh >/var/log/backup.log 2>&1
EOFAutomated System Resilience
- Integrity Checksums:Always produce SHA256 hashes alongside archives to verify data validity prior to restorations.
- Strict Script Flags:Using set -euo pipefail ensures errors halt execution immediately instead of producing corrupt or partial archives.
- System Log Forwarding:Calling the logger command pipes backup status directly into systemd journal and syslog facilities.
15.1.5 Document the Build as a Runbook
In professional environments, an undocumented system is an unmaintainable liability. A production runbook provides step-by-step instructions that allow any administrator to recreate the host or recover from fatal incidents.
| Runbook Section | Required Details | Operational Target |
|---|---|---|
| System Metadata | Hostname, OS release, kernel version, IP addresses, DNS records | Inventory tracking and network reachability |
| Base Configuration | Created user accounts, assigned UIDs, SSH options, active firewall rules | Security compliance and auditable access control |
| Service Architecture | Installed packages, active daemon names, listening sockets, virtual host paths | Application runtime support and dependency resolution |
| Maintenance and Backup | Script locations, backup storage path, retention policy, cron schedules | Data protection and automated scheduled maintenance |
| Disaster Recovery | Restoration commands, configuration rollbacks, failover contact matrix | Rapid incident mitigation and minimal downtime |
15.2 Certification Roadmap
Certifications provide verifiable proof of practical command-line ability. Select a credential aligned with your career focus, whether vendor-neutral cloud engineering or enterprise RHEL environments.
15.2.1 LFCS: Domains and Weighting
The Linux Foundation Certified System Administrator (LFCS) exam is a 100% performance-based practical exam administered in a live terminal environment. Candidates choose between Ubuntu and Rocky Linux (RHEL derivative).
| LFCS Domain | Weighting | Core Exam Objectives |
|---|---|---|
| Essential Commands | 20% | File operations, grep, sed, tar, file globbing, shell scripts, IO redirection |
| Operation of Running Systems | 20% | Boot process, systemd targets, journalctl, process signals, load analysis |
| User and Group Management | 10% | User creation, group modifications, password aging, sudo privileges, PAM |
| Networking | 12% | IP routing, network links, DNS clients, firewalls (nftables, UFW, firewalld) |
| Service Configuration | 20% | SSH daemon, HTTP servers, storage sharing (NFS), NTP clock synchronization |
| Storage Management | 18% | GPT partitioning, LVM physical/volume groups, swap, fstab mount options |
LFCS Exam Strategies
- Keyboard Efficiency:Use Bash history expansions, reverse search (Ctrl+R), and tab autocompletion to maximize time.
- Offline Manual Lookups:Use man -k keyword and /EXAMPLE inside man pages to verify flags without web access.
- Service Restart Validation:Always verify daemons remain active across system reboots before marking exam tasks complete.
15.2.2 RHCSA and CompTIA Linux+ Compared
Choosing the right certification depends on your targeted industry vertical, employer requirements, and personal background:
| Certification | Exam Format | Ecosystem Focus | Best Suited For |
|---|---|---|---|
| LFCS (Linux Foundation) | Hands-on practical terminal (2 hours) | Ubuntu or Rocky Linux (Generic) | DevOps, Cloud engineers, Open Source roles |
| RHCSA (Red Hat EX200) | Live VM lab exam (3 hours) | RHEL (SELinux, Stratis, Podman, firewalld) | Enterprise SysAdmin, Government, Banking |
| CompTIA Linux+ (XK0-005) | Multiple-choice + simulated PBQs (90 min) | Vendor-neutral broad theoretical concepts | Helpdesk Tier 2, Junior Admin, Foundation |
15.2.3 Study Plan and Exam Tips
Achieving high scores on practical exams requires discipline and structured laboratory repetition. Follow this battle-tested preparation framework:
Practical Exam Survival Protocol
- Week 1-2 (Foundation):Master user accounts, POSIX permissions, ACLs, vim navigation, and shell pipeline scripting.
- Week 3-4 (Storage & Services):Practice LVM volume creation, filesystem expansion, swap allocation, and systemd units.
- Week 5-6 (Network & Security):Drill IP configuration, firewall packet filtering, SSH hardening, and troubleshooting connectivity.
- Week 7-8 (Mock Lab Drills):Complete timed break-fix challenges with zero internet access, using only local man pages.
- Ten-Minute Rule:Never stay stuck on a single exam item for more than 10 minutes; flag the task and return later.
- Reboot Verification:Always restart the testing virtual machine 15 minutes before exam conclusion to ensure persistent mounts.
15.3 Practice Exams and Labs
Practical exams assess your ability to solve real configuration challenges under time pressure. Work through these performance scenarios without referring to external search engines.
15.3.1 Timed Command-Line Exercises
Exercise 1: Collaborative Directory with SGID and Sticky Bit
Task: Create a directory named /srv/developers. Set its group ownership to devops. Ensure all new files created inside inherit the group ownership automatically, while ensuring only file owners can delete or rename their files.
Shared directory with SGID and sticky bit
Create the group and directory, then apply mode 3770.
# 1. Create group and target directory sudo groupadd devops sudo mkdir -p /srv/developers # 2. Assign group ownership sudo chgrp devops /srv/developers # 3. Apply SGID (2) and Sticky bit (1) with read/write/execute for owner and group sudo chmod 3770 /srv/developers # 4. Verify resulting mode string ls -ld /srv/developers # Expected output: drwxrws--t 2 root devops 4096 /srv/developers
Exercise 2: LVM Volume Expansion Without Downtime
Task: Expand the logical volume /dev/vg_data/lv_app by 5GB and resize the underlying ext4 filesystem without unmounting the active partition.
Extend an LVM volume online
Check free space, extend, and verify the filesystem.
# 1. Verify available space inside the volume group sudo vgs # 2. Extend logical volume and underlying filesystem simultaneously sudo lvextend -r -L +5G /dev/vg_data/lv_app # 3. If the -r flag is not available on legacy tools, resize manually: # sudo lvextend -L +5G /dev/vg_data/lv_app # sudo resize2fs /dev/vg_data/lv_app # for ext4 # sudo xfs_growfs /mount/point # for XFS # 4. Verify new filesystem dimensions df -h /dev/vg_data/lv_app
Exercise 3: Custom Systemd Unit with Automatic Restart
Task: Create a systemd service unit for /usr/local/bin/worker.py that runs as user deployer, restarts automatically on failure after 5 seconds, and starts on boot in multi-user target.
systemd service for a worker
Write the unit file, then enable and start it.
# 1. Create unit file at /etc/systemd/system/app-worker.service sudo tee /etc/systemd/system/app-worker.service << 'EOF' [Unit] Description=Application Background Worker After=network.target [Service] Type=simple User=deployer ExecStart=/usr/bin/python3 /usr/local/bin/worker.py Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target EOF # 2. Reload systemd, enable at boot, and start the service sudo systemctl daemon-reload sudo systemctl enable --now app-worker.service sudo systemctl status app-worker.service
15.3.2 Mock Exam Walkthrough
In this practical scenario, an administrator accidentally introduced a broken mount entry in /etc/fstab, causing the system to drop into emergency mode during boot. Follow the recovery path:
Emergency mode recovery
The prompt text is a comment; the rest run in the emergency shell.
# 1. At the emergency prompt, supply the root password to access the shell # Prompt shown: Give root password for maintenance # (or press Control-D to continue): # 2. Remount the root filesystem as read-write to enable modifications mount -o remount,rw / # 3. Inspect recent boot journal entries to find the failing mount point journalctl -xb | grep -i mount # 4. Edit /etc/fstab and comment out or fix the erroneous device UUID vim /etc/fstab # 5. Reload systemd mount generators to verify resolution systemctl daemon-reload mount -a # 6. Exit emergency shell and resume normal multi-user target boot exit
15.4 Career Paths and Interview Preparation
Linux proficiency is the single most transferable skill in infrastructure engineering. From bare-metal hardware to containerized Kubernetes nodes and serverless clouds, Linux powers modern digital services.
15.4.1 Linux Admin, DevOps, Cloud, and Security
| Career Role | Primary Linux Focus | Core Toolchain | Key Responsibility |
|---|---|---|---|
| Linux SysAdmin | User lifecycle, storage, PAM, backups, systemd | Bash, LVM, NFS, Samba, cron, rsyslog | Host uptime, patch management, bare-metal health |
| DevOps / Platform | Container runtimes, namespaces, build pipelines | Docker, Ansible, Git, CI/CD, Kubernetes | Automating software delivery and cluster scaling |
| Site Reliability (SRE) | Kernel parameters, performance bottlenecks, TCP tuning | Prometheus, Grafana, sysctl, strace, eBPF | Availability SLAs, latency mitigation, incident review |
| Cloud Engineer | Cloud-init images, ephemeral storage, VPC networking | Terraform, AWS EC2, GCP Compute, Azure VMs | Cloud architecture provisioning and cost efficiency |
15.4.2 Building a Portfolio on GitHub
Employers prioritize working demonstrations over generic resumes. Showcase concrete infrastructure repositories with clear documentation:
High-Impact Portfolio Projects
- Hardening Repository:Publish an Ansible playbook or Bash script that applies CIS Benchmark hardening to fresh Ubuntu/RHEL instances.
- Production Runbook:Include a complete Markdown runbook documenting architecture diagrams, backup tests, and failover steps.
- Monitoring Stack:Provide a docker-compose setup deploying Prometheus and Grafana dashboards for Linux host metric visualization.
- Automated CI/CD Pipeline:Build a GitHub Actions workflow that executes ShellCheck on scripts and runs Docker container test suites.
15.4.3 Technical Interview Questions
Review these deep-dive questions frequently asked during systems engineering and DevOps hiring rounds:
Q1: What happens from the moment you execute a command in Bash until it finishes?
Answer: The shell reads input, parses tokens, and checks for aliases or shell builtins. If external, it resolves the executable via $PATH. The shell issues a fork() system call to duplicate its process, sets up file descriptors (stdin, stdout, stderr), calls execve() to load the executable into memory, and executes waitpid() until the child terminates, returning its exit code.
Q2: How do you troubleshoot high CPU wait (wa) reported by top?
Answer: High wa indicates CPU cores idling while waiting for disk or network I/O operations to complete. Investigate using vmstat 1 to monitor blocked processes (b column), iostat -xz 1 to identify disks with high service times and saturation (%util), and iotop -oPa to pinpoint the specific process producing excessive write or read operations.
Q3: Why would a filesystem report "No space left on device" when df -h shows 40% free?
Answer: Two common causes: 1) Inode exhaustion caused by millions of tiny files or zero-byte logs; verify with df -i. 2) Open unlinked files: a process holds an open file descriptor to a large file that was deleted from disk; verify with lsof +L1 and restart the offending daemon to release allocated blocks.
Q4: What is the technical difference between a hard link and a symbolic (soft) link?
Answer: A hard link is another directory entry pointing to the identical underlying inode number. Hard links cannot span across different filesystems or link to directories. A symbolic link is a separate file with its own unique inode containing a textual path reference to the target file. Deleting a target breaks a symlink, whereas a hard link retains the data until its link count hits zero.
15.5 Interactive Lab: Final System Review
Execute final verification commands across user accounts, storage usage, and system specifications with the commands below. Press Load in Terminal to open them in the Linux terminal:
Final System Review
Load these in the terminal, then press Enter to run them.
$ id$ df -h$ df -i$ cat /etc/os-release
Knowledge Check: Module 15 Capstone and Certification Prep
1. Which exam format does the Linux Foundation LFCS and Red Hat RHCSA use?
2. What is the most reliable way to find command syntax during an offline practical exam?
3. In an operational runbook, which section documents steps to restore services following host corruption?
4. Which command flag creates a new user account with a designated primary numeric UID?
5. What is the primary operational difference between CompTIA Linux+ and LFCS exams?
6. Before concluding any practical exam scenario, what action ensures configurations persist across reboots?