Written by the

Linux: Capstone and Certification Prep

Synthesize the entire Linux curriculum into a production-grade infrastructure deployment. Master industry certification objectives across LFCS, RHCSA, and CompTIA Linux+, work through timed hands-on exam scenarios, and prepare for systems engineering interviews.

15.1 Capstone Project: Deploy a Secure Web Server

This capstone project transforms an unconfigured Linux machine into an audited, hardened, monitored production web host. Follow each step in sequence to construct an operational environment incorporating identity management, boundary network defense, web services, encrypted transport, scheduled maintenance, and disaster recovery.

15.1.1 Provision a Server and Create Users

Begin by establishing basic system identity, refreshing repository indexes, and creating dedicated unprivileged administrative accounts. Never perform day-to-day operations or deploy application code directly under the root account.

Provision the host and deployer account

Set the hostname, create the user, and install the SSH key.

# 1. Set static system hostname and update package repositories
sudo hostnamectl set-hostname web-prod-01.internal
sudo apt update && sudo apt -y upgrade

# 2. Create unprivileged administrator account with custom UID
sudo useradd -m -u 1200 -s /bin/bash -c "Primary Deployer" deployer

# 3. Add deployer to administrative group (sudo on Debian/Ubuntu, wheel on RHEL)
sudo usermod -aG sudo deployer

# 4. Configure password aging policy (expire password every 90 days)
sudo chage -M 90 -W 7 deployer

# 5. Provision public key SSH authentication for deployer
sudo mkdir -p /home/deployer/.ssh
sudo cp ~/.ssh/authorized_keys /home/deployer/.ssh/
sudo chown -R deployer:deployer /home/deployer/.ssh
sudo chmod 700 /home/deployer/.ssh
sudo chmod 600 /home/deployer/.ssh/authorized_keys

User Provisioning Principles

  • Dedicated Service Accounts:Create system users with no login shell (/usr/sbin/nologin) for daemons like web servers and databases.
  • Granular Sudo Grants:Use drop-in files in /etc/sudoers.d/ rather than editing /etc/sudoers directly to preserve vendor package defaults.
  • Strict Permissions:SSH directories must remain 0700 and authorized_keys 0600, or sshd will reject incoming key handshakes.

15.1.2 Harden SSH and Configure the Firewall

Lock down the remote management plane before placing any services online. Disable root login, enforce public key cryptography, and establish a default-deny packet filtering baseline.

Harden SSH and enable the firewall

Write a drop-in SSH config, reload sshd, and configure UFW.

# 1. Create drop-in hardened SSH configuration
sudo tee /etc/ssh/sshd_config.d/99-hardened.conf << 'EOF'
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
EOF

# 2. Test configuration syntax before reloading the daemon
sudo sshd -t
sudo systemctl reload sshd

# 3. Configure packet filtering firewall with UFW
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp comment "SSH Remote Access"
sudo ufw allow 80/tcp comment "HTTP Web Traffic"
sudo ufw allow 443/tcp comment "HTTPS Encrypted Web Traffic"
sudo ufw --force enable
sudo ufw status verbose

Firewall Defense in Depth

  • Default Deny Rule:Always block inbound ports by default and explicitly whitelist required service ports one by one.
  • Daemon Pre-flight Validation:Run sshd -t to verify syntax before reloading to avoid getting locked out of remote servers.
  • State Tracking:Both UFW and firewalld preserve established connections while rejecting unsolicited ingress packets.

15.1.3 Install Nginx, Deploy a Site, Add HTTPS

Install Nginx, isolate the application document root under appropriate group permissions, configure a virtual host, and obtain a trusted TLS certificate via Let's Encrypt.

Nginx site with HTTPS

Install Nginx, write the site, enable it, and request a certificate.

# 1. Install Nginx and Let's Encrypt Certbot
sudo apt install -y nginx certbot python3-certbot-nginx

# 2. Create web root directory and demo landing page
sudo mkdir -p /var/www/capstone/html
sudo chown -R deployer:www-data /var/www/capstone
sudo chmod -R 755 /var/www/capstone

cat << 'EOF' | sudo tee /var/www/capstone/html/index.html
<!DOCTYPE html>
<html lang="en">
<head><title>Production Host</title></head>
<body><h1>Production Capstone Host Active</h1></body>
</html>
EOF

# 3. Define Nginx server block
sudo tee /etc/nginx/sites-available/capstone.conf << 'EOF'
server {
    listen 80;
    server_name capstone.example.com;
    root /var/www/capstone/html;
    index index.html;

    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";

    location / {
        try_files $uri $uri/ =404;
    }
}
EOF

# 4. Enable configuration and test syntax
sudo ln -sf /etc/nginx/sites-available/capstone.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# 5. Acquire automated SSL/TLS certificate with Certbot
sudo certbot --nginx -d capstone.example.com --non-interactive --agree-tos -m admin@example.com

15.1.4 Add Monitoring, Backups and a Cron Job

A production system requires automated backups and status health checks. Implement a backup script with tar compression, SHA256 integrity verification, retention pruning, and scheduled execution via crontab.

Automated backup script and cron job

Write the script, set permissions, and schedule it daily.

# 1. Create backup script at /usr/local/bin/system-backup.sh
sudo tee /usr/local/bin/system-backup.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/var/backups/web-capstone"
TIMESTAMP="$(date +%Y%m%d_%H%M%S)"
ARCHIVE_NAME="capstone_${TIMESTAMP}.tar.gz"

mkdir -p "${BACKUP_DIR}"

# Create compressed archive of web files and configs
tar -czf "${BACKUP_DIR}/${ARCHIVE_NAME}" -C /var/www/capstone html

# Generate checksum for archive integrity
sha256sum "${BACKUP_DIR}/${ARCHIVE_NAME}" > "${BACKUP_DIR}/${ARCHIVE_NAME}.sha256"

# Remove archives older than 7 days
find "${BACKUP_DIR}" -type f -name "capstone_*.tar.gz*" -mtime +7 -delete

logger -t capstone-backup "Backup completed: ${ARCHIVE_NAME}"
EOF

# 2. Assign execution permissions
sudo chmod 750 /usr/local/bin/system-backup.sh
sudo chown root:root /usr/local/bin/system-backup.sh

# 3. Install scheduled daily backup cron job
sudo tee /etc/cron.d/capstone-backup << 'EOF'
# Run daily at 02:30 AM as root
30 2 * * * root /usr/local/bin/system-backup.sh >/var/log/backup.log 2>&1
EOF

Automated System Resilience

  • Integrity Checksums:Always produce SHA256 hashes alongside archives to verify data validity prior to restorations.
  • Strict Script Flags:Using set -euo pipefail ensures errors halt execution immediately instead of producing corrupt or partial archives.
  • System Log Forwarding:Calling the logger command pipes backup status directly into systemd journal and syslog facilities.

15.1.5 Document the Build as a Runbook

In professional environments, an undocumented system is an unmaintainable liability. A production runbook provides step-by-step instructions that allow any administrator to recreate the host or recover from fatal incidents.

Runbook SectionRequired DetailsOperational Target
System MetadataHostname, OS release, kernel version, IP addresses, DNS recordsInventory tracking and network reachability
Base ConfigurationCreated user accounts, assigned UIDs, SSH options, active firewall rulesSecurity compliance and auditable access control
Service ArchitectureInstalled packages, active daemon names, listening sockets, virtual host pathsApplication runtime support and dependency resolution
Maintenance and BackupScript locations, backup storage path, retention policy, cron schedulesData protection and automated scheduled maintenance
Disaster RecoveryRestoration commands, configuration rollbacks, failover contact matrixRapid incident mitigation and minimal downtime

15.2 Certification Roadmap

Certifications provide verifiable proof of practical command-line ability. Select a credential aligned with your career focus, whether vendor-neutral cloud engineering or enterprise RHEL environments.

15.2.1 LFCS: Domains and Weighting

The Linux Foundation Certified System Administrator (LFCS) exam is a 100% performance-based practical exam administered in a live terminal environment. Candidates choose between Ubuntu and Rocky Linux (RHEL derivative).

LFCS DomainWeightingCore Exam Objectives
Essential Commands20%File operations, grep, sed, tar, file globbing, shell scripts, IO redirection
Operation of Running Systems20%Boot process, systemd targets, journalctl, process signals, load analysis
User and Group Management10%User creation, group modifications, password aging, sudo privileges, PAM
Networking12%IP routing, network links, DNS clients, firewalls (nftables, UFW, firewalld)
Service Configuration20%SSH daemon, HTTP servers, storage sharing (NFS), NTP clock synchronization
Storage Management18%GPT partitioning, LVM physical/volume groups, swap, fstab mount options

LFCS Exam Strategies

  • Keyboard Efficiency:Use Bash history expansions, reverse search (Ctrl+R), and tab autocompletion to maximize time.
  • Offline Manual Lookups:Use man -k keyword and /EXAMPLE inside man pages to verify flags without web access.
  • Service Restart Validation:Always verify daemons remain active across system reboots before marking exam tasks complete.

15.2.2 RHCSA and CompTIA Linux+ Compared

Choosing the right certification depends on your targeted industry vertical, employer requirements, and personal background:

CertificationExam FormatEcosystem FocusBest Suited For
LFCS (Linux Foundation)Hands-on practical terminal (2 hours)Ubuntu or Rocky Linux (Generic)DevOps, Cloud engineers, Open Source roles
RHCSA (Red Hat EX200)Live VM lab exam (3 hours)RHEL (SELinux, Stratis, Podman, firewalld)Enterprise SysAdmin, Government, Banking
CompTIA Linux+ (XK0-005)Multiple-choice + simulated PBQs (90 min)Vendor-neutral broad theoretical conceptsHelpdesk Tier 2, Junior Admin, Foundation

15.2.3 Study Plan and Exam Tips

Achieving high scores on practical exams requires discipline and structured laboratory repetition. Follow this battle-tested preparation framework:

Practical Exam Survival Protocol

  • Week 1-2 (Foundation):Master user accounts, POSIX permissions, ACLs, vim navigation, and shell pipeline scripting.
  • Week 3-4 (Storage & Services):Practice LVM volume creation, filesystem expansion, swap allocation, and systemd units.
  • Week 5-6 (Network & Security):Drill IP configuration, firewall packet filtering, SSH hardening, and troubleshooting connectivity.
  • Week 7-8 (Mock Lab Drills):Complete timed break-fix challenges with zero internet access, using only local man pages.
  • Ten-Minute Rule:Never stay stuck on a single exam item for more than 10 minutes; flag the task and return later.
  • Reboot Verification:Always restart the testing virtual machine 15 minutes before exam conclusion to ensure persistent mounts.

15.3 Practice Exams and Labs

Practical exams assess your ability to solve real configuration challenges under time pressure. Work through these performance scenarios without referring to external search engines.

15.3.1 Timed Command-Line Exercises

Exercise 1: Collaborative Directory with SGID and Sticky Bit

Task: Create a directory named /srv/developers. Set its group ownership to devops. Ensure all new files created inside inherit the group ownership automatically, while ensuring only file owners can delete or rename their files.

Shared directory with SGID and sticky bit

Create the group and directory, then apply mode 3770.

# 1. Create group and target directory
sudo groupadd devops
sudo mkdir -p /srv/developers

# 2. Assign group ownership
sudo chgrp devops /srv/developers

# 3. Apply SGID (2) and Sticky bit (1) with read/write/execute for owner and group
sudo chmod 3770 /srv/developers

# 4. Verify resulting mode string
ls -ld /srv/developers
# Expected output: drwxrws--t 2 root devops 4096 /srv/developers

Exercise 2: LVM Volume Expansion Without Downtime

Task: Expand the logical volume /dev/vg_data/lv_app by 5GB and resize the underlying ext4 filesystem without unmounting the active partition.

Extend an LVM volume online

Check free space, extend, and verify the filesystem.

# 1. Verify available space inside the volume group
sudo vgs

# 2. Extend logical volume and underlying filesystem simultaneously
sudo lvextend -r -L +5G /dev/vg_data/lv_app

# 3. If the -r flag is not available on legacy tools, resize manually:
# sudo lvextend -L +5G /dev/vg_data/lv_app
# sudo resize2fs /dev/vg_data/lv_app       # for ext4
# sudo xfs_growfs /mount/point             # for XFS

# 4. Verify new filesystem dimensions
df -h /dev/vg_data/lv_app

Exercise 3: Custom Systemd Unit with Automatic Restart

Task: Create a systemd service unit for /usr/local/bin/worker.py that runs as user deployer, restarts automatically on failure after 5 seconds, and starts on boot in multi-user target.

systemd service for a worker

Write the unit file, then enable and start it.

# 1. Create unit file at /etc/systemd/system/app-worker.service
sudo tee /etc/systemd/system/app-worker.service << 'EOF'
[Unit]
Description=Application Background Worker
After=network.target

[Service]
Type=simple
User=deployer
ExecStart=/usr/bin/python3 /usr/local/bin/worker.py
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# 2. Reload systemd, enable at boot, and start the service
sudo systemctl daemon-reload
sudo systemctl enable --now app-worker.service
sudo systemctl status app-worker.service

15.3.2 Mock Exam Walkthrough

In this practical scenario, an administrator accidentally introduced a broken mount entry in /etc/fstab, causing the system to drop into emergency mode during boot. Follow the recovery path:

Emergency mode recovery

The prompt text is a comment; the rest run in the emergency shell.

# 1. At the emergency prompt, supply the root password to access the shell
# Prompt shown: Give root password for maintenance
# (or press Control-D to continue):

# 2. Remount the root filesystem as read-write to enable modifications
mount -o remount,rw /

# 3. Inspect recent boot journal entries to find the failing mount point
journalctl -xb | grep -i mount

# 4. Edit /etc/fstab and comment out or fix the erroneous device UUID
vim /etc/fstab

# 5. Reload systemd mount generators to verify resolution
systemctl daemon-reload
mount -a

# 6. Exit emergency shell and resume normal multi-user target boot
exit

15.4 Career Paths and Interview Preparation

Linux proficiency is the single most transferable skill in infrastructure engineering. From bare-metal hardware to containerized Kubernetes nodes and serverless clouds, Linux powers modern digital services.

15.4.1 Linux Admin, DevOps, Cloud, and Security

Career RolePrimary Linux FocusCore ToolchainKey Responsibility
Linux SysAdminUser lifecycle, storage, PAM, backups, systemdBash, LVM, NFS, Samba, cron, rsyslogHost uptime, patch management, bare-metal health
DevOps / PlatformContainer runtimes, namespaces, build pipelinesDocker, Ansible, Git, CI/CD, KubernetesAutomating software delivery and cluster scaling
Site Reliability (SRE)Kernel parameters, performance bottlenecks, TCP tuningPrometheus, Grafana, sysctl, strace, eBPFAvailability SLAs, latency mitigation, incident review
Cloud EngineerCloud-init images, ephemeral storage, VPC networkingTerraform, AWS EC2, GCP Compute, Azure VMsCloud architecture provisioning and cost efficiency

15.4.2 Building a Portfolio on GitHub

Employers prioritize working demonstrations over generic resumes. Showcase concrete infrastructure repositories with clear documentation:

High-Impact Portfolio Projects

  • Hardening Repository:Publish an Ansible playbook or Bash script that applies CIS Benchmark hardening to fresh Ubuntu/RHEL instances.
  • Production Runbook:Include a complete Markdown runbook documenting architecture diagrams, backup tests, and failover steps.
  • Monitoring Stack:Provide a docker-compose setup deploying Prometheus and Grafana dashboards for Linux host metric visualization.
  • Automated CI/CD Pipeline:Build a GitHub Actions workflow that executes ShellCheck on scripts and runs Docker container test suites.

15.4.3 Technical Interview Questions

Review these deep-dive questions frequently asked during systems engineering and DevOps hiring rounds:

Q1: What happens from the moment you execute a command in Bash until it finishes?

Answer: The shell reads input, parses tokens, and checks for aliases or shell builtins. If external, it resolves the executable via $PATH. The shell issues a fork() system call to duplicate its process, sets up file descriptors (stdin, stdout, stderr), calls execve() to load the executable into memory, and executes waitpid() until the child terminates, returning its exit code.

Q2: How do you troubleshoot high CPU wait (wa) reported by top?

Answer: High wa indicates CPU cores idling while waiting for disk or network I/O operations to complete. Investigate using vmstat 1 to monitor blocked processes (b column), iostat -xz 1 to identify disks with high service times and saturation (%util), and iotop -oPa to pinpoint the specific process producing excessive write or read operations.

Q3: Why would a filesystem report "No space left on device" when df -h shows 40% free?

Answer: Two common causes: 1) Inode exhaustion caused by millions of tiny files or zero-byte logs; verify with df -i. 2) Open unlinked files: a process holds an open file descriptor to a large file that was deleted from disk; verify with lsof +L1 and restart the offending daemon to release allocated blocks.

Q4: What is the technical difference between a hard link and a symbolic (soft) link?

Answer: A hard link is another directory entry pointing to the identical underlying inode number. Hard links cannot span across different filesystems or link to directories. A symbolic link is a separate file with its own unique inode containing a textual path reference to the target file. Deleting a target breaks a symlink, whereas a hard link retains the data until its link count hits zero.

15.5 Interactive Lab: Final System Review

Execute final verification commands across user accounts, storage usage, and system specifications with the commands below. Press Load in Terminal to open them in the Linux terminal:

Final System Review

Load these in the terminal, then press Enter to run them.

$ id
$ df -h
$ df -i
$ cat /etc/os-release

Knowledge Check: Module 15 Capstone and Certification Prep

1. Which exam format does the Linux Foundation LFCS and Red Hat RHCSA use?

2. What is the most reliable way to find command syntax during an offline practical exam?

3. In an operational runbook, which section documents steps to restore services following host corruption?

4. Which command flag creates a new user account with a designated primary numeric UID?

5. What is the primary operational difference between CompTIA Linux+ and LFCS exams?

6. Before concluding any practical exam scenario, what action ensures configurations persist across reboots?