Module 7: Processes, Services and Scheduling - systemd, cron & Signals

7.1 Understanding Processes

In Linux, a process is an active instance of a computer program executing in memory. Every process holds allocated address space, file descriptor tables, security credentials, and a distinct numerical identifier.

7.1.1 PID, PPID, and Process Execution States

When the Linux kernel boots, it initializes systemd as PID 1 (Process ID 1). All subsequent user and system processes are spawned as child processes by cloning existing ones (via fork() and execve() system calls):

State Code	State Name	Kernel Behavior	Administrative Significance
R	Running / Runnable	Actively executing CPU instructions or waiting in the scheduler run queue	Consumes active CPU processing cycles
S	Interruptible Sleep	Waiting for an event, network packet, or timer; wakes up upon receiving signals	Normal idle state for background daemons
D	Uninterruptible Sleep	Waiting on low-level physical disk I/O; cannot be interrupted by any kill signal	High counts indicate storage hardware bottlenecks or hung NFS
Z	Zombie (Defunct)	Terminated execution, but its parent process has not yet read its exit status code	Consumes no RAM or CPU, but holds an entry in the system process table
T	Stopped / Suspended	Suspended by a debugger or by pressing `Ctrl+Z`	Can be resumed using job control (`fg`, `bg`)

7.1.2 Inspecting Processes: ps and pstree

`Process Listing Commands`

ps aux (BSD Syntax):Displays every active process on the system with user, PID, %CPU, %MEM, virtual memory size (VSZ), resident memory (RSS), terminal, state code, and full launch arguments.
ps -ef (POSIX Standard):Prints all system processes displaying UID, PID, PPID (Parent PID), and start time.
pstree -p:Renders an ASCII tree visual showing parent-child hierarchy with process IDs, making it easy to identify which daemon spawned worker threads.

7.1.3 Job Control: Foreground, Background, and Suspensions

When running long operations (large backups, database imports) inside an active shell session, job control allows switching between foreground and background states:

`Job Control Syntax`

Append Ampersand (&):Launches the program directly into the background: python3 worker.py & (returns job number [1] 4821 and frees the prompt).
Ctrl + Z (Suspend):Sends SIGTSTP to the active foreground command, stopping its execution and returning you to the terminal prompt.
jobs:Lists all backgrounded and suspended jobs belonging to your current shell session.
bg %1:Resumes suspended job 1 in the background without tying up your terminal.
fg %1:Brings background job 1 back into the active foreground.

7.1.4 Detached Persistence: nohup and tmux

When an SSH connection drops, the kernel sends a SIGHUP (Hangup) signal to all child processes spawned by that login shell, killing them immediately:

`Session Survival Tools`

nohup command &:Ignores the SIGHUP signal and redirects standard output to nohup.out, allowing scripts to continue running after terminal exit.
tmux (Terminal Multiplexer):A modern terminal workspace manager. Start a persistent session with tmux new -s deploy, detach safely with Ctrl+B d, and reattach later from any SSH connection using tmux a -t deploy.

7.2 Monitoring and Killing Processes

7.2.1 Real-Time Monitoring: top and htop

`System Resource Metrics`

Load Average (1, 5, 15 min):The average number of runnable and uninterruptible processes. On an 8-core CPU, a load average above 8.0 indicates the system is queueing tasks faster than cores can execute them.
CPU States:us (user processes), sy (kernel system calls), id (idle), wa (iowait - waiting for disk access).
htop Interactive Monitor:An enhanced ncurses viewer. Use F3 to search, F4 to filter by process name, F6 to sort by memory/CPU, and F9 to dispatch kill signals interactively.

7.2.2 Process Termination: kill, killall, and pkill

`Process Killing Utilities`

kill PID:Dispatches a signal to a specific numeric process ID (defaults to SIGTERM 15).
killall program_name:Terminates all running processes matching the exact binary name (e.g., killall nginx).
pkill pattern:Kills processes based on regex pattern matching or user ownership: pkill -u student node.
pgrep pattern:Inspects matching process IDs without sending signals (e.g., pgrep -l python).

7.2.3 POSIX Signals: SIGTERM vs SIGKILL

Programs communicate with processes by dispatching software signals:

Signal	Number	Catchable?	Standard Purpose
SIGTERM	15	Yes	Graceful termination. Allows the program to save state, close sockets, and delete temp files
SIGKILL	9	No	Immediate uncatchable termination handled by kernel. Program cannot clean up resources or flush buffers
SIGHUP	1	Yes	Terminal disconnection notification; daemons frequently intercept this to reload configuration files
SIGINT	2	Yes	Interactive keyboard interrupt dispatched by pressing `Ctrl+C` in the terminal

7.2.4 Process Scheduling Priorities: nice and renice

Linux assigns CPU execution priorities via "niceness" values ranging from -20 (highest priority / least nice) to +19 (lowest priority / very nice to other tasks). Default priority is 0:

`Priority Tuning Commands`

nice -n 10 backup.sh:Starts a background backup script with lower CPU priority (+10) so it does not degrade database query responsiveness.
sudo renice -5 -p 2140:Elevates the priority of an active running process (-5). Only the root superuser can assign negative niceness values.

7.3 Managing Services with systemd

systemd is the standard system and service manager on modern Linux distributions. It initializes userspace, tracks background daemon health, resolves unit dependencies, and supervises processes automatically.

7.3.1 Service Lifecycle: systemctl start, stop, restart, status

Command	Operational Effect	Typical Use Case
sudo systemctl start nginx	Launches the service daemon immediately	Starting a stopped service manually
sudo systemctl stop nginx	Sends SIGTERM to stop the active service	Performing offline server maintenance
sudo systemctl restart nginx	Terminates and spawns a new service process	Applying major structural configuration changes
sudo systemctl reload nginx	Sends SIGHUP to reload configs without dropping active client connections	Zero-downtime website configuration updates
systemctl status nginx	Prints running state, active PID, memory footprint, and recent log entries	Primary troubleshooting command for service failures

7.3.2 Enabling and Disabling Boot Autostart

`Boot Management`

sudo systemctl enable nginx:Creates symbolic links in /etc/systemd/system/multi-user.target.wants/ pointing to the unit file, ensuring Nginx starts automatically during system boot.
sudo systemctl disable nginx:Removes symlink files, preventing the service from launching at boot (does not stop an already running instance).
sudo systemctl is-enabled nginx:Quick programmatic query returning 'enabled' or 'disabled'.

7.3.3 Writing Custom Systemd Service Units

To run custom Python apps, Node.js servers, or Go binaries as background daemons, create a unit definition inside /etc/systemd/system/myapp.service:

`Service Unit File Blueprint`

[Unit] Section:Description=My Web Application and After=network.target (ensures network stack is online before starting).
[Service] Section:Defines execution parameters: Type=simple, User=www-data, WorkingDirectory=/opt/app, ExecStart=/usr/bin/python3 app.py, and Restart=always.
[Install] Section:WantedBy=multi-user.target (defines which runlevel target activates this service at boot).
Applying Changes:Always execute sudo systemctl daemon-reload after modifying any unit file to instruct systemd to reparse configuration tables.

7.3.4 Targets (Modern Runlevels)

`System Targets`

multi-user.target:Standard headless server environment with networking and multiple user logins, but no GUI desktop.
graphical.target:Desktop environment including X11 or Wayland display managers.
rescue.target:Single-user emergency maintenance mode with root shell and local filesystems mounted.
systemctl isolate multi-user.target:Switches current operating state immediately without rebooting.

7.4 System Logging and Log Rotation

7.4.1 Querying Logs with journalctl

systemd-journald collects binary, indexed system logs across kernel events, boot sequences, and service outputs:

`Common journalctl Query Filters`

journalctl -u nginx:Filters log output strictly to messages produced by the Nginx service unit.
journalctl -u nginx -f:Follow mode: streams new incoming log entries live on screen as they happen.
journalctl -p err -b:Displays only error-level messages (-p err) recorded during the current boot cycle (-b).
journalctl --since "30 minutes ago":Filters records by human-readable time windows (supports dates like --since "2026-10-24 14:00").
journalctl -n 50 --no-pager:Prints the 50 most recent lines directly to standard output without piping through a fullscreen pager.

7.4.2 Legacy Log Files in /var/log

`Standard Plaintext Log Files`

/var/log/syslog:Traditional plain text system activity stream maintained by rsyslog.
/var/log/auth.log:Authentication records: ssh logins, sudo attempts, and failed password alerts.
/var/log/dmesg:Hardware device driver detection messages generated during kernel boot.

7.4.3 Automated Log Rotation with logrotate

To prevent text logs from consuming entire storage volumes, the logrotate daemon compresses, archives, and deletes old files based on rules in /etc/logrotate.d/:

`logrotate Rule Directives`

daily / weekly:Defines rotation frequency.
rotate 14:Retains 14 historical backup archives before purging oldest.
compress & delaycompress:Compresses rotated logs with gzip to conserve disk space, delaying compression on the most recent cycle so active loggers finish writing.
postrotate ... endscript:Executes a shell command after rotation (e.g., instructing Nginx to reopen its log file descriptor).

7.5 Task Scheduling with cron

The cron daemon is the standard background scheduler that executes commands, maintenance scripts, and backups at scheduled calendar intervals.

7.5.1 The Crontab 5-Field Syntax

Field	Allowed Range	Special Characters	Example Pattern
1: Minute	0 - 59	* , - /	*/15 (every 15 minutes)
2: Hour	0 - 23 (24h clock)	* , - /	2 (at 02:00 AM)
3: Day of Month	1 - 31	* , - /	1 (first day of month)
4: Month	1 - 12	* , - /	* (every month)
5: Day of Week	0 - 7 (0 or 7 = Sunday)	* , - /	1-5 (Monday through Friday)

7.5.2 User Crontabs vs System Crontabs

`Crontab Locations`

crontab -e:Edits the current user's individual crontab file stored in /var/spool/cron/crontabs/.
crontab -l:Lists all scheduled cron entries for the active account.
System Crontab (/etc/crontab):System-wide crontab featuring a 6th field specifying which username executes the task: 0 2 * * * root /usr/local/bin/backup.sh.

7.5.3 Common Cron Pitfalls

`Cron Debugging Checklist`

Minimal PATH Variable:Cron executes with a stripped-down PATH (typically /usr/bin:/bin). Always use full absolute paths for binaries and scripts (e.g., /usr/bin/python3 rather than python3).
Unredirected Output:By default, cron sends stdout and stderr to the local user mail spool. Always route output to a log file: 0 3 * * * /opt/backup.sh >> /var/log/backup.log 2>&1.
Working Directory Assumptions:Cron runs commands from the user home directory by default. Always specify explicit working paths or add cd /opt/app && ....

7.5.4 Real-World Cron Examples

`Production Cron Schedule Recipes`

Nightly Database Backup (02:30 AM):30 2 * * * /usr/bin/mysqldump -u root mydb > /backups/db.sql 2>&1
Weekly SSL Cert Renewal Check (Monday 04:00 AM):0 4 * * 1 /usr/bin/certbot renew --quiet
Hourly Temp Cleanup:0 * * * * /usr/bin/find /tmp/uploads -type f -mmin +120 -delete

7.6 One-Off at Jobs and Modern systemd Timers

7.6.1 One-Off Scheduling with at

`Executing Non-Recurring Tasks`

echo 'sh deploy.sh' | at 03:00 tomorrow:Queues a one-time execution at 03:00 AM on the following day.
atq:Lists all pending one-off jobs waiting in the queue.
atrm job_id:Cancels and deletes a queued task before execution.

7.6.2 Systemd Timers vs Cron

Modern Linux environments increasingly replace cron with systemd.timer units:

`Why Systemd Timers Outperform Cron`

Centralized Journal Logging:Timer execution outputs flow directly into journalctl -u mytask.service with timestamped records.
Monotonic Timers:Can schedule tasks relative to boot or service activation (e.g., OnBootSec=15min, OnUnitActiveSec=1h).
Catch-Up Execution (Persistent=true):If a laptop or server was powered off when the timer was scheduled to run, systemd triggers the missed job immediately upon next boot.
systemctl list-timers:Displays all active timers, when they last executed, and countdown until their next trigger.

Knowledge Check: Module 7 Processes, Services and Scheduling

1. Which signal requests graceful termination allowing cleanup?

2. Which command configures a systemd service to start automatically at boot?

3. What cron schedule expression executes a job every midnight?

4. Which utility filters systemd binary journal logs by service unit?

5. What command moves a stopped background job into active background execution?

6. What is the PID of the systemd init process in Linux?

Previous Next