Module 4: Text Processing, Editors and Pipes - Vim, Streams, Grep, Sed & Awk

4.1 Terminal Text Editors

System administration frequently requires editing configuration files directly on remote headless servers over SSH. Two primary command line editors are standard across distributions: GNU nano and Vim.

4.1.1 nano for Quick Edits

GNU nano is a modeless, beginner-friendly editor displaying keyboard shortcuts along the bottom of the screen:

`Core nano Shortcuts`

Ctrl + O (WriteOut):Saves current modifications to disk without exiting the editor.
Ctrl + X (Exit):Exits nano. Prompts whether to save unsaved modifications if any exist.
Ctrl + W (Where Is):Searches for a text string forward through the document.
Ctrl + K / Ctrl + U:Cuts the current line into clipboard buffer (Ctrl+K) and pastes it at cursor (Ctrl+U).
Alt + / (Go to End) & Alt + \ (Go to Start):Quickly jumps cursor to the bottom or top of the file.

4.1.2 Vim Modes: Normal, Insert, and Command-Line

Vim (Vi IMproved) is a modal editor. Keystrokes act as navigation and editing commands rather than typing text directly unless switched explicitly into Insert mode:

Mode	How to Enter	Core Functionality	How to Return to Normal
Normal Mode	Default mode on launch; or press `Esc`	Text navigation, word deletion (`dw`, `dd`), copying (`yy`), and pasting (`p`)	Already in Normal Mode
Insert Mode	i (at cursor), a (after), o (new line)	Direct character typing and writing new text into the document buffer	Press Esc
Command Mode	: (colon in Normal Mode)	Saving files, quitting, running shell commands, and regex search/replace	Press Esc or Enter
Visual Mode	v (character), V (line), Ctrl+v (block)	Highlighting text blocks for mass deletion, indentation, or yank operations	Press Esc

4.1.3 Essential Vim Commands

`Vim Survival Reference`

:w (Write):Saves changes to disk without closing the file buffer.
:q (Quit):Closes Vim if no unsaved changes exist.
:wq or ZZ:Saves changes and quits in a single step.
:q! (Force Quit):Discards all unsaved edits and exits immediately.
/pattern (Search Forward):Searches for a text pattern. Press n to jump to next match, N for previous match.
:%s/old/new/g (Global Replace):Substitutes 'old' with 'new' across all lines in the document (add c flag to confirm each replacement).
u (Undo) & Ctrl+r (Redo):Reverts the last action (u) or re-applies an undone edit (Ctrl+r).

4.1.4 Setting the Default System Editor

When utilities like git commit, crontab -e, or visudo launch an editor, they consult environment variables and Debian system alternatives:

`Default Editor Configuration`

sudo update-alternatives --config editor:Interactive Debian/Ubuntu menu allowing selection of the system-wide default editor binary.
User Level: export EDITOR="vim":Add export EDITOR="vim" and export VISUAL="vim" inside ~/.bashrc to enforce personal preference for all child shells.

4.2 Input, Output and Redirection

Every process started in Linux opens three standard data streams by default. Connecting, separating, and redirecting these streams forms the core of Unix pipeline processing.

4.2.1 Standard Streams: stdin, stdout, stderr

Descriptor (FD)	Stream Name	Default Source / Destination	Typical Purpose
0	stdin	Keyboard input	Receives text and interactive parameters from the user
1	stdout	Terminal screen display	Outputs standard successful results and command responses
2	stderr	Terminal screen display	Outputs error diagnostics and warning messages separately

4.2.2 Redirection Operators: >, >>, 2>, &>

`Redirection Syntax Reference`

command > file.txt (Overwrite stdout):Redirects standard output to a file, overwriting existing file content completely.
command >> file.txt (Append stdout):Appends standard output to the end of the file without deleting existing lines.
command 2> errors.log (Redirect stderr):Routes only error messages to the file, leaving standard output on the terminal screen.
command > out.log 2>&1 or command &> out.log:Merges both standard output and standard error into a single combined log file.
command < input.txt (Input Redirection):Feeds the content of a file directly into the standard input of a process.

4.2.3 Pipelines with |

The pipe operator (|) connects the standard output of the command on its left directly into the standard input of the command on its right in memory without writing temporary files to disk:

`Pipeline Architecture`

cat /var/log/syslog | grep -i error | wc -l:Streams syslog, filters lines matching 'error', and counts the total matching lines.
Non-blocking streaming:Commands in a pipeline run concurrently in subshells, with the Linux kernel buffering data between them automatically.

4.2.4 The tee Utility and /dev/null

`Tee and the Black Hole`

tee filename (T-Junction Splitter):Copies standard input to both standard output (screen) and one or more files simultaneously: echo "deb url ..." | sudo tee /etc/apt/sources.list.d/custom.list.
/dev/null (The Bit Bucket):A special pseudo-device that discards any data written to it. To silence all error output, append 2> /dev/null. To silence all output completely, use &> /dev/null.

4.3 Searching Text with grep

The grep (Global Regular Expression Print) command scans files and pipeline streams for lines matching specific text patterns.

4.3.1 Common grep Options

Flag	Behavior	Example Usage
-i	Case-insensitive matching (matches 'Error', 'error', 'ERROR')	grep -i "failed" /var/log/auth.log
-v	Inverts match (prints only lines that DO NOT match pattern)	grep -v "^#" /etc/nginx/nginx.conf
-n	Displays line numbers alongside matching output	grep -n "listen" /etc/nginx/nginx.conf
-r / -R	Recursively searches all files in subdirectories	grep -r "192.168.1.1" /etc/
-c	Prints count of matching lines instead of displaying them	grep -c "404" access.log
-E	Enables Extended Regular Expressions (ERE, same as egrep)	grep -E "404\|500" access.log

4.3.2 Basic vs Extended Regular Expressions

`Regex Metacharacters`

^ (Caret):Anchor matching the start of a line (e.g., grep "^root" /etc/passwd).
$ (Dollar):Anchor matching the end of a line (e.g., grep "bash$" /etc/passwd).
^$ (Empty Line):Matches lines containing no characters. Combine with grep -v "^$" to strip blank lines.
. (Dot):Matches any single character except newline.
[0-9] and [a-z]:Character sets matching any single digit or letter in the specified range.
+ and ? and | (Extended ERE):One or more (+), zero or one (?), and logical OR (|) (e.g., grep -E "(error|fatal)" app.log).

4.3.3 Practical Log-Searching Examples

`Real-World Log Analysis`

Filter SSH Failed Logins:grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr (counts failed attempts by attacking IP address).
Strip Comments and Blank Lines:grep -v -E "^#|^$" /etc/redis/redis.conf (displays active configuration settings cleanly).
Context Display (-A, -B, -C):grep -C 3 "Exception" application.log (displays 3 lines of context before and after every match).

4.4 Transforming Text

Linux includes specialized utilities designed to slice columns, sort records, translate character encodings, perform stream substitutions, and parse delimited tabular data.

4.4.1 cut, sort, uniq, tr, wc

`Single-Purpose Text Utilities`

cut -d: -f1 /etc/passwd:Extracts fields delimited by a colon (-d:), printing field 1 (usernames).
sort -n -r file.txt:Sorts lines numerically (-n) in reverse descending order (-r). Use -k 2 to sort by column 2.
uniq -c:Filters adjacent duplicate lines and prefixes each with repetition count (requires input sorted first via sort | uniq -c).
tr '[:lower:]' '[:upper:]':Translates characters from standard input, converting all lowercase characters to uppercase.
wc -l:Counts line count in input stream (e.g., ps aux | wc -l counts running processes).

4.4.2 Stream Editing with sed

The sed (stream editor) parses input lines sequentially, applying editing instructions without human intervention:

`Common sed Operations`

Find and Replace: sed 's/foo/bar/g' file.txt:Replaces every occurrence of 'foo' with 'bar' across the entire file stream.
In-Place Modification (sed -i):Modifies the file directly on disk instead of writing to standard output: sed -i 's/127.0.0.1/0.0.0.0/g' server.conf.
Delete Matching Lines: sed '/pattern/d':Drops any line matching 'pattern' from the output stream.
Line Range Print: sed -n '10,25p' file.txt:Suppresses automatic printing (-n) and prints lines 10 through 25 exclusively.

4.4.3 Data Extraction with awk

awk is a complete pattern-scanning and field-processing programming language. By default, awk splits input lines into columns delimited by whitespace:

`awk Field Processing`

$0 vs $1, $2, ...:$0 represents the complete raw input line; $1 is the first field, $2 is the second, etc.
$NF (Number of Fields):References the final field on the line regardless of how many columns precede it.
Custom Delimiter (-F):awk -F: '{print $1, $3}' /etc/passwd (parses colon-separated files, printing username and UID).
Filtering with Conditionals:awk '$3 > 1000 {print $1}' /etc/passwd (prints usernames for accounts with UID above 1000).
Column Summation One-Liner:ls -l | awk '{sum += $5} END {print sum/1024 " KB"}' (sums byte sizes of listed files).

4.4.4 Comparing Files: diff and comm

`File Comparison Utilities`

diff -u file1.txt file2.txt:Unified diff format displaying lines added (+) and lines deleted (-) between revisions.
diff -r dir1/ dir2/:Recursively compares two directories, reporting files present in only one folder or showing text diffs for modified files.
comm -12 sorted1.txt sorted2.txt:Compares two sorted files line by line: column 1 shows lines unique to file1, column 2 unique to file2, and column 3 shows common lines (flag -12 suppresses columns 1 and 2, printing only shared entries).

Knowledge Check: Module 4 Text Processing, Editors and Pipes

1. Which mode in Vim accepts commands like ":wq"?

2. Which operator redirects only standard error to a file?

3. What does the pipe symbol "|" do in Linux?

4. Which sed command replaces "test" with "prod" globally in a stream?

5. In awk, which variable prints the entire input line?

6. What is the special device /dev/null used for?

Previous Next