Variables & Strings
Variables & Assignment
Bash variables are untyped strings by default; arithmetic works if the value is numeric. NEVER put spaces around = in assignments — 'name = Alice' tries to run a command called name. Use $(...) for command substitution (preferred over backticks). export makes a variable available to child processes; readonly makes it immutable. Environment variables (exported) are inherited by scripts and programs you launch.
# no spaces around = in assignments
name="Alice"
age=30
PI=3.14
active=true
# use variables with $ prefix
echo "Hello, $name!"
echo "Age: $age"
# command substitution: capture command output
today=$(date +%Y-%m-%d)
files=$(ls | wc -l)
echo "Today is $today, $files files"
# readonly and environment variables
readonly MAX=100
export PATH="$PATH:/opt/bin"
env | grep PATHString Parameter Expansion
Parameter expansion is one of Bash's most powerful features, enclosed in ${...}. ${#var} gives length; ${var:offset:length} extracts substrings (negative offset counts from end — note the space before -). ,, and ^^ convert case (Bash 4+). :- provides defaults without setting the var; := sets it as a side effect; :? aborts with an error — great for required-argument checks in scripts.
s="Hello World"
echo ${#s} # length: 11
echo ${s:0:5} # substring: Hello
echo ${s:6} # from index 6: World
echo ${s: -5} # last 5 chars: World
# case conversion (Bash 4+)
echo ${s,,} # lowercase: hello world
echo ${s^^} # uppercase: HELLO WORLD
echo ${s^l} # capitalize first if 'l': Hello World
# default values
echo ${name:-Guest} # use Guest if name unset
echo ${name:=Guest} # assign Guest if unset, then use
echo ${name:?required} # error and exit if unsetString Search & Replace
The / pattern substitutes; // substitutes all; # matches at the start; % matches at the end. ## and %% are greedy (longest match) while # and % are non-greedy. These are essential for path manipulation: ${path##*/} extracts the basename (like basename command), and ${path%/*} extracts the directory (like dirname). Mastering these avoids many calls to external commands like sed and cut.
s="Hello World Hello"
# replace first occurrence
echo ${s/Hello/Hi} # Hi World Hello
# replace all occurrences
echo ${s//Hello/Hi} # Hi World Hi
# replace at beginning / end
echo ${s/#Hello/Hi} # Hi World Hello (prefix)
echo ${s/%Hello/Hi} # Hello World Hello (no suffix match)
# delete patterns (replace with empty)
echo ${s//Hello/} # World
path="/usr/local/bin/app"
echo ${path##*/} # app (longest prefix)
echo ${path%/*} # /usr/local/bin (shortest suffix)
# extract filename and extension
file="report.tar.gz"
echo ${file%.*} # report.tar (remove ext)
echo ${file##*.} # gz (keep ext)Quoting & Special Characters
Single quotes preserve everything literally — no variable expansion, no escape processing (you cannot even include a single quote inside single quotes). Double quotes allow $, backtick, and \ expansion while preserving spaces and special characters — almost always prefer double quotes around variables to prevent word-splitting and globbing bugs. Use $'...' for ANSI-C escapes like \n and \t.
# single quotes: literal, no expansion
echo 'Hello $USER' # Hello $USER
# double quotes: expansion happens
echo "Hello $USER" # Hello alice
# escape special chars with backslash
echo "Price: \\$5.00" # Price: $5.00
echo 'It\'s a test' # It's a test (escape inside single)
# concatenate strings
a="Hello"
b="World"
c="$a, $b!"
echo $c # Hello, World!
# multi-line string
msg="Line 1
Line 2
Line 3"
echo "$msg"Read & User Input
read captures user input. -p shows a prompt, -t sets a timeout (returns non-zero), -a reads into an array. When reading files line-by-line, always use 'IFS= read -r' — IFS= preserves leading/trailing whitespace, and -r disables backslash escaping so backslashes are preserved literally. This is the canonical safe pattern for processing files line by line.
# read a single value
echo -n "Enter your name: "
read name
echo "Hi, $name!"
# read multiple values
read -p "First and last: " first last
# read with timeout (seconds)
read -t 5 -p "Quick! " answer || echo "too slow"
# read into an array
echo "Enter colors:"
read -a colors
echo "First: ${colors[0]}"
# read line by line from a file
while IFS= read -r line; do
echo ">>> $line"
done < file.txtArrays
Indexed Arrays
Bash arrays are zero-indexed. @ and * expand to all elements — always quote them ("${arr[@]}") to handle elements with spaces correctly. ${#arr[@]} gives the count. += appends. unset leaves a gap in indices; to reindex, use arr=("${arr[@]}"). Slicing with ${arr[@]:start:count} returns a subset.
# create and populate
fruits=("apple" "banana" "cherry")
fruits[3]="date"
# access elements
echo ${fruits[0]} # apple
echo ${fruits[@]} # all: apple banana cherry date
echo ${fruits[*]} # same as @ in most contexts
# count and slice
echo ${#fruits[@]} # count: 4
echo ${fruits[@]:1:2} # elements 1-2: banana cherry
# append
fruits+=("elderberry")
# iterate
for f in "${fruits[@]}"; do
echo "- $f"
done
# delete an element
unset fruits[1] # removes "banana" (gap remains)Associative Arrays (Maps)
Associative arrays (declare -A) map string keys to values, like dictionaries in other languages — available in Bash 4+. Use ${!arr[@]} to get keys and ${arr[@]} for values. The -v test checks if a key exists. Unlike indexed arrays, keys are arbitrary strings, so quoting keys with spaces is essential. These require Bash 4+ (macOS ships Bash 3 by default).
# must declare before use (Bash 4+)
declare -A ages
# key-value assignment
ages[alice]=30
ages[bob]=25
ages["carol smith"]=28
# access by key
echo ${ages[alice]} # 30
echo ${ages[bob]} # 25
# all keys and values
echo ${!ages[@]} # keys: alice bob carol smith
echo ${ages[@]} # values: 30 25 28
echo ${#ages[@]} # count: 3
# iterate keys
for name in "${!ages[@]}"; do
echo "$name is ${ages[$name]}"
done
# check if key exists
if [[ -v ages[alice] ]]; then
echo "alice exists"
fiArray Iteration Patterns
There are three main iteration styles: by value (for x in "${arr[@]}"), by index (for i in "${!arr[@]}"), and C-style. The by-index form is useful when you need the position. Always quote "${arr[@]}" so elements containing spaces are preserved as single items. Building arrays with += in a loop is the idiomatic way to accumulate results.
arr=(one two three four five)
# iterate by value
for item in "${arr[@]}"; do
echo "$item"
done
# iterate by index
for i in "${!arr[@]}"; do
echo "[$i] = ${arr[$i]}"
done
# C-style indexed loop
for ((i=0; i<${#arr[@]}; i++)); do
echo "${arr[$i]}"
done
# map/transform: build a new array
upper=()
for w in "${arr[@]}"; do
upper+=("$(echo $w | tr a-z A-Z)")
done
echo "${upper[@]}"Array from Command Output
mapfile/readarray (Bash 4+) reads lines into an array in one shot — much faster than a while loop for large files. To split a delimited string, set IFS and use read -ra. For filenames (which may contain spaces or newlines), always use find -print0 with read -d '' to split on null bytes — the only safe delimiter for filenames. The < <(...) process substitution feeds the loop without a subshell.
# split a string into an array
csv="a,b,c,d"
IFS=',' read -ra parts <<< "$csv"
echo ${parts[2]} # c
# read lines of a file into an array
mapfile -t lines < file.txt
echo "${lines[0]}" # first line
echo "${#lines[@]}" # line count
# alternative: readarray (same as mapfile)
readarray -t lines2 < file.txt
# from command output (words split on IFS)
files=($(ls *.txt))
echo "${files[@]}"
# safer: read null-delimited output
while IFS= read -r -d '' f; do
files+=("$f")
done < <(find . -name "*.txt" -print0)Special Variables
These special variables are essential for scripting. $0 is the script path; $1-$9 are positional args (use ${10} for two digits). Always quote "$@" to preserve arguments with spaces — "$*" joins them into one string. $? gives the exit status (0 = success) and is the basis of all error checking. $$ and $! are useful for PID files and process management. shift discards $1 and shifts the rest down.
# positional parameters
echo $0 # script name
echo $1 # first argument
echo $# # argument count
echo $@ # all arguments (separate words)
echo "$@" # all arguments (preserves quoting)
echo $* # all arguments (single string)
# process and shell info
echo $$ # current shell PID
echo $! # PID of last background command
echo $? # exit status of last command (0 = success)
echo $- # current shell option flags
echo $_ # last argument of previous command
# shift arguments
echo "$1 $2" # a b
shift
echo "$1 $2" # b c
# iterate all args safely
for arg in "$@"; do
echo "arg: $arg"
doneControl Flow & Tests
If / Elif / Else
[ ] is the POSIX test command (portable but limited); [[ ]] is Bash's enhanced version supporting pattern matching (== with wildcards), regex (=~), && / || operators, and no need to quote variables. Prefer [[ ]] in Bash scripts. Always quote variables in [ ] to avoid syntax errors with empty or space-containing values. The -f, -r, -d, -e tests check file existence and attributes.
# basic if
if [ "$age" -ge 18 ]; then
echo "adult"
elif [ "$age" -ge 13 ]; then
echo "teen"
else
echo "child"
fi
# modern [[ ]] test (Bash-specific, safer)
if [[ $name == "Alice" ]]; then
echo "hi Alice"
fi
# pattern matching in [[ ]]
if [[ $file == *.txt ]]; then
echo "text file"
fi
# regex matching
if [[ $email =~ ^[a-z]+@[a-z]+.[a-z]+$ ]]; then
echo "valid email"
fi
# multiple conditions
if [[ -f file.txt && -r file.txt ]]; then
echo "readable file"
fiTest Operators
File tests (-f, -d, -r, etc.) check filesystem attributes. String tests use = and != (or == in [[ ]]); -z checks for empty, -n for non-empty. Integer comparisons use -eq, -ne, -lt, -gt, -le, -ge — do NOT use < > for integers in [ ] (they redirect!). In [[ ]] and (( )) you can use the familiar < > <= >= operators. Memorizing these operators is essential for writing correct conditionals.
# file tests
[ -f file ] # exists and is regular file
[ -d dir ] # exists and is directory
[ -e path ] # exists (any type)
[ -r file ] # readable
[ -w file ] # writable
[ -x file ] # executable
[ -s file ] # exists and is non-empty
[ file1 -nt file2 ] # newer than
[ file1 -ot file2 ] # older than
# string tests
[ -z "$s" ] # empty string
[ -n "$s" ] # non-empty string
[ "$a" = "$b" ] # equal (use == in [[ ]])
[ "$a" != "$b" ] # not equal
# integer tests
[ $n -eq 5 ] # equal
[ $n -ne 5 ] # not equal
[ $n -lt 5 ] # less than
[ $n -gt 5 ] # greater than
[ $n -le 5 ] # less or equal
[ $n -ge 5 ] # greater or equalCase Statement
case is the Bash equivalent of switch — it matches a value against glob patterns. | separates alternatives. ;; ends a branch (like break). The *) pattern is the default. Patterns support wildcards (*, ?, [abc]) but not full regex. case is cleaner than long if-elif chains for dispatching on a single value, and it's the standard way to build command-line subcommands (start/stop/restart).
case $1 in
start)
echo "Starting service..."
systemctl start app
;;
stop)
echo "Stopping service..."
systemctl stop app
;;
restart)
$0 stop
$0 start
;;
status)
systemctl status app
;;
--help|-h)
echo "Usage: $0 {start|stop|restart|status}"
;;
*)
echo "Unknown command: $1" >&2
exit 1
;;
esac
# pattern matching in case
case $file in
*.jpg|*.png) echo "image" ;;
*.mp3|*.wav) echo "audio" ;;
*.txt) echo "text" ;;
*) echo "other" ;;
esacArithmetic & (( ))
$(( )) evaluates arithmetic expressions and returns the result. Inside (( )), variable names don't need $ — just use the name. (( )) as a command returns exit status 0 (true) if the result is non-zero, making it ideal for numeric conditions. Bash only does integer arithmetic; for floating-point use bc or awk. Operators: + - * / % ** and C-style comparison and bitwise operators all work.
# arithmetic expansion
x=5
y=$((x + 3))
echo $y # 8
echo $((x * 2)) # 10
echo $((2 ** 10)) # 1024 (exponent)
echo $((17 % 5)) # 2 (modulo)
echo $((17 / 5)) # 3 (integer division)
# (( )) condition: no $ needed, returns exit status
x=10
if (( x > 5 && x < 20 )); then
echo "in range"
fi
# increment / decrement
((x++)) # post-increment
((x--)) # post-decrement
((++x)) # pre-increment
# bitwise operators
echo $((5 & 3)) # 1 (AND)
echo $((5 | 2)) # 7 (OR)
echo $((5 << 1)) # 10 (left shift)Short-Circuit & Ternary
&& and || are short-circuit operators that double as concise control flow: 'cmd1 && cmd2' runs cmd2 only if cmd1 succeeds; 'cmd1 || cmd2' runs cmd2 only if cmd1 fails. This replaces simple if-else in one line. The pattern 'A && B || C' mimics a ternary but is subtly buggy if B can fail — for robust code, use a real if statement. The colon (:) is a no-op command useful for empty branches.
# && and || as short-circuit control flow
[[ -f file.txt ]] && echo "exists"
[[ -f file.txt ]] || echo "missing"
# combined: do this OR fail
mkdir -p build && cd build && make || exit 1
# ternary-like (Bash doesn't have a real ternary)
result=$([[ $x -gt 0 ]] && echo "positive" || echo "non-positive")
# default value via &&
echo ${name:-"anonymous"}
# run command, fallback on failure
ping -c1 host && echo "up" || echo "down"
# null command as no-op
if condition; then
: # do nothing
fiLoops & Iteration
For Loops
The for loop iterates over a list of words. Brace expansion {1..5} generates sequences (with optional step {start..end..step}). The C-style for ((init; cond; update)) is best for numeric counters. When iterating files with *.txt, unquoted globs expand safely, but if no files match you get the literal '*.txt' — enable shopt -s nullglob to get an empty list instead.
# iterate a list
for fruit in apple banana cherry; do
echo "$fruit"
done
# iterate an array
colors=("red" "green" "blue")
for c in "${colors[@]}"; do
echo "Color: $c"
done
# range with brace expansion
for i in {1..5}; do
echo "Number: $i"
done
# range with step
for i in {0..20..2}; do echo $i; done # 0 2 4 ... 20
# C-style for loop
for ((i=0; i<5; i++)); do
echo "Iteration $i"
done
# iterate command output
for file in *.txt; do
echo "Processing $file"
doneWhile & Until Loops
while runs as long as the condition succeeds (exit 0); until runs until it succeeds — they're opposites. The 'while read' pattern is the canonical way to process files line by line; always use 'IFS= read -r' for safety. Note that piping into a while loop runs it in a subshell, so variable changes don't persist — use process substitution '< <(cmd)' instead when you need to keep variables.
# while: runs while condition is true
count=0
while [ $count -lt 5 ]; do
echo "Count: $count"
((count++))
done
# read lines from a file
while IFS= read -r line; do
echo "Line: $line"
done < input.txt
# infinite loop with break
while true; do
read -p "Quit? (y/n) " ans
[[ $ans == "y" ]] && break
done
# until: runs until condition is true (opposite of while)
n=0
until [ $n -ge 3 ]; do
echo "n=$n"
((n++))
done
# while reading from a pipeline
seq 1 5 | while read n; do
echo "got $n"
doneBreak, Continue & Select
break exits the loop (break N exits N nested loops); continue jumps to the next iteration. select creates an interactive numbered menu from a list — it loops until broken. These control statements work in for, while, and until loops. The break N form is essential for escaping deeply nested loops but should be used sparingly as it can make code harder to follow.
# break exits the loop
for i in {1..10}; do
[[ $i -eq 5 ]] && break
echo $i # prints 1 2 3 4
done
# continue skips to next iteration
for i in {1..6}; do
(( i % 2 == 0 )) && continue
echo $i # prints 1 3 5 (odd only)
done
# break out of nested loops
for i in 1 2 3; do
for j in a b c; do
[[ $j == "b" ]] && break 2 # break both loops
echo "$i$j"
done
done
# select menu
select opt in "Start" "Stop" "Quit"; do
case $opt in
Start) echo "starting" ;;
Stop) echo "stopping" ;;
Quit) break ;;
*) echo "invalid" ;;
esac
doneLooping Over Files Safely
Parsing ls output is a classic bug — filenames with spaces, newlines, or special characters break it. Use shell globs (*.txt) directly in for loops, and quote the variable. For recursive or complex searches, use find -print0 piped to 'read -d ''' — null bytes are the only safe delimiter for filenames. Enable nullglob so unmatched globs produce an empty list rather than the literal pattern.
# BAD: breaks on filenames with spaces
for f in $(ls *.txt); do
cat "$f"
done
# GOOD: glob expands safely
for f in *.txt; do
[[ -f "$f" ]] || continue
cat "$f"
done
# BEST: handle spaces, newlines, and no matches
shopt -s nullglob dotglob
for f in *.txt; do
echo "Processing: $f"
done
# find with -print0 and while read -d ''
while IFS= read -r -d '' f; do
echo "Found: $f"
done < <(find . -type f -name "*.txt" -print0)
# process files modified today
for f in $(find . -mtime -1 -print); do
echo "Recent: $f"
doneRanges & Sequence Generation
Brace expansion {a..b} generates sequences at parse time — fast and built-in. It supports letters, numbers, zero-padding, and steps ({start..end..step}). seq is an external command with more formatting options (-f for printf-style). Brace expansion also creates multiple arguments: echo file{1..3}.{txt,log} generates 6 filenames. Prefer brace expansion for simple ranges; use seq when you need custom formatting.
# brace expansion (not a loop, but generates lists)
echo {1..10} # 1 2 3 4 5 6 7 8 9 10
echo {a..e} # a b c d e
echo {01..10} # zero-padded: 01 02 ... 10
echo file{1..3}.txt # file1.txt file2.txt file3.txt
# seq command
seq 1 5 # 1 2 3 4 5
seq 1 2 10 # 1 3 5 7 9 (step 2)
seq -f "%03d" 1 5 # 001 002 003 004 005
# use seq in a for loop
for i in $(seq 1 5); do
echo $i
done
# generate a list of dates
for d in $(seq -f "2024-01-%02g" 1 31); do
echo "$d"
doneFunctions
Defining & Calling Functions
Functions in Bash are defined with name() or function name. Call them by name with space-separated arguments (no parentheses). ALWAYS use 'local' for variables inside functions to avoid polluting the global scope — without it, assignments leak out. Bash functions can't return values directly (return is an exit status 0-255); to return a string, echo it and capture with $().
# two equivalent syntaxes
greet() {
echo "Hello, $1!"
}
function greet2() {
echo "Hi, $1!"
}
# call with arguments (no parens)
greet "Alice" # Hello, Alice!
greet2 "Bob" # Hi, Bob!
# local variables (function-scoped)
counter() {
local count=0
((count++))
echo $count
}
# return a value via echo + command substitution
add() {
echo $(($1 + $2))
}
result=$(add 3 4) # 7Arguments & Parameters
Inside a function, $1, $@, $# refer to the function's arguments, not the script's — they shadow the script-level parameters. $FUNCNAME holds the function's name (useful for debugging). shift inside a function only affects the function's parameters. To pass the script's arguments to a function, use func "$@". This shadowing is why functions are reusable building blocks.
# all positional params work inside functions
show_args() {
echo "Function name: $FUNCNAME"
echo "First arg: $1"
echo "Second arg: $2"
echo "All args: $@"
echo "Arg count: $#"
}
show_args a b c
# Function name: show_args
# First arg: a
# Second arg: b
# All args: a b c
# Arg count: 3
# pass all script args to a function
process "$@"
# shift within a function (local to it)
parse() {
while (($#)); do
echo "arg: $1"
shift
done
}Return Values & Exit Status
Bash's 'return' only sets an exit status (0-255), so functions double as boolean tests. To return a string, echo it and capture with $(). For multiple values, use a nameref (local -n) — Bash 4.3+ — which lets the function assign to a caller's variable by name. This is the cleanest way to return complex data. Avoid using global variables for return values as it makes functions non-reentrant.
# return sets exit status (0-255), not a value
is_even() {
if (($1 % 2 == 0)); then
return 0 # true/success
else
return 1 # false/failure
fi
}
# use as a condition
if is_even 4; then
echo "4 is even"
fi
# chain with && and ||
is_even 4 && echo "yes" || echo "no"
# return a string via echo (capture with $())
to_upper() {
echo "${1^^}"
}
UP=$(to_upper hello) # HELLO
# return multiple values via global or nameref
set_pair() {
local -n _r=$1
_r=(10 20)
}
set_pair result
echo "${result[@]}" # 10 20Variable Scope
Variables in Bash are global by default — even when assigned inside a function! This is a common source of bugs. Always use 'local' for function-internal variables. 'declare -g' explicitly creates a global from within a function. Recursion works but is slow in Bash (each call forks a subshell for $()); use it sparingly. The factorial example shows the pattern: local variables + recursive call + arithmetic.
# global by default
x=1
modify() {
x=2 # changes the GLOBAL x!
local y=3 # local to function
echo "inside: x=$x y=$y"
}
modify
echo "outside: x=$x" # x=2 (modified!)
echo "outside: y=$y" # empty (y is local)
# declare -g for global inside a function
set_global() {
declare -g NEW_VAR=42
}
set_global
echo $NEW_VAR # 42
# recursion with local
factorial() {
local n=$1
if (( n <= 1 )); then
echo 1
else
local prev=$(factorial $((n - 1)))
echo $((n * prev))
fi
}
echo $(factorial 5) # 120Libraries & Sourcing
source (or .) executes a file in the current shell, so its functions and variables become available — this is how you build reusable libraries. A common pattern is a utils.sh with helper functions that scripts source. Unlike executing a script (which runs in a subshell), sourced code can modify the caller's environment. This is also how .bashrc and .bash_profile work — they're sourced at shell startup.
# utils.sh — a reusable library
#!/bin/bash
log() {
echo "[$(date +%H:%M:%S)] $*" >&2
}
die() {
log "ERROR: $*"
exit 1
}
confirm() {
read -p "$1 [y/N] " ans
[[ $ans =~ ^[Yy]$ ]]
}
# main.sh — use the library
source ./utils.sh # or . ./utils.sh
log "Starting script"
confirm "Continue?" || die "aborted"
log "Done"
# source vs execute
# source: runs in current shell (shares variables)
# ./script.sh: runs in a subshell (isolated)Text Processing
grep — Pattern Search
grep finds lines matching a pattern. -i ignores case, -v inverts, -n shows line numbers, -r recurses, -E uses extended regex (like +, |, {}). -A/-B/-C show context lines around matches — invaluable for understanding errors. Use --include to filter file types during recursive search. grep returns exit status 0 if a match is found, making it useful in conditions: if grep -q pattern file; then...
# basic search
grep "error" app.log
grep -i "error" app.log # case-insensitive
grep -v "debug" app.log # invert (non-matching lines)
grep -c "error" app.log # count matches
grep -n "error" app.log # line numbers
grep -w "error" app.log # whole word only
# recursive search
grep -rn "TODO" src/ # recursive + line numbers
grep -rl "TODO" src/ # files with matches only
# extended regex (or use egrep)
grep -E "error|warn|fatal" log
grep -E "^[0-9]{4}-" log # lines starting with date
# context lines
grep -A 2 "error" log # 2 lines after
grep -B 2 "error" log # 2 lines before
grep -C 2 "error" log # 2 lines before and after
# search multiple files
grep "function" *.js --include="*.js" -r .sed — Stream Editor
sed edits text streams non-interactively. The s command substitutes; g makes it global. -i edits in place (always test without -i first!). The delimiter can be any char — use | or # for paths to avoid escaping slashes. sed processes line by line, so multi-line operations need the N command or hold space. For complex edits, awk or perl may be clearer. Always quote sed scripts to prevent shell expansion.
# substitute (replace)
sed 's/old/new/' file.txt # first occurrence per line
sed 's/old/new/g' file.txt # all occurrences (global)
sed 's/old/new/3' file.txt # 3rd occurrence only
sed 's/old/new/gi' file.txt # global + case-insensitive
# in-place editing
sed -i 's/foo/bar/g' file.txt # edit file in place
sed -i.bak 's/foo/bar/g' file.txt # keep a backup
# delete lines
sed '/^#/d' file.txt # delete comment lines
sed '/^$/d' file.txt # delete blank lines
sed '5d' file.txt # delete line 5
sed '5,10d' file.txt # delete lines 5-10
# print specific lines
sed -n '10,20p' file.txt # print lines 10-20
sed -n '/pattern/p' file.txt # print matching lines
# use a different delimiter for paths
sed 's|/usr/local|/opt|g' paths.txt
# multiple commands
sed -e 's/a/A/g' -e 's/b/B/g' file.txtawk — Column Processing
awk is a mini programming language for columnar data. $1, $2... are fields; $0 is the whole line; $NF is the last field. -F sets the input separator; OFS sets output. BEGIN runs before processing, END after. NR is the record (line) number; NF is the field count. awk is ideal for CSV/TSV processing, log analysis, and generating reports — far more powerful than cut for anything beyond simple extraction.
# print columns (default separator: whitespace)
awk '{print $1}' file.txt # first column
awk '{print $1, $3}' file.txt # columns 1 and 3
awk '{print $NF}' file.txt # last column
awk -F: '{print $1}' /etc/passwd # split on :
# filter and print
awk '$3 > 100' file.txt # lines where col 3 > 100
awk '$1 == "ERROR" {print $2}' log # col 2 of ERROR lines
awk 'NR == 5' file.txt # 5th line only
awk 'NR >= 10 && NR <= 20' file.txt # lines 10-20
# BEGIN and END blocks
awk 'BEGIN {print "Start"} {sum += $1} END {print sum}' nums.txt
# field separator and output
awk -F, 'BEGIN {OFS="|"} {print $1, $2}' csv.txt
# count lines matching a pattern
awk '/error/' file.txt # like grep
awk '/error/ {count++} END {print count}' log
# built-in variables: NR (row), NF (fields), FS, OFS
awk '{print NR, NF, $0}' file.txtcut, tr, sort & uniq
cut is simple field/character extraction — fast but limited (no quoting support). tr translates characters (great for case conversion or delimiter swapping) and -d deletes. sort orders lines (-n numeric, -r reverse, -k field). uniq only removes ADJACENT duplicates, so always pipe through sort first. uniq -c with sort -rn is the classic pattern for frequency analysis: 'sort | uniq -c | sort -rn' shows the most common lines.
# cut: extract fields or characters
cut -d: -f1 /etc/passwd # field 1, delimiter :
cut -c1-5 file.txt # characters 1-5
cut -d, -f2,3 data.csv # fields 2 and 3
# tr: translate or delete characters
echo "Hello" | tr 'a-z' 'A-Z' # HELLO
echo "a,b,c" | tr ',' ' ' # a b c
echo "hello" | tr -d 'l' # heo (delete)
tr -s ' ' < file.txt # squeeze repeated spaces
# sort lines
sort file.txt # alphabetical
sort -n nums.txt # numeric
sort -rn nums.txt # reverse numeric
sort -t: -k3 -n /etc/passwd # by 3rd field, numeric
sort -u file.txt # sort + unique
# uniq: filter or count adjacent duplicates
sort file.txt | uniq # unique lines
sort file.txt | uniq -c # count occurrences
sort file.txt | uniq -d # only duplicates
sort file.txt | uniq -u # only unique linesPipes & Redirection
Pipes connect stdout to stdin, building powerful pipelines. > redirects stdout (>> appends), 2> redirects stderr, &> captures both. Here-docs (<<EOF) feed multi-line strings; quoting the delimiter ('EOF') disables variable expansion. Process substitution <(cmd) treats a command's output as a temporary file — essential for feeding while loops without a subshell, and for commands like diff that expect files.
# pipe: chain commands (stdout of one -> stdin of next)
cat file | grep "x" | sort | uniq -c | sort -rn | head
# redirect stdout
echo "hello" > file.txt # overwrite
echo "world" >> file.txt # append
# redirect stderr
command 2> error.log # stderr to file
command 2>&1 # stderr to stdout
command > all.log 2>&1 # both to same file
command &> all.log # both (Bash shortcut)
# stdin from file
command < input.txt
# here-string
grep "x" <<< "some text with x"
# here-document (multi-line input)
cat << EOF
Line 1
Line 2 with $VAR expansion
EOF
# quoted delimiter disables expansion
cat << 'EOF'
Literal $VAR no expansion
EOF
# process substitution
diff <(ls dir1) <(ls dir2) # compare outputs
while read line; do echo "$line"; done < <(grep x file)File & Directory Operations
find — File Search
find is the most powerful file search tool. -name matches globs (use -iname for case-insensitive); -type filters by f/d/l; -mtime/-mmin filter by modification time (- = within, + = older than); -size filters by size. -exec runs a command on each result; {} is the filename, \; runs per file, + batches them. -delete is safer than -exec rm for deletion. Always test find without -delete first!
# search by name
find . -name "*.js"
find . -iname "*.JS" # case-insensitive
find / -name "*.conf" 2>/dev/null # suppress permission errors
# search by type
find . -type f -name "*.txt" # files only
find . -type d -name src # directories only
find . -type l # symlinks
# search by time and size
find . -mtime -7 # modified < 7 days ago
find . -mtime +30 # modified > 30 days ago
find . -mmin -60 # modified < 60 min ago
find . -size +10M # larger than 10MB
find . -size -1k # smaller than 1KB
find . -empty # empty files/dirs
# act on results (-exec)
find . -name "*.log" -exec rm {} \;
find . -name "*.js" -exec wc -l {} +
find . -type f -exec chmod 644 {} \;
# safe deletion with -delete
find /tmp -type f -mtime +7 -deletePermissions & Ownership
Permissions are three triplets: owner, group, others. Each digit is r(4)+w(2)+x(1), so 755 = rwxr-xr-x. chmod changes permissions; chown changes owner/group. Symbolic notation (u+x, g-w) is clearer for incremental changes. umask sets default permissions for new files (subtracted from 666 for files, 777 for dirs). For web servers, 755 for dirs and 644 for files is standard; never use 777 in production.
# view permissions
ls -l file.txt
# -rw-r--r-- 1 user group 1024 Jan 1 10:00 file.txt
# ^^^ ^^^ ^^^
# owner group others
# chmod: change permissions
chmod 755 script.sh # rwxr-xr-x
chmod 644 file.txt # rw-r--r--
chmod +x script.sh # add execute for all
chmod u+x,g-w file.txt # symbolic: user +x, group -w
chmod -R 755 directory/ # recursive
# chown: change owner
chown alice file.txt
chown alice:developers file.txt
chown -R alice:developers project/
# umask: default permissions for new files
umask # show current (e.g. 022)
umask 077 # new files: 600, dirs: 700
# numeric permission reference:
# 7=rwx 6=rw 5=rx 4=r 3=wx 2=w 1=x 0=noneCopy, Move, Remove & Link
cp -r copies directories recursively; -i prompts before overwriting (safer); -p preserves timestamps and permissions. mv is both move and rename. rm -rf is dangerous — it deletes recursively without prompting; always double-check the path. Hard links point to the same inode (same file, can't cross filesystems or link dirs); symlinks are path references (can link anything, but break if target moves). Use ln -s for symlinks.
# copy
cp file.txt backup.txt
cp -r src/ dest/ # recursive (directories)
cp -i file.txt dest/ # interactive (prompt before overwrite)
cp -u file.txt dest/ # update (only if newer)
cp -p file.txt dest/ # preserve attributes
# move / rename
mv old.txt new.txt
mv file.txt /other/dir/
mv -i file.txt dest/ # prompt before overwrite
# remove
rm file.txt
rm -r directory/ # recursive
rm -f file.txt # force (no error if missing)
rm -rf node_modules/ # force + recursive (DANGER)
# links
ln target hardlink # hard link (same inode)
ln -s target symlink # symbolic link (path reference)
ln -sf target symlink # force recreate symlink
# view link target
readlink symlink
readlink -f symlink # canonical absolute pathArchives & Compression
tar bundles files into one archive; gzip/bzip2/xz compress it. The flags: c=create, x=extract, t=list, f=file, z=gzip, j=bzip2, J=xz, v=verbose. .tar.gz is the Unix standard; .zip is common on Windows. bzip2 compresses better than gzip but slower; xz is best but slowest. Use -C to extract to a specific directory. The -k flag preserves the original when compressing.
# tar + gzip (most common)
tar -czf archive.tar.gz dir/ # create
tar -xzf archive.tar.gz # extract
tar -xzf archive.tar.gz -C /opt/ # extract to /opt
tar -tzf archive.tar.gz # list contents
# tar + bzip2 (better compression)
tar -cjf archive.tar.bz2 dir/
tar -xjf archive.tar.bz2
# zip / unzip
zip -r archive.zip dir/
unzip archive.zip
unzip archive.zip -d /target
# gzip / gunzip (single files)
gzip file.txt # -> file.txt.gz
gunzip file.txt.gz # -> file.txt
gzip -k file.txt # keep original
# xz (best compression)
tar -cJf archive.tar.xz dir/
tar -xJf archive.tar.xz
# mnemonic: eXtract, Create, List, File, gZip, bZip2, xzFile Content & Comparison
cat dumps the whole file; head/tail show ends; tail -f streams live updates (essential for logs). less is an interactive pager with search (/) and navigation. diff compares files; -u produces the unified format used by patch. comm compares two sorted files and shows lines unique to each or common to both — useful for comparing lists. Always sort inputs before comm, as it requires sorted files.
# view file contents
cat file.txt # entire file
head -n 20 file.txt # first 20 lines
tail -n 20 file.txt # last 20 lines
tail -f app.log # follow (live updates)
less file.txt # pager (q to quit, / to search)
# line/word/char count
wc file.txt # lines words bytes
wc -l file.txt # lines only
wc -w file.txt # words only
# compare files
diff file1.txt file2.txt
diff -u file1.txt file2.txt # unified format (for patches)
diff -r dir1/ dir2/ # recursive
# patch
diff -u old new > change.patch
patch < change.patch
patch -R < change.patch # reverse
# find duplicate lines across files
comm -12 <(sort a) <(sort b) # lines in both
comm -23 <(sort a) <(sort b) # only in a
comm -13 <(sort a) <(sort b) # only in bProcess Management & Signals
Background Jobs & Job Control
Appending & runs a command in the background, returning immediately. jobs lists active jobs; fg/bg move them between foreground and background. Ctrl+Z suspends the foreground job (sends SIGTSTP). wait blocks until background jobs finish — essential for scripts that launch parallel work. disown removes a job from the shell's job table so it keeps running after you log out (unlike nohup, it works on already-running jobs).
# run in background (append &)
sleep 100 &
# [1] 12345 (job 1, PID 12345)
# list jobs
jobs
jobs -l # with PIDs
# bring to foreground / send to background
fg # foreground the current job
fg %1 # foreground job 1
bg %2 # background job 2 (continue running)
# suspend a running foreground job: Ctrl+Z
# then resume in background:
bg
# wait for background jobs
sleep 5 &
wait # wait for all background jobs
wait $! # wait for the last background job
wait 12345 # wait for specific PID
# disown: detach from shell (survives logout)
long_task &
disown %1kill & Signals
kill sends signals to processes. SIGTERM (default) asks the process to exit gracefully — it can clean up. SIGKILL (-9) is forceful and immediate; the process cannot catch or ignore it, so it may leave resources in a bad state. Always try SIGTERM first, wait, then SIGKILL only if necessary. killall/pkill kill by name. pkill -f matches the full command line (more flexible). Use kill -l to list all signal names.
# send signals to processes
kill 12345 # SIGTERM (graceful, default)
kill -15 12345 # SIGTERM explicitly
kill -9 12345 # SIGKILL (force, cannot be caught)
kill -HUP 12345 # SIGHUP (reload config)
# kill by name
killall nginx # kill all nginx processes
pkill -f "python app.py" # match full command line
# common signals
# SIGTERM (15) - graceful termination (default)
# SIGKILL (9) - force kill (cannot be caught/ignored)
# SIGINT (2) - interrupt (Ctrl+C)
# SIGHUP (1) - hangup (often reload config)
# SIGSTOP (19) - pause (cannot be caught)
# SIGCONT (18) - resume
# SIGTSTP (20) - terminal stop (Ctrl+Z)
# list all signals
kill -l
# priority: try SIGTERM first, SIGKILL as last resort
kill 12345; sleep 2; kill -9 12345 2>/dev/nulltrap — Signal Handling
trap registers commands to run when the script receives a signal — essential for cleanup. The EXIT pseudo-signal fires on any exit (normal, error, or killed), making it perfect for removing temp files. Always set traps before creating the resources they clean up. Common pattern: trap cleanup EXIT INT TERM. trap '' SIGNAL ignores it; trap - SIGNAL restores default. This is how robust scripts ensure cleanup even when interrupted.
# clean up on exit (EXIT signal)
trap 'rm -f $TMPFILE' EXIT
TMPFILE=$(mktemp)
echo "working with $TMPFILE"
# when script exits (normally or via error), TMPFILE is removed
# handle Ctrl+C (SIGINT)
trap 'echo "Caught Ctrl+C, exiting"; exit 130' INT
# handle multiple signals
cleanup() {
echo "Cleaning up..."
rm -rf "$WORKDIR"
exit
}
trap cleanup INT TERM EXIT
# ignore a signal
trap '' INT # ignore Ctrl+C
trap - INT # restore default behavior
# reload on SIGHUP
trap 'echo "reloading config"; source config.sh' HUP
# list current traps
trap -pProcess Inspection
ps shows a snapshot of processes; top/htop show real-time. pstree displays the parent-child hierarchy. pgrep finds PIDs by name (safer than ps|grep). lsof -i :PORT finds which process uses a port — essential for debugging 'port in use' errors. ss (socket statistics) is the modern replacement for netstat. The ps aux | grep pattern is ubiquitous but pgrep is cleaner. Use --sort to find resource hogs.
# list processes
ps aux # all processes (BSD style)
ps -ef # all processes (System V style)
ps -ef | grep nginx # find nginx processes
# interactive viewer (like Task Manager)
top # real-time, all processes
htop # nicer interactive viewer (install separately)
# process tree
pstree # tree view
pstree -p # with PIDs
# find process by name/port
pgrep -f "node app" # print PIDs matching
pgrep -fl "node" # full command line
# who's listening on a port
lsof -i :8080 # process using port 8080
ss -tlnp # all listening TCP ports
netstat -tlnp # legacy alternative
# process resource usage
top -b -n 1 | head # snapshot
ps aux --sort=-%mem | head # top memory usersnohup, disown & tmux
nohup makes a process ignore SIGHUP so it survives logout — output goes to nohup.out. disown achieves the same for an already-running background job. setsid starts a process in a new session, fully detaching it. For long-running interactive work, tmux or screen are better: they keep a full terminal session alive that you can reattach to later, so even text editors survive disconnection. This is how sysadmins manage remote servers.
# nohup: run immune to hangups (survives logout)
nohup ./long_script.sh &
# output goes to nohup.out by default
nohup ./server.sh > server.log 2>&1 &
# disown: detach an already-running job
./long_task &
disown %1
# setsid: start in a new session
setsid ./daemon.sh &
# tmux: terminal multiplexer (persistent sessions)
tmux # start new session
tmux new -s work # named session
tmux ls # list sessions
tmux attach -t work # reattach
# inside tmux: Ctrl+B then D to detach
# processes keep running after detach
# screen: alternative to tmux
screen -S work
screen -ls
screen -r work
# check if a process is running
pgrep -x nginx && echo "running" || echo "stopped"Scripting & Advanced Topics
Shebang & Script Structure
The shebang (#!) tells the kernel which interpreter to use. #!/usr/bin/env bash is most portable. A well-structured script starts with 'set -euo pipefail' for safety, defines usage(), and parses arguments with getopts (for short flags) or a manual loop (for long flags). OPTIND tracks the next argument; shift past parsed options so $1 is the first positional argument. This structure makes scripts robust and user-friendly.
#!/usr/bin/env bash
# ^ portable shebang (finds bash via PATH)
# #!/bin/bash # absolute path (common)
# #!/usr/bin/bash # on some systems
# #!/bin/sh # POSIX sh (most portable)
set -euo pipefail # strict mode (see below)
# script metadata
readonly PROGNAME=$(basename "$0")
readonly VERSION="1.0.0"
usage() {
cat << EOF
Usage: $PROGNAME [OPTIONS] FILE
Options:
-h, --help Show this help
-v, --version Show version
-o OUTPUT Output file
EOF
}
# parse options with getopts
while getopts "hvo:" opt; do
case $opt in
h) usage; exit 0 ;;
v) echo $VERSION; exit 0 ;;
o) output=$OPTARG ;;
*) usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))Strict Mode (set -euo pipefail)
set -e exits immediately if any command fails (returns non-zero) — catches errors early. set -u treats references to unset variables as errors. pipefail makes a pipeline return non-zero if ANY command in it fails (by default, only the last command's status matters). Together they catch most bugs. Use 'cmd || true' to allow expected failures, and 'if ! cmd' for explicit checks. Some commands (grep, test) return non-zero legitimately, so wrap them.
# the 'strict mode' trio
set -e # exit on error (any non-zero status)
set -u # error on unset variable
set -o pipefail # pipeline fails if any command fails
# combined: set -euo pipefail
# example: without -e, errors are silently ignored
mkdir /root/nope # fails (permission denied)
echo "still running" # this runs anyway!
# with set -e, the script exits at the failure
set -e
mkdir /root/nope # script exits here
echo "never reached"
# handle expected failures
set -e
if ! grep -q "x" file; then
echo "not found" # grep returning 1 won't kill script
fi
# temporarily disable -e
set +e
risky_command
set -e
# || true to ignore a command's failure
mkdir -p dir || trueDebugging Techniques
set -x (xtrace) prints each command before execution — the #1 debugging tool. PS4 customizes the prompt (adding line numbers helps locate issues). bash -n checks syntax without running. trap ERR runs a command when any error occurs, perfect for logging where things went wrong. For complex scripts, wrap suspicious functions with set -x/+x to trace just that part. Combine with set -e to stop at the first error and inspect.
# trace execution: print each command before running
set -x # turn on
set +x # turn off
bash -x script.sh # run with tracing from the start
# verbose: print input lines as read
set -v
# combine: -xv for full detail
bash -xv script.sh
# print commands with line numbers
export PS4='+ ${BASH_SOURCE}:${LINENO}: '
set -x
# check syntax without executing
bash -n script.sh # syntax check only
# step through interactively
bash -x script.sh 2>&1 | less
# debug a specific function
debug_func() {
set -x
actual_func "$@"
set +x
}
# use trap ERR to catch errors
trap 'echo "Error on line $LINENO" >&2' ERRArithmetic & Math
Bash only does integer arithmetic with $(( )). For floating-point, pipe expressions to bc (scale=N sets decimal places) or use awk. bc -l loads the math library (sqrt, sin, cos, etc.). $RANDOM gives a pseudo-random integer 0-32767; for cryptographic randomness use /dev/urandom. The (( )) command supports +=, -=, *=, /=, %= and ++/-- like C. For serious math, Python or awk are better choices.
# integer arithmetic
echo $((5 + 3)) # 8
echo $((10 / 3)) # 3 (integer division)
echo $((10 % 3)) # 1 (modulo)
echo $((2 ** 8)) # 256 (exponent)
# floating point with bc
echo "scale=2; 10/3" | bc # 3.33
echo "3.14 * 2" | bc # 6.28
result=$(echo "scale=4; $a / $b" | bc)
# floating point with awk
awk 'BEGIN {printf "%.2f\n", 10/3}' # 3.33
awk "BEGIN {print $a * $b}"
# math functions with bc -l
echo "scale=4; s(0)" | bc -l # 0 (sine, radians)
echo "scale=4; sqrt(16)" | bc -l # 4.0000
# random numbers
echo $RANDOM # 0-32767
echo $((RANDOM % 100)) # 0-99
head -c 4 /dev/urandom | od -An -tu4 # 32-bit random
# increment/decrement
n=5
((n++)); echo $n # 6
((n+=10)); echo $n # 16Regex & Pattern Matching
Bash has three pattern systems: globs (*.txt, ?, [abc]) for filenames, extended globs (extglob: !(), @(), *()) for more complex matching, and ERE regex (=~ in [[ ]]). =~ captures groups into BASH_REMATCH (index 0 = full match, 1+ = groups). Regex uses ERE syntax (like grep -E). Enable extglob for powerful negation and alternation in filename patterns. Regex is one of Bash's most useful features for input validation.
# glob patterns (filename matching)
*.txt # any .txt file
file?.txt # single char: file1.txt, fileA.txt
file[0-9].txt # digit: file0.txt ... file9.txt
file[!0-9].txt # NOT a digit
# extended globs (shopt -s extglob)
shopt -s extglob
echo !(*.bak) # everything except .bak files
echo @(*.jpg|*.png) # jpg or png
echo *(foo) # zero or more "foo"
# regex with =~ in [[ ]]
if [[ $email =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ]]; then
echo "valid email"
fi
# capture groups with BASH_REMATCH
if [[ $date =~ ^([0-9]{4})-([0-9]{2})-([0-9]{2})$ ]]; then
echo "Year: ${BASH_REMATCH[1]}"
echo "Month: ${BASH_REMATCH[2]}"
echo "Day: ${BASH_REMATCH[3]}"
fi
# case-insensitive regex
shopt -s nocasematch
if [[ $s =~ ^hello ]]; then echo "starts with hello"; figrep Deep Dive
Basic grep Patterns
grep searches text using patterns. -i ignores case; -w matches whole words (prevents matching 'errors' when searching 'error'); -v inverts (non-matching lines); -c counts; -n shows line numbers; -l lists only filenames with matches; -h suppresses filename prefix when searching multiple files. By default, grep uses Basic Regular Expressions (BRE) where metacharacters need backslash escaping. Use -E for Extended Regex (cleaner syntax) or -F for fixed strings (no regex, faster for literal searches).
# Basic search (literal string)
grep "error" logfile.txt
# Case-insensitive
grep -i "error" logfile.txt
# Whole word match
grep -w "error" logfile.txt
# Invert match (lines NOT matching)
grep -v "debug" logfile.txt
# Count matches
grep -c "error" logfile.txt
# Show line numbers
grep -n "error" logfile.txt
# Multiple files with filename prefix
grep "error" *.log
# Suppress filename prefix (single file behavior)
grep -h "error" *.log
# Only show filenames with matches
grep -l "error" *.loggrep with Regex
grep -E (or egrep) uses Extended Regular Expressions with cleaner syntax: +, ?, |, () work without escaping. ^ and $ anchor to start/end of line. [] defines character classes; {} specifies repetition. -P enables Perl-compatible regex (PCRE) with features like \d, \w, lookaheads — but this is GNU-specific and not portable. For complex regex, consider ripgrep (rg) which is faster and uses PCRE-like syntax by default. Always quote patterns to prevent shell interpretation of special characters like $ and *.
# Extended regex (-E or egrep)
grep -E "error|warning|fatal" logfile.txt
# Anchors: ^ start, $ end
grep -E "^ERROR:" logfile.txt # lines starting with ERROR:
grep -E "completed$" logfile.txt # lines ending with completed
# Character classes
grep -E "[0-9]{4}-[0-9]{2}-[0-9]{2}" dates.txt # date format
grep -E "[A-Z][a-z]+" names.txt # Capitalized words
# Quantifiers
grep -E "ab+c" file.txt # one or more b's
grep -E "ab*c" file.txt # zero or more b's
grep -E "ab?c" file.txt # zero or one b
# Alternation and grouping
grep -E "(cat|dog) food" file.txt
# PCRE (-P, GNU grep only)
grep -P "\d{3}-\d{4}" phones.txt # \d for digitsgrep Context & Output Control
Context flags (-B, -A, -C) show surrounding lines, essential for understanding log entries. -o outputs only the matched portion (useful for extracting URLs, numbers, etc.). --color highlights matches in terminal. -r searches recursively (excludes symlinks); -R follows symlinks. --include/--exclude/--exclude-dir filter which files to search — invaluable for large codebases (always exclude node_modules, .git, vendor). -a forces binary files to be treated as text. For code search, ripgrep (rg) is a modern, faster alternative with sensible defaults.
# Show context: lines before/after match
grep -B 2 "error" log.txt # 2 lines Before
grep -A 3 "error" log.txt # 3 lines After
grep -C 2 "error" log.txt # 2 lines Context (both)
# Only show matched part (not full line)
grep -o "https?://[^ ]+" urls.txt
# Color highlight matches
grep --color=auto "error" log.txt
# Recursive search
grep -r "TODO" ./src/
grep -R "TODO" ./src/ # follows symlinks
# Include/exclude file patterns
grep -r --include="*.py" "import" .
grep -r --exclude="*.test.js" "TODO" .
grep -r --exclude-dir=node_modules "TODO" .
# Binary files
grep -a "pattern" binary_file # treat as textgrep with stdin & Pipelines
grep is most powerful in pipelines. The [n]ginx trick prevents grep from matching its own process: the bracket makes the pattern not match the literal 'grep' string in the process list. zgrep searches compressed files without manual decompression. pgrep is a specialized process finder (better than ps | grep). grep -rl lists files containing a pattern; piping to xargs grep searches within those files for another pattern — a common code archaeology technique. For interactive code search, use ripgrep or ack which are designed for source code.
# Search command output
ps aux | grep nginx
ps aux | grep "[n]ginx" # trick: prevents matching grep itself
# Search compressed logs
zcat log.gz | grep "error"
zgrep "error" log.gz # direct
# Chain multiple greps
cat log.txt | grep "error" | grep -v "timeout" | grep -c
# Extract and filter
grep -oE "[0-9.]+" response.txt | sort -n | uniq
# Find processes excluding grep
pgrep -f "node server.js" # better than ps | grep
# Search command history
history | grep "git rebase"
# Find files containing pattern, then search more
grep -rl "config" . | xargs grep "database"grep Exit Status & Scripting
grep's exit status makes it ideal for scripting: 0 (match found), 1 (no match), 2 (error). -q (quiet) suppresses output for pure conditional checks. In set -e scripts, grep returning 1 (no match) would exit the script — use '|| true' to prevent this. The while-read pattern processes matching lines one at a time. grep -c returns a count (0 if no matches), which you can compare numerically. This scripting capability makes grep a building block for log monitoring, validation scripts, and CI/CD checks.
# grep returns exit codes for scripting
# 0 = match found, 1 = no match, 2 = error
if grep -q "error" /var/log/syslog; then
echo "Errors found!"
# send alert, etc.
fi
# Silent check (-q), no output
grep -q "^$" file.txt && echo "has blank lines"
# Use in while loop
grep "pattern" file.txt | while read -r line; do
echo "Found: $line"
done
# Count and branch
errors=$(grep -c "ERROR" log.txt)
if [ "$errors" -gt 10 ]; then
echo "Too many errors: $errors"
fi
# grep with set -e (exits on non-zero)
set -e
grep "missing" file.txt || true # prevent exitsed Deep Dive
sed Substitution
sed (stream editor) transforms text line by line. The s command substitutes text: s/pattern/replacement/flags. g (global) replaces all occurrences per line; without it, only the first match is replaced. You can restrict operations to specific lines (by number, range, or pattern). -i edits files in-place (dangerous — always test without -i first, or use -i.bak for a backup). sed processes each line independently. The delimiter doesn't have to be / — use s|old|new|g when patterns contain slashes (e.g., file paths).
# Basic substitution (first occurrence per line)
sed 's/old/new/' file.txt
# Replace all occurrences
sed 's/old/new/g' file.txt
# Case-insensitive (GNU sed)
sed 's/old/new/gi' file.txt
# Replace Nth occurrence
sed 's/old/new/2' file.txt # 2nd occurrence only
# Only on lines matching pattern
sed '/error/s/old/new/' file.txt
# Only on specific line numbers
sed '5s/old/new/' file.txt # line 5 only
sed '5,10s/old/new/g' file.txt # lines 5-10
sed '$s/old/new/' file.txt # last line
# In-place edit (modifies file)
sed -i 's/old/new/g' file.txt
sed -i.bak 's/old/new/g' file.txt # keeps backupsed with Regex & Capture Groups
sed -E (or -r) uses Extended Regular Expressions with cleaner syntax (no backslash before parentheses/braces). Capture groups are referenced as \1, \2, etc. in the replacement. & represents the full match. The date reformatting example shows the power: capture year, month, day separately and rearrange. Multiple commands can be chained with semicolons (s/.../.../;s/.../.../). Always escape special characters in patterns (., *, [, etc.) and escape / in replacements (or use a different delimiter).
# Extended regex (-E or -r)
sed -E 's/[0-9]+/NUMBER/g' file.txt
# Capture groups with \1, \2 (BRE)
sed 's/\(foo\)\(bar\)/\2\1/' file.txt # swap: foobar -> barfoo
# Capture groups with () (ERE, cleaner)
sed -E 's/(foo)(bar)/\2\1/' file.txt
# Reformat date: 2025-01-15 -> 15/01/2025
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/' file.txt
# Add prefix to lines
sed 's/^/>> /' file.txt
# Add suffix
sed 's/$/ <<</' file.txt
# Remove leading/trailing whitespace
sed 's/^[ \t]*//;s/[ \t]*$//' file.txt
# Use & for full match
sed 's/[0-9]\{3\}/[&]/g' file.txt # wrap 3 digits in bracketssed Delete & Print
The d command deletes lines; p prints lines. With -n (no auto-print), only explicit p commands produce output — this turns sed into a selective printer (like grep). Deleting blank lines (sed '/^$/d') is a common cleanup. sed -n '5,10p' is equivalent to sed -n '5,10p' or head/tail combinations. The ~ syntax (0~3p) prints every 3rd line (GNU extension). Remember: without -n, sed prints every line (possibly modified); with -n, it prints nothing unless you use p. This duality makes sed both an editor and a filter.
# Delete lines
sed '5d' file.txt # delete line 5
sed '5,10d' file.txt # delete lines 5-10
sed '/pattern/d' file.txt # delete matching lines
sed '/^$/d' file.txt # delete blank lines
sed '/^[ \t]*$/d' file.txt # delete whitespace-only lines
sed '$d' file.txt # delete last line
# Print only specific lines (like head/tail)
sed -n '5p' file.txt # print line 5 only
sed -n '5,10p' file.txt # print lines 5-10
sed -n '/pattern/p' file.txt # print matching lines (like grep)
sed -n '5,${/pattern/p}' file.txt # from line 5 to end
# Print with line numbers
sed '=' file.txt | sed 'N;s/\n/\t/'
# Print every Nth line
sed -n '0~3p' file.txt # every 3rd line (GNU)sed Multi-line & Hold Space
sed's hold space allows multi-line operations — the pattern space (current line) can be saved to hold space and retrieved later. h/H copies/appends to hold; g/G copies/appends from hold; x exchanges them. The line-joining trick (:a;N;$!ba;s/\n/ /g) reads the entire file into pattern space then replaces newlines. N appends the next line to pattern space. These advanced features make sed Turing-complete but also very cryptic. For complex multi-line transformations, awk or Perl are more readable. Use sed for simple line-based edits; use awk for field-based or multi-line logic.
# Join lines (replace newline with space)
sed ':a;N;$!ba;s/\n/ /g' file.txt
# Join every 2 lines
sed 'N;s/\n/ /' file.txt
# Print paragraph (blocks separated by blank lines)
sed '/./{H;d;};x;s/\n/ /g' file.txt
# Reverse line order (tac alternative)
sed '1!G;h;$!d' file.txt
# Double-space a file
sed 'G' file.txt
# Remove last line of each paragraph
sed -n '/^$/{p;h;};/./{H;};/^$/{x;s/\n.*//;p;}' file.txt
# Hold space (h,H,g,G,x) stores lines between cycles
# h: copy pattern space to hold
# H: append pattern space to hold
# g: copy hold to pattern space
# G: append hold to pattern space
# x: exchange pattern and holdsed in Pipelines & Scripts
sed excels in pipelines for text transformation. When substituting variables, escape special characters (& and \) to prevent interpretation. Multiple -e flags apply multiple commands in sequence. -f reads commands from a file (useful for complex scripts). The CSV-to-TSV conversion shows a practical use. For config file editing in scripts, always validate the change (grep after sed) and consider using dedicated tools (jq for JSON, yq for YAML). sed's strength is simple, fast, line-based edits — it's a staple of Unix text processing alongside grep and awk.
# Clean up output in pipeline
cat messy.txt | sed 's/[ \t]*$//' | sed '/^$/d' | sort
# Extract and transform
echo "name=John;age=30" | sed 's/;/\n/g' | sed 's/=/: /'
# Modify config files in scripts
sed -i "s/PORT=8080/PORT=$NEW_PORT/" config.env
# Escape special chars for safe substitution
ESCAPED=$(printf '%s\n' "$URL" | sed 's/[&\]/\\&/g')
sed "s|URL|$ESCAPED|g" template.txt
# Multiple substitutions
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt
# Read commands from file
sed -f script.sed input.txt
# Convert CSV to TSV
sed 's/,/\t/g' data.csvawk Deep Dive
awk Basics & Fields
awk automatically splits each line into fields ($1, $2, ..., $NF). -F sets the input field separator; OFS sets the output separator. $0 is the entire line. NR (Number of Records) is the line number; NF (Number of Fields) is the field count for the current line. awk processes each line through pattern-action pairs: pattern { action }. If no pattern, the action runs for every line. If no action, the line is printed. This makes awk ideal for column-based data extraction from CSV, TSV, /etc/passwd, and log files.
# Print specific fields (default delimiter: whitespace)
awk '{print $1}' file.txt # first field
awk '{print $1, $3}' file.txt # first and third
awk '{print $NF}' file.txt # last field
awk '{print $(NF-1)}' file.txt # second-to-last
# Custom field separator
awk -F: '{print $1, $7}' /etc/passwd # colon-separated
awk -F',' '{print $2}' data.csv # CSV
awk -F'\t' '{print $1}' data.tsv # TSV
# Print entire line ($0)
awk '{print}' file.txt
awk '{print NR, $0}' file.txt # with line number
# NR = record (line) number
# NF = number of fields in current record
# NR and NF are built-in variables
# Change output separator
awk 'BEGIN{OFS="|"} {print $1, $2, $3}' file.txtawk Patterns & Conditions
awk patterns can be regex (/pattern/), comparisons ($3 > 100), line numbers (NR == 5), or ranges (/start/,/end/). ~ and !~ apply regex to specific fields. Conditions can be combined with &&, ||, and !. This makes awk a powerful filter — more expressive than grep because you can compare field values numerically or as strings. The range pattern (/start/,/end/) prints all lines between two markers (inclusive). awk evaluates conditions per line; if true, the action runs. Without an action, the default is {print}.
# Pattern: only process matching lines
awk '/error/ {print}' log.txt # like grep
awk '$3 > 100' data.txt # 3rd field > 100
awk 'NR == 5' file.txt # line 5 only
awk 'NR >= 10 && NR <= 20' file.txt # lines 10-20
awk 'NF == 0' file.txt # empty lines
awk '$1 == "ERROR"' log.txt # first field equals ERROR
# Range pattern: from start to end
awk '/start/,/end/' file.txt
# Combine conditions
awk '$3 > 50 && $3 < 100 {print $1, $3}' data.txt
awk '$1 == "GET" || $1 == "POST" {print}' access.log
# Negation
awk '!/debug/ {print}' log.txt # lines NOT containing debug
# Field comparison with regex
awk '$2 ~ /^[0-9]+$/' file.txt # 2nd field is numeric
awk '$2 !~ /pattern/' file.txt # 2nd field doesn't matchawk BEGIN/END & Variables
BEGIN runs before any input is read (ideal for initialization, headers, variable setup). END runs after all input is processed (ideal for summaries, totals). User variables don't need declaration — they default to 0 (numeric) or empty (string). -v passes external variables into awk. This makes awk a mini programming language for data processing: sum, average, min, max, count — all in one pass. The pattern 'NR == 1 {max = $1}' initializes max from the first line, then subsequent lines update it. This is far more efficient than multiple grep/sort/cut pipelines.
# BEGIN: runs before processing (setup)
# END: runs after processing (summary)
awk 'BEGIN {print "Total:"} {sum += $1} END {print sum}' nums.txt
# Count lines
awk 'END {print NR " lines"}' file.txt
# Sum a column
awk '{sum += $2} END {print "Total:", sum}' sales.csv
# Average
awk '{sum += $2; count++} END {print sum/count}' data.txt
# Find max value
awk 'NR == 1 {max = $1} $1 > max {max = $1} END {print max}' nums.txt
# User-defined variables (no declaration needed)
awk '{total += $1; if ($1 > max) max = $1}
END {print "Sum:", total, "Max:", max, "Avg:", total/NR}'
# Pass variables with -v
awk -v threshold=50 '$1 > threshold {print}' data.txt
awk -v date="$(date +%Y-%m-%d)" '{print date, $0}' file.txtawk Control Flow
awk has full control flow: if/else, for, while, and associative arrays (key-value). This makes it a complete data processing language. The word-count example (words[$1]++) is the classic awk use case — count occurrences of each value in a column, then sort by frequency. Arrays in awk are associative (like dictionaries), indexed by string or number. for (key in array) iterates keys (unordered — pipe through sort). awk can replace entire pipelines of cut, sort, uniq, and wc with a single, efficient pass. For complex data processing, awk is often clearer than chained Unix commands.
# if-else
awk '{
if ($3 >= 90) grade = "A"
else if ($3 >= 80) grade = "B"
else if ($3 >= 70) grade = "C"
else grade = "F"
print $1, grade
}' students.txt
# for loop
awk 'BEGIN {
for (i = 1; i <= 5; i++) print i, i*i
}'
# for-in (iterate array)
awk '{count[$1]++} END {
for (key in count) print key, count[key]
}' access.log | sort -rnk2
# while loop
awk '{
i = 1
while (i <= NF) {
print i, $i
i++
}
}' file.txt
# Arrays (associative)
awk '{words[$1]++} END {
for (w in words) print words[w], w
}' file.txt | sort -rn | headawk Practical Examples
These practical examples show awk's real-world power. The IP frequency count is essential for log analysis. The file-size-by-extension summary uses split() to extract extensions. CSV filtering by column value replaces complex grep/sed pipelines. Percentile calculation demonstrates awk's mathematical capabilities (asorti sorts array indices). Reformatting columns (changing delimiter and selecting fields) is a common ETL task. awk is the go-to tool for structured text processing — when data has columns/fields, awk is almost always the right choice. For JSON, use jq; for CSV, awk or csvkit.
# Top 10 IP addresses from access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# Or entirely in awk:
awk '{ip[$1]++} END {for (i in ip) print ip[i], i}' access.log | sort -rn | head
# Sum file sizes by extension
ls -l | awk '{split($NF, a, "."); ext=a[2]; size[ext]+=$5}
END {for (e in size) print e, size[e]}' | sort -rnk2
# Filter CSV by column value
awk -F',' '$3 == "active" {print $1, $2}' users.csv
# Calculate response time percentiles from log
awk '{times[NR] = $2} END {
n = asorti(times, sorted)
print "p50:", sorted[int(n*0.5)]
print "p95:", sorted[int(n*0.95)]
print "p99:", sorted[int(n*0.99)]
}' response_times.txt
# Reformat columns
awk -F',' 'BEGIN{OFS="|"} {print $1, $2, $4}' data.csvfind & xargs
find by Name & Type
find searches the filesystem by various criteria. -name matches the filename (case-sensitive); -iname is case-insensitive. -type f/d/l filters by file type. -path matches the full path; -regex uses full-path regex. -o (or) combines conditions; use \( \) for grouping. -maxdepth limits recursion depth (important for performance on large filesystems). find outputs paths; combine with -exec or xargs to act on results. Always quote patterns to prevent shell glob expansion. For code search, ripgrep (rg) is faster but find is more flexible for filesystem operations.
# Find by name (exact)
find . -name "config.txt"
# Case-insensitive name
find . -iname "config.txt"
# Wildcard pattern
find . -name "*.py"
find . -name "test_*"
# Find by type
find . -type f -name "*.js" # files only
find . -type d -name "node_modules" # directories
find . -type l # symlinks
# Find by path (full path match)
find . -path "*/src/*.test.js"
# Find by regex
find . -regex ".*\.\(py\|js\)$"
# Multiple name patterns
find . \( -name "*.py" -o -name "*.js" \)
# Max depth (don't recurse too deep)
find . -maxdepth 2 -name "*.txt"find by Time & Size
find's time-based search is essential for cleanup and auditing. -mtime (modification), -atime (access), -ctime (metadata change) use days; -mmin/-amin/-cmin use minutes. - (less than) and + (more than) prefix the value. Size uses suffixes: c (bytes), k (KB), M (MB), G (GB). -empty finds zero-length files or empty directories. -perm checks permissions: exact match (644), all bits set (-u+x), or any bit set (/4000). Finding SUID files (/4000) is a security audit technique. These criteria can be combined with -a (and, default) and -o (or).
# Find by modification time
find . -mtime -1 # modified in last 24 hours
find . -mtime +7 # modified more than 7 days ago
find . -mtime 7 # modified exactly 7 days ago
# Access time (-atime) and change time (-ctime)
find . -atime -1 # accessed in last 24h
find . -ctime -1 # status changed in last 24h
# Minutes instead of days
find . -mmin -30 # modified in last 30 minutes
# Find by size
find . -size +100M # larger than 100MB
find . -size -1k # smaller than 1KB
find . -size 10M # exactly 10MB
# Find empty files/directories
find . -empty -type f
find . -empty -type d
# Find by permissions
find . -perm 644 # exactly 644
find . -perm -u+x # has execute for user
find / -perm /4000 # SUID files (security audit)find -exec & -delete
-exec runs a command on each result. {} is the placeholder for the filename; \; ends the command (runs once per file); + batches files (runs once with all files — more efficient). -delete removes matched files (faster than -exec rm, but test first with -print). The chmod pattern (dirs 755, files 644) is a common web server setup. -exec with + is preferred for commands that accept multiple files (grep, ls, wc). For deletion, always run with -print first to verify what will be deleted, then replace with -delete. -ok instead of -exec prompts for confirmation per file.
# Execute command on each result
find . -name "*.log" -exec ls -lh {} \;
# Execute with + (batch, more efficient)
find . -name "*.log" -exec ls -lh {} +
# Delete files (DANGEROUS - test first!)
find /tmp -name "*.tmp" -type f -delete
# Safer: use -exec rm
find /tmp -name "*.tmp" -type f -exec rm -i {} \;
# Execute a complex command
find . -name "*.py" -exec wc -l {} \; | sort -rn | head
# Change permissions
find . -type d -exec chmod 755 {} \; # dirs to 755
find . -type f -exec chmod 644 {} \; # files to 644
# Find and grep
find . -name "*.py" -exec grep "TODO" {} +
# Print before deleting (confirmation)
find . -name "*.bak" -print -deletexargs Basics
xargs converts stdin into command arguments. It's the bridge between find output and commands that don't read stdin. -n limits arguments per command; -I {} defines a placeholder for custom positioning. -0 (with find -print0) handles filenames with spaces/newlines correctly — always use this pair for safety. -P enables parallel execution (great for CPU-bound tasks like image conversion). -t (trace) shows commands before running; -p prompts for confirmation. xargs is essential when -exec is too slow (one process per file) — xargs batches arguments efficiently.
# Basic: take stdin and pass as arguments
echo "file1 file2 file3" | xargs rm
# One argument per line
find . -name "*.bak" | xargs rm
# Control arguments per command (-n)
echo "1 2 3 4 5" | xargs -n 2 echo # echo 1 2; echo 3 4; echo 5
# Show commands before running (-t, verbose)
find . -name "*.log" | xargs -t rm
# Prompt before each command (-p)
find . -name "*.tmp" | xargs -p rm
# Handle filenames with spaces (-0 with find -print0)
find . -name "*.txt" -print0 | xargs -0 grep "pattern"
# Replace string (-I)
find . -name "*.py" | xargs -I {} cp {} /backup/
# Parallel execution (-P)
find . -name "*.png" | xargs -P 4 -I {} convert {} {}.thumb.png
# Limit with -L (lines per command)
cat urls.txt | xargs -L 1 curl -Ofind + xargs Patterns
find + xargs is the classic Unix pattern for batch file operations. -print0 | xargs -0 is the safe combination for filenames with spaces or special characters. The bulk grep pattern is much faster than -exec grep (one grep process vs many). Parallel xargs (-P N) dramatically speeds up CPU-bound tasks like audio/video conversion. The archive pattern (find old logs, tar them) is a common log rotation technique. Always use -print0/-0 for robustness — without it, filenames with spaces, quotes, or newlines will break the pipeline. This combination is fundamental to Unix system administration.
# Find and bulk grep (efficient)
find . -name "*.py" -print0 | xargs -0 grep -l "import django"
# Find large files and show sizes
find . -type f -size +100M -print0 | xargs -0 ls -lhS
# Find and archive
find . -name "*.log" -mtime +30 -print0 |
xargs -0 tar -czf old_logs.tar.gz
# Find and batch rename
find . -name "*.JPG" | xargs -I {} mv {} {}.bak
# Or with rename command:
find . -name "*.JPG" -exec rename 's/\.JPG$/.jpg/' {} +
# Find empty directories and remove
find . -type d -empty -print0 | xargs -0 rmdir
# Find recently modified files and copy
find . -name "*.py" -mtime -1 -print0 |
xargs -0 -I {} cp {} /tmp/recent/
# Parallel processing
find . -name "*.wav" -print0 |
xargs -0 -P 8 -I {} ffmpeg -i {} {}.mp3Advanced Bash Techniques
Here Document & Here String
Here documents (<< DELIMITER) feed multi-line text as stdin to a command — useful for generating config files, SQL queries, or any multi-line input. Unquoted delimiter allows variable expansion; quoted ('EOF') treats content literally. Here strings (<<<) feed a single string as stdin — cleaner than echo | command for simple cases. The delimiter can be any word (EOF, END, DONE); convention is uppercase. Here docs are essential for scripts that generate files or interact with interactive programs (mysql, psql, ssh). Indented delimiters (<<-) strip leading tabs for readability.
# Here document: multi-line input to a command
cat << EOF
Line 1
Line 2 with $VARIABLE expansion
Line 3
EOF
# Quoted delimiter: no variable expansion
cat << 'EOF'
This $won't expand
\n stays literal
EOF
# Write to file
cat > config.txt << EOF
host=localhost
port=8080
EOF
# Here string: single string as stdin
grep "pattern" <<< "search this text"
read -r line <<< "hello world"
# With variables
name="Alice"
greeting=$(cat <<< "Hello, $name!")
# Pipe multi-line to command
mysql << EOF
USE mydb;
SELECT * FROM users;
EOFParameter Expansion
Parameter expansion is Bash's built-in string manipulation — no need for sed/awk/cut for simple operations. :- provides defaults; # and ## strip prefixes; % and %% strip suffixes. The file extension examples are extremely common: ${file##*.} gets extension, ${file%.*} gets basename without extension. / replaces first match; // replaces all. ^^ and ,, convert case (Bash 4+). These operations are faster than spawning external commands. Master parameter expansion to write efficient, readable Bash scripts without unnecessary subshells.
# Default values
echo ${name:-"default"} # use "default" if name unset/empty
echo ${name:="default"} # set name to "default" if unset/empty
# String length
echo ${#var}
# Substring extraction
var="Hello World"
echo ${var:0:5} # Hello (start:length)
echo ${var:6} # World (start to end)
# Remove from beginning (# shortest, ## longest)
file="archive.tar.gz"
echo ${file#*.} # tar.gz (remove up to first .)
echo ${file##*.} # gz (remove up to last .)
# Remove from end (% shortest, %% longest)
echo ${file%.*} # archive.tar (remove from last .)
echo ${file%%.*} # archive (remove from first .)
# Replace
echo ${var/World/Bash} # Hello Bash (first match)
echo ${var//l/L} # HeLLo WorLd (all matches)
# Case conversion (Bash 4+)
echo ${var^^} # HELLO WORLD (uppercase)
echo ${var,,} # hello world (lowercase)trap & Signal Handling
trap registers handlers for signals and events. EXIT fires on script termination (normal, error, or killed) — ideal for cleanup (temp files, locks). INT (Ctrl-C), TERM (kill), HUP (terminal closed) are common signals. The temp-file pattern (trap 'rm -f' EXIT) ensures cleanup even if the script fails. DEBUG fires before every command — useful for tracing. Always quote the trap command (single quotes prevent immediate expansion). trap is essential for robust scripts: without it, temp files accumulate and locks may not release on failure. Always clean up after yourself.
# Clean up on exit (any reason)
cleanup() {
echo "Cleaning up..."
rm -f /tmp/myapp.lock
exit 0
}
trap cleanup EXIT
# Handle specific signals
trap 'echo "Interrupted!"; cleanup' INT TERM
trap 'echo "Ctrl-C pressed"' INT
# Ignore a signal
trap '' INT # ignore Ctrl-C
# Reset trap
trap - INT # restore default behavior
# Common pattern: temp file cleanup
TMPFILE=$(mktemp)
trap 'rm -f "$TMPFILE"' EXIT
echo "data" > "$TMPFILE"
# ... use TMPFILE ...
# File is removed when script exits (even on error)
# Debug trap (trace execution)
trap 'echo "DEBUG: $BASH_COMMAND"' DEBUGDebugging Bash Scripts
set -x traces execution (prints each command with + prefix) — the primary debugging tool. set -euo pipefail (strict mode) catches errors early: -e exits on non-zero, -u catches typos in variable names, pipefail makes pipes fail correctly (without it, only the last command's exit code matters). bash -n checks syntax without running. PS4 customizes the trace prefix (showing file:line is very helpful). trap ERR fires on any error (with -e), giving you line number context. For complex scripts, add set -x at suspicious sections rather than globally. Always use strict mode in production scripts.
# Trace execution: print each command before running
bash -x script.sh
# Or within script:
set -x # enable tracing
set +x # disable tracing
# Verbose: print input lines as read
bash -v script.sh
# Strict mode (recommended for all scripts)
set -euo pipefail
# -e: exit on error
# -u: error on undefined variable
# -o pipefail: pipe fails if any command fails
# -o pipefail: pipe fails if any command fails
# Print specific debug info
echo "DEBUG: var=$var" >&2 # to stderr
# Check syntax without executing
bash -n script.sh
# PS4: customize trace prompt
export PS4='+${BASH_SOURCE}:${LINENO}: '
bash -x script.sh
# Trap ERR for error handling
trap 'echo "Error on line $LINENO"' ERRcurl & wget
curl and wget are essential HTTP clients. curl is for API testing (supports all HTTP methods, headers, JSON). -d sends POST data; -H sets headers; -X specifies method; -L follows redirects; -O saves with remote filename; -s is silent (for scripting). wget is optimized for downloading (resumes with -c, recursive with -r). For REST API testing, curl with -H 'Content-Type: application/json' and -d for JSON body is the standard. -w '%{http_code}' extracts just the status code for scripting. For complex API testing, consider httpie (simpler syntax) or postman.
# Basic HTTP request
curl https://api.example.com/data
curl -O https://example.com/file.zip # save to file
# HTTP methods
curl -X POST https://api.example.com/users
curl -X PUT -d '{"name":"Alice"}' https://api.example.com/users/1
curl -X DELETE https://api.example.com/users/1
# Headers and JSON
curl -H "Content-Type: application/json" \
-H "Authorization: Bearer TOKEN" \
-d '{"key":"value"}' \
https://api.example.com/data
# Download with wget
wget https://example.com/file.zip
wget -q https://example.com/file.zip # quiet
wget -c https://example.com/file.zip # continue/resume
# Follow redirects
curl -L https://short.url/abc
# Save response, show headers
curl -s -o response.json -w "%{http_code}" https://api.example.com
# Download with progress bar
curl --progress-bar -O https://example.com/largefile.iso
# POST form data
curl -d "name=Alice&[email protected]" https://api.example.com/formSignals & Traps
trap cleanup handlers
trap registers handlers for signals. EXIT is special — it fires on shell exit for any reason, making it perfect for cleanup. Always clean up lock files, temp dirs, and child processes in a trap. Common signals to trap: INT (Ctrl-C), TERM (default kill), HUP (terminal closed), EXIT.
cleanup() {
echo "Cleaning up..."
rm -f /tmp/myapp.lock
exit 0
}
# catch INT (Ctrl-C), TERM, EXIT
trap cleanup INT TERM EXIT
# normal exit also triggers EXIT trap
echo "Working..."
sleep 30
# cleanup runs automatically
# remove trap
trap - EXIT # disable
trap - INT TERM EXIT # disable alltemp files with cleanup
Never hand-roll temp file names with $$ or $RANDOM — predictable paths invite symlink attacks. mktemp creates a uniquely named file/dir atomically. Pair with a trap on EXIT to guarantee cleanup even on error or interrupt. Set TMPDIR env var to control location.
TMPDIR=$(mktemp -d)
TMPFILE=$(mktemp)
cleanup() {
rm -rf "$TMPDIR" "$TMPFILE"
}
trap cleanup EXIT
# use temp files safely
echo "data" > "$TMPFILE"
process_data < "$TMPFILE"
# mktemp is safer than $RANDOM — avoids symlink attacks
# -d creates a directory
# -p /var/tmp specifies parent
# mktemp -t myapp.XXXXXX # use TMPDIR envsignal handling in scripts
For long-running scripts, use a flag variable flipped by the signal handler to break out of the main loop gracefully. This lets you finish the current iteration, persist state, and exit cleanly. Avoid doing heavy work inside the trap handler itself — just set flags.
#!/bin/bash
# graceful shutdown for a long-running script
RUNNING=1
stop() {
echo "Stopping..."
RUNNING=0
}
trap stop INT TERM
while [ $RUNNING -eq 1 ]; do
echo "Tick $(date)"
sleep 1
done
echo "Done."
# send signals: kill -TERM <pid> or Ctrl-Ctimeout and watch
timeout runs a command and kills it (SIGTERM by default) after a duration. Exit code 124 means it timed out. Use --kill-after to escalate to SIGKILL if the process ignores SIGTERM. Combine with loops to build health checks. watch is for human-facing periodic display, not scripts.
# timeout kills a command after N seconds
timeout 30 slow_command
timeout 5s curl https://slow.example.com
timeout --signal=KILL 60 build.sh # escalate to SIGKILL
timeout -k 5 30 command # kill -9 after 5s grace
# exit status: 124 if timed out
timeout 2 sleep 10
echo $? # 124
# watch reruns a command
watch -n 1 date
watch -n 5 'kubectl get pods'
# tlimit-style loops
while true; do
timeout 5 curl -s https://api.example.com/health || echo "down"
sleep 60
doneignoring and re-raising
trap '' SIGNAL ignores a signal (handy for un-interruptible critical sections). To forward signals to a child, capture its PID with $! and re-send. The ERR trap fires on any command failure when set -e is enabled — useful for logging the failing line number via $LINENO.
# ignore a signal entirely
trap '' INT # ignore Ctrl-C
trap - INT # restore default
# re-raise / propagate to children
trap 'kill -TERM $childpid; wait $childpid' TERM
# forward signals to a subprocess
childpid=
run_child() {
some_server &
childpid=$!
wait $childpid
}
trap 'kill -TERM $childpid' TERM INT
run_child
# exit on any error
set -e
trap 'echo "Failed at line $LINENO"' ERRFunctions & Libraries
defining functions
Functions group reusable commands. Two syntaxes: name() {...} and function name {...}. Arguments inside functions are $1, $2, etc. — $0 is still the script name. $@ expands to all args (always quote as "$@" to preserve args with spaces). $# is the count.
# two equivalent syntaxes
greet() {
echo "Hello, $1!"
}
function greet { # also valid
echo "Hello, $1!"
}
greet Alice # Hello, Alice!
greet "Bob Smith" # quoted args
# arguments: $1, $2, ... $@, $#, $*
echo_args() {
echo "Count: $#"
echo "All: $@"
echo "First: $1"
for arg in "$@"; do echo "- $arg"; done
}return values and locals
return sets the exit status (0=success, 1-255=failure) — it does NOT return a value. To get a value back, echo it and capture with $(). Use local for variables inside functions to avoid polluting the global scope. local is essential in recursive functions.
# return is exit status (0-255), not a value
is_even() {
if (( $1 % 2 == 0 )); then return 0; else return 1; fi
}
if is_even 4; then echo "even"; fi
# capture stdout as a value
get_date() { date +%Y-%m-%d; }
today=$(get_date)
# local variables
counter() {
local count=0 # scoped to function
((count++))
echo $count
}sourcing libraries
source (or .) executes a file in the current shell — so functions, variables, and aliases become available. This is how you build reusable libraries. BASH_SOURCE[0] vs $0 distinguishes sourced vs executed — handy for files that are both a library and a runnable script.
# library file: lib/utils.sh
log() {
echo "[$(date +%H:%M:%S)] $*" >&2
}
CONFIG_PATH="/etc/myapp"
# main script
source lib/utils.sh # or: . lib/utils.sh
log "Starting..."
echo "Config: $CONFIG_PATH"
# check if sourced vs executed
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
echo "Running directly"
else
echo "Sourced"
firecursive functions
Bash supports recursion, but it's slow (each call spawns subshells for $()) and has a limited stack depth (~1000s). For compute-heavy work, prefer awk, Python, or external tools. Always use local variables in recursive functions — otherwise they clobber each other across calls.
# factorial
factorial() {
local n=$1
if (( n <= 1 )); then
echo 1
else
local prev=$(factorial $((n - 1)))
echo $((n * prev))
fi
}
factorial 5 # 120
# fibonacci
fib() {
if (( $1 < 2 )); then echo $1; return; fi
echo $(( $(fib $(($1 - 1))) + $(fib $(($1 - 2))) ))
}
# caution: bash recursion is slow and limited depth
# for heavy work, use awk, python, or external toolsdefault and optional arguments
${1:-default} substitutes a default if $1 is unset or empty. ${1:?message} exits with an error if missing — great for required args. shift pops $1 off the argument list. Combining case + shift is the idiomatic way to parse flags and positional args in bash functions.
# default values
greet() {
local name="${1:-World}" # default if unset or empty
local greeting="${2:-Hello}"
echo "$greeting, $name!"
}
greet # Hello, World!
greet Alice # Hello, Alice!
greet Alice Hi # Hi, Alice!
# required argument
require_arg() {
: "${1:?Usage: require_arg <name>}" # exits with message if missing
echo "Got: $1"
}
# shift through args
process() {
while (( $# )); do
case "$1" in
-v) verbose=1; shift ;;
*) files+=("$1"); shift ;;
esac
done
}Arrays & Associative Arrays
indexed arrays
Indexed arrays use 0-based integer keys. Always quote "${arr[@]}" when iterating to handle elements with spaces correctly. ${#arr[@]} is the count. ${!arr[@]} gives the indices (useful for sparse arrays). unset 'arr[i]' (quoted) removes an element.
# declare and assign
fruits=("apple" "banana" "cherry")
fruits+=("date") # append
fruits[10]="kiwi" # sparse
# access
echo "${fruits[0]}" # first
echo "${fruits[@]}" # all values
echo "${#fruits[@]}" # count
echo "${!fruits[@]}" # indices
# iterate
for f in "${fruits[@]}"; do
echo "$f"
done
# slice
echo "${fruits[@]:1:2}" # 2 items starting at index 1
# delete
unset 'fruits[1]'associative arrays (maps)
Associative arrays (declare -A) are bash 4+ feature (macOS ships bash 3 by default — use brew install bash). Keys are strings. -v test checks if a key exists. Iterate keys with ${!arr[@]}. macOS default /bin/bash is 3.2 — scripts using -A need #!/usr/bin/env bash pointing to a newer bash.
# requires bash 4+
declare -A ages
ages["alice"]=30
ages["bob"]=25
ages["carol"]=42
# access
echo "${ages["alice"]}"
echo "${ages[@]}" # all values
echo "${!ages[@]}" # all keys
# iterate keys
for name in "${!ages[@]}"; do
echo "$name => ${ages[$name]}"
done
# check key exists
if [[ -v ages["alice"] ]]; then echo "yes"; fi
# count
echo "${#ages[@]}"
# delete
unset 'ages[bob]'reading into arrays
mapfile (a.k.a. readarray) is the fastest way to slurp lines into an array — always use -t to strip trailing newlines. The while-read loop is portable but slower. To split a delimited string, set IFS and use read -ra. < <(cmd) process substitution avoids subshell variable scope issues.
# read lines into array
mapfile -t lines < file.txt # bash 4+
echo "${lines[0]}"
# alternative (portable)
lines=()
while IFS= read -r line; do
lines+=("$line")
done < file.txt
# split string into array
IFS=',' read -ra csv <<< "a,b,c,d"
echo "${csv[2]}" # c
# from command
mapfile -t files < <(find . -name "*.js")
# process each
for f in "${files[@]}"; do
echo "Processing $f"
donearray operations
Bash arrays lack many built-in operations (no .indexOf, .reverse, .unique). For these, write loops or pipe through sort/uniq. The unique-array trick uses an associative array as a set. For complex array manipulation, consider awk or a real programming language — bash arrays are best for simple lists.
# append
arr=(a b c)
arr+=(d e) # (a b c d e)
# length
echo "${#arr[@]}"
# slice
echo "${arr[@]:1:2}" # (b c)
# find index (no built-in; loop)
idx=-1
for i in "${!arr[@]}"; do
if [[ "${arr[$i]}" == "b" ]]; then idx=$i; break; fi
done
# reverse
n=${#arr[@]}
for ((i=n-1; i>=0; i--)); do
echo "${arr[$i]}"
done
# unique (preserve order)
declare -A seen
unique=()
for x in "${arr[@]}"; do
[[ -v seen[$x] ]] && continue
seen[$x]=1
unique+=("$x")
donenested data with arrays
Bash has no native nested arrays. Workarounds: (1) store records as delimiter-joined strings and split with IFS, (2) use an associative array with composite keys like "r,c", (3) use variable indirection with ${!name}. For real nested data, switch to jq + JSON or a real language.
# array of "records" (delimiter-based)
users=("alice:30:admin" "bob:25:user")
for u in "${users[@]}"; do
IFS=':' read -r name age role <<< "$u"
echo "$name ($age) - $role"
done
# simulate 2D with name-mangled vars
declare -A grid
grid["0,0"]=1; grid["0,1"]=2
grid["1,0"]=3; grid["1,1"]=4
for key in "${!grid[@]}"; do
IFS=',' read -r r c <<< "$key"
echo "[$r,$c] = ${grid[$key]}"
done
# arrays of arrays (not directly supported)
# workaround: use indirection
rows=("row0" "row1")
row0=(a b c)
row1=(d e f)
ref="${rows[0]}[@]"
echo "${!ref}" # a b cString Manipulation
parameter expansion
Parameter expansion is bash's most powerful string tool. # and ## strip from the front (shortest/longest match); % and %% strip from the end. / replaces first, // replaces all. These avoid spawning subshells — much faster than piping through sed for simple ops.
path="/var/log/app.log"
# length
echo "${#path}" # 16
# substring (offset:length)
echo "${path:5}" # log/app.log
echo "${path:5:3}" # log
# remove from front (shortest/longest)
echo "${path#*/}" # var/log/app.log
echo "${path##*/}" # app.log (basename)
# remove from end
echo "${path%/*}" # /var/log (dirname)
echo "${path%%.*}" # /var/log/app
# replace
echo "${path/log/LOG}" # /var/LOG/app.log (first)
echo "${path//log/LOG}" # all occurrencescase conversion
Bash 4+ added ,, (lower), ^^ (upper), ^ (capitalize first char), , (lower first char). declare -u/-l auto-converts assignments. For older bash (macOS default), use tr or awk. Case conversion is common when normalizing user input or building identifiers.
s="Hello World"
# bash 4+
echo "${s,,}" # hello world (lower)
echo "${s^^}" # HELLO WORLD (upper)
echo "${s^}" # Hello World (capitalize first)
echo "${s,}" # hello World (lowercase first)
# pattern-based
echo "${s^^[aeiou]}" # hEllO wOrld (vowels upper)
# portable (pre-bash 4)
echo "$s" | tr 'a-z' 'A-Z'
echo "$s" | awk '{print toupper($0)}'
# declare typed variables
declare -u UPPER="hello" # HELLO
declare -l LOWER="WORLD" # worldsplitting and joining
Splitting: set IFS and use read -ra. Joining: ${arr[*]} with IFS set works but is finicky — the join_by function is more reliable. The printf '%s' "$d%s" trick prepends the delimiter to each remaining arg, producing a clean join. tr can convert between delimiters and newlines.
# split on delimiter
csv="a,b,c,d"
IFS=',' read -ra parts <<< "$csv"
echo "${parts[2]}" # c
for p in "${parts[@]}"; do echo "$p"; done
# join with delimiter (bash 4+)
arr=(a b c d)
joined=$(IFS=,; echo "${arr[*]}")
echo "$joined" # a,b,c,d
# custom joiner
join_by() {
local d=$1; shift
local first=$1; shift
printf '%s' "$first"
printf '%s' "$d%s" "$@"
}
join_by " | " a b c # a | b | c
# split into lines
tr ',' '\n' <<< "a,b,c"search and test
Pattern matching: == with wildcards (* ? [..]) does glob matching. =~ does extended regex; capture groups land in BASH_REMATCH. shopt -s nocasematch makes [[ ]] comparisons case-insensitive. expr is legacy — prefer [[ ]] for new code.
s="Hello World"
# substring search
if [[ "$s" == *"World"* ]]; then echo "contains"; fi
if [[ "$s" == Hello* ]]; then echo "starts with"; fi
if [[ "$s" == *World ]]; then echo "ends with"; fi
# regex match
if [[ "$s" =~ ^H[a-z]+\sW[a-z]+$ ]]; then echo "match"; fi
echo "${BASH_REMATCH[0]}" # whole match
echo "${BASH_REMATCH[1]}" # first group
# case-insensitive
shopt -s nocasematch
[[ "$s" == *world* ]] && echo "yes"
shopt -u nocasematch
# index (no built-in; use expr)
expr "$s" : ".*World" # length up to matchprintf formatting
printf is more powerful than echo — supports C-style format specifiers (%s, %d, %f, %x), width/precision, and alignment. %()T formats time without spawning the date command. printf -v stores the result in a variable. printf '=%.0s' {1..40} is a neat trick to repeat a character N times.
# basic
printf '%s\n' "hello" "world" # one per line
printf '%s = %d\n' "count" 42
# width and padding
printf '%10s\n' "right" # right-aligned, width 10
printf '%-10s|\n' "left" # left-aligned
printf '%05d\n' 42 # 00042 (zero-padded)
printf '%.2f\n' 3.14159 # 3.14
# date formatting
printf 'Today: %(%Y-%m-%d)T\n' -1
# to variable
printf -v today '%(%Y%m%d)T' -1
echo "$today" # 20260621
# repeat
printf '=%.0s' {1..40}; echo # 40 equals signsRegular Expressions
bash regex with =~
=~ uses ERE (extended regex). Capture groups populate BASH_REMATCH (index 0 = whole match, 1+ = groups). Quote the string but NOT the regex (quoting the regex makes it literal). For case-insensitive matching, use shopt -s nocasematch. Regex is POSIX ERE — no \d, \w; use [0-9], [A-Za-z0-9_].
s="user_42"
if [[ "$s" =~ ^([a-z]+)_([0-9]+)$ ]]; then
echo "Match!"
echo "Name: ${BASH_REMATCH[1]}" # user
echo "ID: ${BASH_REMATCH[2]}" # 42
fi
# email-ish check
email="[email protected]"
if [[ "$email" =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ]]; then
echo "valid"
fi
# case-insensitive
shopt -s nocasematch
[[ "$s" =~ USER ]] && echo "yes"
shopt -u nocasematchgrep regex patterns
grep defaults to BRE (basic regex) where +, ?, |, () need backslashes. -E enables ERE (cleaner syntax). -P enables PCRE with \d, \w, \b, lookarounds — most powerful but less portable (GNU grep only). For scripts that must run on macOS/BSD, prefer -E.
# basic (BRE) — default
grep 'foo' file
grep 'a\.' file # literal dot (escape in BRE)
grep 'a*b' file # zero or more a
# extended (ERE) — -E
grep -E 'a+b' file # one or more
grep -E 'a|b' file # alternation
grep -E '(ab)+' file # grouping
grep -E '[0-9]{3}' file # quantifier
# perl (PCRE) — -P
grep -P '\d{3}' file # digits
grep -P '\bword\b' file # word boundary
grep -P '(?<=foo)bar' file # lookbehind
# anchors
grep -E '^Error' file # line start
grep -E 'done$' file # line end
grep -E '^Error.*timeout$' filesed regex substitution
sed uses BRE by default (capture groups need \( \), backrefs \1). -E switches to ERE (cleaner () and +, ?, |). The I flag makes substitution case-insensitive (GNU sed). Use a different delimiter (s|...|...|) when the pattern contains slashes. [[:space:]] is portable for whitespace.
# BRE (default)
sed 's/foo/bar/' file
sed 's/a\.b/x/' file # literal dot
sed 's/a\(b\)c/\1/' file # capture with backrefs
# ERE with -E
sed -E 's/a+b/x/' file
sed -E 's/(foo|bar)/X/g' file
sed -E 's/([0-9]{4})/Year: \1/' file
# case-insensitive
sed 's/foo/bar/I' file
sed 's/[fF][oO][oO]/bar/g' file
# common recipes
sed -E 's/^[[:space:]]+//' file # trim leading whitespace
sed -E 's/[[:space:]]+$//' file # trim trailing
sed -E 's/ +/ /g' file # collapse spaces
sed -E 's#https?://[^ ]+##g' file # strip URLsawk regex
awk uses ERE. The ~ operator tests a string against a regex. /pat1/, /pat2/ is a range pattern (matches from pat1 to pat2 inclusive). gsub does in-place global substitution. gawk's match() with a third arg captures groups into an array — handy but not portable to mawk/POSIX awk.
# match operator ~
awk '$1 ~ /^[0-9]+/' file # field 1 is numeric
awk '$0 !~ /debug/' file # line doesn't contain
awk '/^Error/,/^$/' file # from Error to blank line
# case-insensitive
awk 'BEGIN { IGNORECASE=1 } /error/' file
awk 'tolower($0) ~ /error/' file
# gsub (global substitute)
awk '{ gsub(/foo/, "bar"); print }' file
awk '{ gsub(/[0-9]+/, "N"); print }' file
# match with capture (gawk)
awk 'match($0, /id=([0-9]+)/, m) { print m[1] }' file
# split on regex
awk '{ n = split($0, parts, /[,;\t]/); print n }' filecommon regex recipes
These recipes cover common text-extraction tasks. -o prints only the matched portion. Note that regex is not ideal for parsing HTML, JSON, or XML — use proper parsers (jq for JSON, xmllint for XML). For IPv4, this regex matches format but not valid ranges (255.255.255.255 vs 999.999.999.999).
# IPv4
grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log
# URL
grep -Eo 'https?://[^[:space:]]+' file
# email
grep -Eo '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' file
# date YYYY-MM-DD
grep -E '[0-9]{4}-[0-9]{2}-[0-9]{2}' file
# hex color
grep -Eo '#[0-9A-Fa-f]{6}' file
# strong password (8+, upper, lower, digit)
grep -E '^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9]).{8,}$' file
# trim whitespace
sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//' file
# extract domain from URL
sed -E 's#https?://([^/]+).*#\1#' fileDebugging
set options
set -e exits on any command failure (must-have for scripts). set -u catches typos in variable names. set -o pipefail makes a pipe fail if any part fails (without it, only the last command's exit code matters). set -x traces execution. Combined as -euo pipefail, this is 'strict mode' — the safest default.
#!/bin/bash
set -e # exit on error
set -u # error on unset variable
set -o pipefail # pipe fails if any command fails
set -x # print commands (trace)
# or combined:
set -euo pipefail
# shorthand
set -Eeuxo pipefail # -E: ERR trap inherits
# strict mode for robust scripts
set -euo pipefail
IFS=$'\n\t'
# trace specific section
set -x
complex_logic
set +x
# debug a script
bash -x script.sh
bash -xv script.sh # verbose + tracetracing with -x
set -x prints each command before execution, with + as the default prefix (PS4). Customizing PS4 to include file:line:function makes traces far more useful for debugging. BASH_XTRACEFD redirects the trace to a different FD, so you can separate trace output from stdout/stderr.
# trace entire script
bash -x script.sh
# trace inside script
set -x
echo "tracing on"
set +x
# customize PS4 (the trace prompt)
export PS4='+ ${BASH_SOURCE:-shell}:${LINENO}: ${FUNCNAME[0]:-main}() '
set -x
my_function
set +x
# trace to file
exec 5> /tmp/trace.log
BASH_XTRACEFD=5
set -x
# ... commands ...
set +x
exec 5>&-error handling and ERR trap
The ERR trap fires when a command fails (with set -e). $? inside the trap gives the failing exit code. $LINENO shows where the failure happened. The caller builtin prints file:line:function — looping it produces a stack trace. set -E makes the ERR trap fire inside functions too (otherwise it's disabled in function context).
set -e
set -E # ERR trap inherits functions
on_error() {
local exit_code=$?
local line=$1
echo "ERROR: command failed at line $line with exit code $exit_code" >&2
echo "Stack:" >&2
local i=0
while caller $i >&2; do ((i++)); done
exit $exit_code
}
trap 'on_error $LINENO' ERR
# now any failing command triggers the handler
false # triggers on_error
# caller builtin gives file:line:func
# stack trace via: while caller $i; do ((i++)); doneshellcheck linter
shellcheck is the de-facto linter for shell scripts — catches quoting issues, unused vars, common pitfalls, and suggests idioms. Run it on every script. The most common fix is SC2086: quote variables to prevent word-splitting and globbing. Many CI pipelines require shellcheck to pass.
# install: apt install shellcheck (or brew, or use shellcheck.net)
# run:
shellcheck script.sh
shellcheck -x script.sh # trust includes (no SC1090 warnings)
shellcheck -S warning script.sh # only warnings+
shellcheck --exclude=SC2086 *.sh # exclude specific rules
# common warnings:
# SC2086: double-quote to prevent globbing (use "$var")
# SC2046: quote $(cmd) to prevent word splitting
# SC2034: unused variable
# SC2155: declare and assign separately (masks return code)
# SC2004: $/${} unnecessary in arithmetic
# in script: disable inline
# shellcheck disable=SC2086
echo $unquoted_var
# disable for next line
# shellcheck disable=SC2086
echo $argslogging and diagnostics
A leveled logger (DEBUG/INFO/WARN/ERROR) with timestamps and colors makes scripts much easier to debug. Always log to stderr so stdout stays clean for data. Support a --verbose flag (enables set -x) and --dry-run (prints commands without executing). ANSI color codes: 31=red, 33=yellow, 32=green, 90=gray.
# logging function with levels
log() {
local level=$1; shift
local msg="$*"
local ts=$(date '+%Y-%m-%d %H:%M:%S')
local color
case $level in
DEBUG) color='\033[90m' ;;
INFO) color='\033[32m' ;;
WARN) color='\033[33m' ;;
ERROR) color='\033[31m' ;;
esac
printf "%b[%s] [%s] %s\033[0m\n" "$color" "$ts" "$level" "$msg" >&2
}
log INFO "Starting process"
log WARN "Disk space low"
log ERROR "Connection failed"
# verbose flag
[[ $VERBOSE ]] && set -x
# dry-run mode
if [[ $DRY_RUN ]]; then
echo "Would: $cmd"
else
eval "$cmd"
fiSecurity
quoting and injection
The #1 bash security rule: quote every variable expansion. Unquoted $var undergoes word-splitting and globbing, leading to bugs and injection. For commands with dynamic args, use an array (cmd=(ls "$dir"); "${cmd[@]}") instead of a string. Never eval or $() user input into a shell string.
# ALWAYS quote variables
file="my file.txt"
rm $file # BAD: rm my file.txt (3 args!)
rm "$file" # GOOD: rm "my file.txt"
# use arrays for command + args
files=("a.txt" "b file.txt" "c.txt")
rm "${files[@]}" # each file as separate arg, safely
# avoid eval
eval "echo $user_input" # DANGEROUS
echo "$user_input" # safe
# avoid command substitution in unsafe places
# BAD: filename from user, executed
cmd="ls $userdir"
$cmd # word-splitting + globbing
# GOOD: use array
cmd=(ls "$userdir")
"${cmd[@]}"safe temp files and secrets
mktemp creates files with unpredictable names and 600 perms — safe from symlink races. Always clean up with a trap. Read secrets from environment variables, never hardcode. Avoid passing secrets as command-line args (visible via ps) — use stdin, env vars, or a config file with 600 perms.
# safe temp files
tmp=$(mktemp) # atomic, unpredictable name
trap 'rm -f "$tmp"' EXIT
# safe temp dir
tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT
# secrets: read from env, never hardcode
api_key="${API_KEY:?Set API_KEY env var}"
# don't leak secrets to process list (ps)
# BAD: curl -H "Authorization: Bearer $TOKEN" ...
# GOOD: pass via stdin or config file
curl -H @- <<EOF https://api.example.com
Authorization: Bearer $TOKEN
EOF
# restrict file perms for secrets
umask 077
echo "$api_key" > ~/.secrets/api
chmod 600 ~/.secrets/apiinput validation
Validate all user input before use. Regex checks format; arithmetic checks ranges. For file paths, reject .. (traversal) and absolute paths unless explicitly allowed. tr -cd strips characters not in the allowed set — useful for sanitizing identifiers. Whitelist (case) is safer than blacklist for allowed actions.
# validate numeric input
read -p "Enter a number: " n
if ! [[ "$n" =~ ^[0-9]+$ ]]; then
echo "Not a number" >&2; exit 1
fi
# validate range
if (( n < 1 || n > 100 )); then
echo "Out of range" >&2; exit 1
fi
# validate file path (no traversal)
path="$1"
case "$path" in
*..*|/*) echo "Invalid path" >&2; exit 1 ;;
esac
# sanitize filename
safe=$(echo "$name" | tr -cd 'A-Za-z0-9._-')
# whitelist allowed values
case "$action" in
start|stop|restart) ;;
*) echo "Invalid action" >&2; exit 1 ;;
esacprivilege drop and setuid
setuid shell scripts are insecure — most systems ignore the setuid bit on scripts. Use sudo with a tight sudoers entry (specific command, no password) instead. Drop root as early as possible via exec sudo -u. For capabilities (e.g., binding port 80), use setcap instead of running as root. chmod 700 keeps scripts private to the owner.
# drop privileges in a script running as root
if [[ $EUID -eq 0 ]]; then
exec sudo -u appuser "$0" "$@"
fi
# check for root
if [[ $EUID -ne 0 ]]; then
echo "Requires root" >&2; exit 1
fi
# setuid scripts are DANGEROUS — avoid them
# instead, use sudoers entries
# /etc/sudoers.d/myapp:
# appuser ALL=(root) NOPASSWD: /usr/local/bin/myapp.sh
# restrict script permissions
chmod 700 script.sh # only owner can run
chown appuser:appuser script.sh
# use capsh instead of full root for fine-grained caps
# setcap 'cap_net_bind_service=+ep' /usr/bin/myappsignatures and integrity
Always verify checksums and signatures for downloaded scripts, especially installers run as root. sha256sum -c compares against a known-good hash. GPG signatures prove both integrity and authenticity (the signer's identity). Piping curl to sha256sum lets you eyeball the hash before executing — but the safer pattern is download-then-verify-then-execute.
# verify checksum
sha256sum -c file.sha256
# file.sha256 contains: <hash> <filename>
# generate checksums
sha256sum release.tar.gz > release.tar.gz.sha256
sha256sum *.iso > isos.sha256
# GPG sign and verify
gpg --sign file.txt # creates file.txt.gpg
gpg --verify file.txt.gpg
gpg --detach-sig --armor file.tar.gz
gpg --verify file.tar.gz.asc file.tar.gz
# download + verify in one go
curl -fsSL https://example.com/install.sh | sha256sum
curl -fsSL https://example.com/install.sh -o install.sh
sha256sum -c <(echo "expected_hash install.sh")Networking
curl Advanced
curl transfers data via URLs. -X sets method, -d sends data, -H sets headers. -O saves with remote filename. -I fetches headers only. -L follows redirects. -u user:pass for auth. -v for verbose.
# POST with data
curl -X POST -d "name=Alice" https://api.example.com
# JSON POST
curl -H "Content-Type: application/json" \
-d '{"name":"Alice"}' https://api.example.com
# Download with progress
curl -O https://example.com/file.zip
# Headers only
curl -I https://example.comwget
wget downloads files non-interactively. -m mirrors sites. -c resumes interrupted downloads. -r recurses, -l limits depth. -b runs in background. Better than curl for recursive downloads.
# Download file
wget https://example.com/file.zip
# Mirror website
wget -m https://example.com
# Resume download
wget -c https://example.com/largefile.zip
# Recursive download
wget -r -l 2 https://example.com
# Background
wget -b https://example.com/file.zipssh & scp
ssh connects to remote machines. -i specifies a key file. -L creates local port forwarding. scp copies files over SSH. -r recurses directories. Use ssh-copy-id to install keys. Configure ~/.ssh/config for aliases.
# SSH connect
ssh user@host
# SSH with key
ssh -i ~/.ssh/id_rsa user@host
# Port forwarding
ssh -L 8080:localhost:80 user@host
# Copy file
scp file.txt user@host:/path/
# Copy directory
scp -r dir/ user@host:/path/netstat & ss
netstat and ss show network connections. -t TCP, -u UDP, -l listening, -n numeric, -p process. ss is faster and more detailed than netstat. Use to find which process uses a port or diagnose connection issues.
# List listening ports
netstat -tlnp
ss -tlnp
# Show all connections
netstat -an
ss -tunap
# Show routing table
netstat -r
# Show interface stats
netstat -iping & traceroute
ping tests reachability and latency (-c limits count). traceroute shows the path to a host. dig queries DNS records (A, MX, NS, TXT). +short gives concise output. Use for network diagnostics and DNS troubleshooting.
# Check connectivity
ping -c 4 google.com
# Trace route
traceroute google.com
# DNS lookup
dig example.com
dig +short example.com
# Get DNS records
dig MX example.com
dig NS example.comShell Scripting Deep
Functions
Functions group commands. local creates function-scoped variables. $1, $2 are arguments. return sets exit status (0-255). Capture output with $(). Functions must be defined before use. Use source to load from files.
greet() {
local name="$1"
echo "Hello, $name"
return 0
}
greet "Alice"
# Return value via $?
result=$(greet "Bob")Conditionals
[ ] is the test command. -f file, -d directory, -z empty string, -n non-empty. = for strings, -eq/-ne/-gt/-lt for numbers. Always quote variables to handle spaces. [[ ]] is bash-enhanced with regex.
if [ -f "/path/file" ]; then
echo "File exists"
elif [ -d "/path" ]; then
echo "Directory exists"
else
echo "Not found"
fi
# String comparison
if [ "$str" = "hello" ]; then ... fi
# Numeric
if [ "$num" -gt 10 ]; then ... fiLoops
for iterates over lists. C-style for uses (( )). while reads until condition fails. while read processes files line by line safely. Always quote variables. Use find for recursive file iteration.
# For loop
for i in 1 2 3; do
echo $i
done
# C-style
for ((i=0; i<5; i++)); do echo $i; done
# While loop
while read line; do
echo "$line"
done < file.txt
# Iterate files
for f in *.txt; do process "$f"; doneArrays
Arrays use parentheses. ${arr[@]} expands all elements. ${#arr[@]} is the length. Always quote "${arr[@]}" to handle spaces. declare -A creates associative arrays (bash 4+). Use read -a to split into array.
# Declare array
fruits=("apple" "banana" "cherry")
# Access
echo ${fruits[0]} # apple
echo ${fruits[@]} # all
echo ${#fruits[@]} # length
# Iterate
for fruit in "${fruits[@]}"; do
echo "$fruit"
done
# Associative array
declare -A config
config[key]="value"Error Handling
set -e exits on error, -u on undefined vars, pipefail catches pipe failures. trap ERR handles errors. trap EXIT runs cleanup. command -v checks if a command exists. Always start scripts with set -euo pipefail.
set -e # Exit on error
set -u # Error on undefined variable
set -o pipefail # Pipeline fails if any command fails
set -euo pipefail # All three (recommended)
trap 'echo "Error on line $LINENO"' ERR
trap 'cleanup' EXIT
# Check command result
if ! command -v docker &> /dev/null; then
echo "docker not installed"
fiFile Operations Deep
find Advanced
find searches files by criteria. -type f/d for files/dirs. -size +1M larger than 1MB. -mtime -7 modified within 7 days. -exec runs a command on each match. {} is the filename; \; ends the command.
# By name
find . -name "*.txt"
# By type and size
find . -type f -size +1M
# By modification time
find . -mtime -7 # Modified in last 7 days
# Execute command
find . -name "*.log" -exec gzip {} \;
# Delete
find . -name "*.tmp" -deletetar & Compression
tar combines files; -z compresses with gzip. -c create, -x extract, -t list, -v verbose, -f filename. For .tar.bz2 use -j. zip/unzip handle ZIP format. Use -C to extract to a specific directory.
# Create tar.gz
tar -czvf archive.tar.gz dir/
# Extract
tar -xzvf archive.tar.gz
# List contents
tar -tzvf archive.tar.gz
# zip/unzip
zip -r archive.zip dir/
unzip archive.ziprsync
rsync efficiently syncs files, transferring only differences. -a archive mode (preserves attributes), -v verbose, -z compress. --delete removes files not in source. Trailing slash on src/ matters: with it, copies contents; without, copies the directory.
# Sync local
rsync -av src/ dest/
# Sync to remote
rsync -avz src/ user@host:/path/
# Delete files not in source
rsync -av --delete src/ dest/
# Dry run
rsync -av --dry-run src/ dest/
# Exclude
rsync -av --exclude='*.log' src/ dest/file Permissions
Permissions: r=4, w=2, x=1. 755 = rwxr-xr-x (owner full, others read/execute). 644 = rw-r--r-- (files). u/g/o/a = user/group/other/all. chmod -R recurses. chown changes ownership.
# Symbolic
chmod u+x file # User execute
chmod g-w file # Group write off
chmod a=r file # All read only
# Numeric
chmod 755 file # rwxr-xr-x
chmod 644 file # rw-r--r--
# Change owner
chown user:group file
# Recursive
chmod -R 755 dir/ln & symlinks
Hard links point to the same inode (same filesystem only). Symbolic links (-s) point to a path. Hard links survive deletion of the original. Symlinks break if the target moves. Use readlink to resolve symlinks.
# Hard link
ln file.txt hardlink.txt
# Symbolic link
ln -s /path/to/target symlink
# Find broken symlinks
find . -type l ! -exec test -e {} \; -print
# Copy symlink target
cp -L symlink dest/Common Pitfalls
Unquoted Variables
Unquoted variables split on spaces and glob. $(ls) breaks with filenames containing spaces. Use globs (*.txt) or find -print0 with read -d. Always quote variables: "$var". Use IFS= read -r for safe line reading.
# BUG: breaks with spaces
for f in $(ls); do echo "$f"; done
# GOOD: handle spaces
for f in *.txt; do echo "$f"; done
# Or with find
find . -name "*.txt" -print0 | while IFS= read -r -d '' f; do
echo "$f"
donecp vs mv
cp copies, mv moves. Both can overwrite without warning. Use -i to prompt before overwrite. Use -n to never overwrite. mv is atomic on the same filesystem, useful for locks and atomic updates.
# cp: copy (original remains)
cp file.txt backup.txt
# mv: move/rename (original removed)
mv file.txt newname.txt
# Common mistake: overwriting
cp important.txt /tmp/ # If /tmp/important.txt exists, overwritten!
# Use -i for safety
cp -i file.txt /tmp/rm Dangers
rm -rf is dangerous, especially with variables. Empty variables cause rm -rf /. Always quote and check variables. Consider trash-cli for recoverable deletion. Never run rm -rf / or with sudo carelessly.
# DANGEROUS: recursive force
rm -rf / # Never do this!
rm -rf $VAR/ # If $VAR is empty, becomes rm -rf /
# Safer: check first
if [ -n "$VAR" ]; then
rm -rf "$VAR/"
fi
# Use trash-cli instead
trash-put file.txtShebang
The shebang (#!) specifies the interpreter. #!/usr/bin/env bash is more portable than #!/bin/bash. Use /bin/sh for maximum portability (avoids bashisms). Always make scripts executable with chmod +x.
#!/bin/bash # bash script
#!/usr/bin/env bash # Portable bash
#!/usr/bin/env python3 # Python
#!/bin/sh # POSIX sh (more portable)
# Make executable
chmod +x script.sh
./script.shExit Status
$? holds the exit status of the last command. 0 = success, non-zero = failure. Check immediately before another command overwrites it. Better to use if command directly. Always exit with meaningful status codes in scripts.
# Check last command
if [ $? -eq 0 ]; then
echo "Success"
fi
# Better: direct check
if command; then
echo "Success"
else
echo "Failed"
fi
# Exit with status
exit 0 # Success
exit 1 # FailureRelated Bash snippets
Copy-paste ready code for common tasks.
Variables and Arrays in Bash
Assign variables, use command substitution, and work with arrays in Bash.
File Operations
File and directory management.
Text Processing
Text processing with grep, sed, awk.
Loops
for and while loops.
Conditionals
if and case conditional statements.
Functions
Function definition and parameters.
Arrays
Bash array operations.
String Operations
Bash string processing.
Git Basics
Basic Git operations.
Git Branches
Branch management and operations.
Git Merge
Merging and conflict resolution.
Git Revert
Undo commits and revert.
Git Tags
Version tag management.
Git Stash
Stash working directory changes.
Git Cherry-pick
Selectively merge commits.
Git Bisect
Binary search to locate problem commits.
Was this helpful?