Reshape output with a pipeline

"Which IP hit us most" and "what is the p95" are pipeline questions. Six tools answer nearly all of them, and the skill is composing them rather than knowing every flag.

1. Make a log to work on

mkdir -p ~/labs/text && cd ~/labs/text
cat > access.log <<'LOG'
10.0.0.7 - GET /api/users 200 120
10.0.0.9 - GET /api/users 200 340
10.0.0.7 - POST /api/login 401 88
10.0.0.3 - GET /health 200 4
10.0.0.7 - GET /api/orders 500 1200
10.0.0.9 - GET /api/orders 200 260
10.0.0.3 - GET /health 200 5
10.0.0.7 - GET /api/users 200 150
LOG
wc -l access.log

Five space-separated fields: client, a dash, method, path, status, duration in ms.

2. Cut a column out

cut -d' ' -f1 access.log
cut -d' ' -f3,4 access.log

cut is the simplest tool and the most brittle: -d' ' means exactly one space, so two spaces between fields produce an empty field. Use it on genuinely fixed-delimiter data — /etc/passwd, CSV — and reach for awk the moment whitespace is irregular.

3. The count-and-rank idiom

This is the pipeline to commit to memory:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn

uniq -c counts adjacent duplicates only, which is why the first sort is mandatory — leave it out and you get nonsense. The second sort -rn ranks numerically, descending. -n matters: without it, 10 sorts before 9.

4. awk for fields, conditions, and arithmetic

awk '{print $1, $5}' access.log
awk '$5 == 500 {print $1, $4}' access.log
awk '$6 > 200 {print $4, $6}' access.log
awk '{sum += $6} END {print "total ms:", sum, "avg:", sum/NR}' access.log

awk splits on runs of whitespace by default, so irregular spacing does not break it. $0 is the whole line, NR is the record number, and an END block runs once at the end — enough for most aggregation without reaching for a real language.

Group and count in one pass:

awk '{count[$4]++} END {for (p in count) print count[p], p}' access.log | sort -rn

An awk associative array replaces sort | uniq -c entirely and reads the file once.

5. sed for substitution

sed 's/10\.0\.0/HOST/' access.log | head -n 3
sed -n '3,5p' access.log
sed '/health/d' access.log

s/// substitutes the first match per line; add g for all of them. -n with p prints only selected lines, and /pattern/d deletes matching lines from the output.

Editing in place needs care:

cp access.log editable.log
sed -i.bak 's/GET/READ/g' editable.log
head -n 2 editable.log editable.log.bak

-i.bak keeps a backup. Plain -i does not, and a bad expression with plain -i destroys the file with no undo. On macOS -i requires an argument, so -i.bak is also the portable form.

6. jq, because half of everything is JSON now

cat > events.json <<'JSON'
[
  {"ts": "2026-01-01T10:00:00Z", "svc": "api", "level": "info", "ms": 12},
  {"ts": "2026-01-01T10:00:01Z", "svc": "api", "level": "error", "ms": 980},
  {"ts": "2026-01-01T10:00:02Z", "svc": "db", "level": "info", "ms": 45},
  {"ts": "2026-01-01T10:00:03Z", "svc": "api", "level": "error", "ms": 1200}
]
JSON
jq '.[0]' events.json
jq -r '.[].svc' events.json
jq -r '.[] | select(.level == "error") | "\(.svc) \(.ms)ms"' events.json
jq '[.[] | .ms] | add / length' events.json
jq -r 'group_by(.svc)[] | "\(.[0].svc) \(length)"' events.json

-r gives raw strings instead of quoted JSON, which is what you want when piping onward. select() filters, \(...) interpolates, and group_by needs sorted input — which it does for you.

7. Put it together

The kind of one-liner this all builds to — slowest endpoints, ranked:

awk '{total[$4] += $6; hits[$4]++} END {
  for (p in total) printf "%8.1f %4d %s\n", total[p]/hits[p], hits[p], p
}' access.log | sort -rn

Average duration, request count, path. Written once, read many times, and no dependency beyond what is already on the box.

Clean up

cd ~ && rm -rf ~/labs/text

Where this goes next

You can turn output into an answer at the prompt. Next: writing that down as a script that does not silently do the wrong thing.