Rewriting Text: sed and awk

grep, covered in Finding Files and Searching Text, finds a line. It does not change it. When a log format needs reshaping, a configuration value needs updating from a script, or a column of numbers in a report needs summing, grep alone is not enough — that is the job of sed and awk.

The examples assume Ubuntu Server LTS with GNU sed and GNU awk (gawk). You should already be comfortable with exit status and pipefail from Streams, Pipes, and Redirection. This article covers rewriting a line safely with sed, checking a change before it touches a file, summarizing column-based data with awk, and deciding which tool fits a given task.

Lab Setup

mkdir -p ~/lab/textproc
cd ~/lab/textproc
printf 'listen_port=8080\nenv=staging\ndebug=true\n' > app.conf
printf '%s\n' \
    '2026-07-29 10:00:00 INFO orders-api 12ms' \
    '2026-07-29 10:00:02 ERROR orders-api 340ms' \
    '2026-07-29 10:00:03 ERROR billing-db 890ms' \
    '2026-07-29 10:00:04 WARN orders-api 45ms' > requests.log

`sed`: Rewriting a Stream Line by Line

sed (stream editor) reads each line in turn, applies a command to it, and writes the result to standard output. It does not touch the original file unless you tell it to explicitly — without a separate in-place flag, the source file is untouched.

The Basic Substitution: `s///`

sed 's/staging/production/' app.conf

listen_port=8080
env=production
debug=true

That result only went to the screen; app.conf itself did not change:

cat app.conf

listen_port=8080
env=staging
debug=true

The syntax is s/pattern/replacement/[flags]. / is the delimiter, but any other character works too — a # is convenient when the pattern itself contains a /:

sed 's#/var/log#/srv/log#' <<< 'path=/var/log/app.log'

Replacing Every Occurrence: the `g` Flag

By default, sed replaces only the first match on each line:

sed 's/ERROR/WARN/' requests.log

2026-07-29 10:00:00 INFO orders-api 12ms
2026-07-29 10:00:02 WARN orders-api 340ms
2026-07-29 10:00:03 WARN billing-db 890ms
2026-07-29 10:00:04 WARN orders-api 45ms

This particular file has only one match per line, so the difference is invisible. When a line contains multiple matches, only the first one changes unless you add g:

sed 's/o/0/' <<< 'foo bar foo'
sed 's/o/0/g' <<< 'foo bar foo'

f0o bar foo
f00 bar f00

Editing a File in Place: `-i`

sed -i.bak 's/env=staging/env=production/' app.conf
cat app.conf

listen_port=8080
env=production
debug=true

-i.bak does two things in GNU sed: it saves the original file as a .bak copy, then edits the original in place. Without a backup suffix (bare -i), the original file is modified irreversibly.

Danger

sed -i without a backup suffix is not reversible. Before touching a production configuration file, either supply a backup suffix (-i.bak) or first run the same substitution without -i and confirm the output only affects the lines you expect. A pattern that is broader than intended can silently rewrite lines you never meant to touch — check with grep beforehand.

The Habit of Checking Before You Change

grep -n 'env=staging' app.conf

Only after confirming that exactly the expected line(s) appear:

sed -i.bak 's/env=staging/env=production/' app.conf
diff app.conf.bak app.conf

Reviewing the diff output is a safer intermediate step than trusting sed -i blindly.

Selecting Specific Lines

sed -n '2p' requests.log
sed -n '2,3p' requests.log
sed '1d' requests.log

-n suppresses the default output, so only lines explicitly printed with p appear.
2,3p selects the range from line 2 through line 3.
1d deletes the first line from the output stream — the file itself is untouched, only the stream is.

A line can also be selected by a regular expression instead of a number:

sed -n '/ERROR/p' requests.log

Groups and Backreferences

A parenthesized group can be reused later as \1, \2, and so on:

sed -E 's/^([0-9-]+) ([0-9:]+) ([A-Z]+)/\3 \1T\2/' requests.log

INFO 2026-07-29T10:00:00 orders-api 12ms
ERROR 2026-07-29T10:00:02 orders-api 340ms
ERROR 2026-07-29T10:00:03 billing-db 890ms
WARN 2026-07-29T10:00:04 orders-api 45ms

-E enables extended regular expressions, where parentheses do not need escaping ((...) instead of $...$). This example moves the date, time, and level fields around to produce something close to ISO 8601.

The `&` Trap in a Replacement

Inside the replacement half of s///, an unescaped & does not mean a literal ampersand — it stands for the entire matched text:

sed 's/orders-api/[&]/' <<< 'service=orders-api'

service=[orders-api]

& expanded to the whole match, wrapping it in brackets. To insert a literal ampersand, escape it as \&. The same caution applies to \1, the delimiter, and backslashes: a replacement string assembled from unescaped or untrusted input can expand in ways you did not intend. This is another reason to preview a substitution without -i before committing it, even when the change looks trivial.

Working Across Lines: the Hold Space

Every example so far treated one line at a time, which is how sed normally works: read a line into the pattern space, apply the commands, print it, move to the next line. Some edits need to see more than one line at once — merging a wrapped line, deduplicating consecutive blank lines, or reversing a file — and for that sed keeps a second scratch buffer called the hold space.

N appends the next input line to the pattern space, joined by a newline, instead of starting a fresh cycle.
D deletes everything in the pattern space up to the first newline, then restarts the cycle without reading a new line — used to work through a multi-line pattern space one line at a time.
h / H copy or append the pattern space into the hold space.
g / G copy or append the hold space back into the pattern space.

A common use is collapsing consecutive blank lines down to a single one:

printf 'one\n\n\n\ntwo\n\nthree\n' > blanks.txt
sed '/^$/{ N; /^\n$/D }' blanks.txt

one

two

three

Reading it step by step: when a blank line is found, N pulls in the next line to check whether it is blank too. If the two-line pattern space is exactly two blank lines (^\n$), D drops the first one and restarts the cycle on what remains — so a run of any length is squeezed down to one blank line, without ever building a full-file solution in awk.

The hold space matters most once a decision needs information from an earlier line while processing a later one — for example, printing the previous line together with the current one:

sed -n '1!G;h;$p' requests.log

2026-07-29 10:00:04 WARN orders-api 45ms
2026-07-29 10:00:03 ERROR billing-db 890ms
2026-07-29 10:00:02 ERROR orders-api 340ms
2026-07-29 10:00:00 INFO orders-api 12ms

Reading it left to right: 1!G means "on every line except the first, append the hold space to the pattern space," h then copies the (growing) pattern space back into the hold space, and $p prints the final accumulated result only on the last line. The net effect (a classic sed idiom) reverses the file. It is worth recognizing this pattern precisely because it shows up in troubleshooting guides and interview questions about sed internals — not because hand-reversing a file is a common task in itself; tac already does that directly and more clearly.

Interview angle

If asked "how does sed handle more than one line at a time," the answer is the pattern space/hold space model above: sed is fundamentally line-oriented, and N/D plus the hold space are the mechanism it offers for the rare cases that need cross-line context, as opposed to awk, which naturally accumulates state across the whole input in ordinary variables and arrays.

`awk`: Working with Columnar Data

sed sees each line as a whole. awk splits every line into fields by whitespace (or another delimiter you specify), which makes it a much better fit for computing over columns.

Printing Specific Fields

awk '{ print $1, $3 }' requests.log

2026-07-29 INFO
2026-07-29 ERROR
2026-07-29 ERROR
2026-07-29 WARN

$1, $2, and so on are whitespace-delimited fields; $0 is the entire line. By default, the field separator is one or more spaces or tabs.

Filtering by Condition

awk '$3 == "ERROR" { print $1, $2, $4 }' requests.log

2026-07-29 10:00:02 orders-api
2026-07-29 10:00:03 billing-db

Unlike grep, awk does not just find a matching line — it can pick out only the columns you actually want from it.

A Custom Field Separator: `-F`

For CSV or key=value data:

printf 'name=orders-api,status=up,latency=12\nname=billing-db,status=down,latency=340\n' > services.csv
awk -F',' '{ print $1 }' services.csv

name=orders-api
name=billing-db

Totals and Counting: `BEGIN`/`END`

awk '$3 == "ERROR" { count++ } END { print "ERROR count:", count+0 }' requests.log

ERROR count: 2

count+0 forces count to print as 0 rather than an empty string if no ERROR line was ever seen. An awk variable that is never assigned defaults to an empty/zero value.

The BEGIN block runs once before any input is read, and END runs once after all input has been consumed:

awk 'BEGIN { print "Analysis started" }
     $3 == "ERROR" { total += $4 + 0 }
     END { print "Total ERROR latency (ms):", total+0 }' requests.log

Here $4 (for example, 340ms) is not automatically numeric. awk reads the leading numeric portion of the string and ignores the trailing ms, because a value is converted to a number only when it is used in an arithmetic context.

Grouping by a Column

awk '{ count[$3]++ } END { for (level in count) print level, count[level] }' requests.log

INFO 1
ERROR 2
WARN 1

count[$3] is an associative array — every awk array is associative — keyed by the log level. for (level in count) iterates over the array's keys; the iteration order is not guaranteed.

Formatting Output with `printf`

print is convenient but gives little control over column widths or number formatting. awk's printf behaves like the C/shell function of the same name and is the better choice for a report meant to be read by a human:

awk '{ printf "%-12s %-8s %5s\n", $4, $3, $5 }' requests.log

orders-api   INFO      12ms
orders-api   ERROR    340ms
billing-db   ERROR    890ms
orders-api   WARN      45ms

%-12s left-aligns a string in a 12-character field, and %5s right-aligns in a 5-character field — the same format specifiers as C's printf or the shell builtin. Unlike print, printf does not add a trailing newline automatically, so the \n at the end of the format string is required.

Working Across Multiple Files: `FNR` vs `NR`

awk accepts more than one input file, which matters for combining or comparing data from separate sources:

printf 'name=orders-api,status=up,latency=12\nname=billing-db,status=down,latency=340\n' > services.csv
awk -F',' '{ print FILENAME, FNR, NR }' requests.log services.csv

requests.log 1 1
requests.log 2 2
requests.log 3 3
requests.log 4 4
services.csv 1 5
services.csv 2 6

NR is the total record count across every file processed so far; FNR resets to 1 at the start of each new file. FILENAME holds the name of the file currently being read. The combination FNR == 1 is the standard way to detect "a new file has just started," which is useful for printing a per-file header or skipping a header line in each file separately:

awk 'FNR == 1 { print "== " FILENAME " ==" } { print FNR, $0 }' requests.log services.csv

Confusing NR with FNR is a frequent source of bugs when a script that was written and tested against a single file is later pointed at several files — any logic keyed on line number (skipping a header, matching a fixed row) silently breaks because NR keeps counting upward instead of resetting.

When to Choose `sed` versus `awk`

Task	Better tool	Why
Replace text within a single line	`sed`	Treats the line as a whole; simple for straightforward substitution
Update one value in a config file	`sed -i` (carefully)	One line, one substitution — nothing more is needed
Extract or compute over specific columns	`awk`	Splits fields automatically and supports arithmetic and arrays
Sum or tally across multiple conditions	`awk`	`BEGIN`/`END` and associative arrays are built for exactly this
Only find a line, without changing it	`grep`	Both tools can do it, but `grep` is simpler and faster

Both tools are, in full, complete programming languages in their own right. What is shown here covers the everyday sysadmin subset.

Practical Scenario: Rotate a Setting and Summarize the Log

Problem: promote env=staging to env=production in app.conf, but only after confirming exactly one line matches, then produce a per-level event count from requests.log for the change record.

Confirm the substitution target before touching the file:
```
grep -nc 'env=staging' app.conf
```
A count of 1 means the pattern is specific enough to edit safely. A 0 or a 2 and up means stop and refine — sed -i would otherwise change nothing, or rewrite more lines than intended, with no second chance if you skipped the backup.
Edit with a backup, then confirm only the expected line moved:
```
sed -i.bak 's/^env=staging$/env=production/' app.conf
diff app.conf.bak app.conf
```
Anchoring the pattern with ^...$ keeps the substitution from firing on env=staging where it appears as a substring of some longer value.

Summarize the log by level for the change ticket:

awk '{ count[$3]++ } END { for (level in count) printf "%-6s %d\n", level, count[level] }' requests.log

INFO   1
ERROR  2
WARN   1

The division of labor is the whole point: sed made a single, reviewed, reversible edit to one line, and awk computed an aggregate across a column. Reaching for awk to edit the config, or sed to tally the log, would have been the wrong tool in each direction.

Common Mistakes

Applying sed -i directly to a production file without a backup or a prior dry run.
Forgetting the g flag in s/// and not noticing that only the first match on a line changed.
Writing an awk comparison such as $3 == "ERROR" with the string unquoted — quoting matters for string comparisons.
Assuming awk's field numbering matches the file's actual delimiter without checking it (for example, assuming whitespace splitting on a comma-separated file).
Printing an awk accumulator like count or total without +0, then being surprised by an empty result when it was never incremented.
Treating sed and grep as the same tool and reaching for sed -n '/pattern/p' when a plain grep would be simpler.

Exercises

Change debug=true to debug=false in app.conf using sed -i.bak, then confirm the change with diff.
Print only the WARN and ERROR lines from requests.log using sed -n, then reproduce the same result with awk. Compare the two approaches.
Use awk to count how many times each service (orders-api, billing-db) appears in requests.log.
Using sed -E, combine the date and time in requests.log into a single field joined by T, similar to the example above but with your own pattern.

Verification criterion: for each result, you should be able to explain which tool (sed or awk) you chose and why it was the better fit for that specific task.

References

GNU sed manual: https://www.gnu.org/software/sed/manual/sed.html
GNU sed: in-place editing: https://www.gnu.org/software/sed/manual/sed.html#sed-Basics
GNU Awk User's Guide: https://www.gnu.org/software/gawk/manual/gawk.html
POSIX: sed: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sed.html
POSIX: awk: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/awk.html

Rewriting Text: sed and awk

Lab Setup

sed: Rewriting a Stream Line by Line

The Basic Substitution: s///

Replacing Every Occurrence: the g Flag

Editing a File in Place: -i