Streams, Pipes, and Redirection

A single filter that prints the right answer on an interactive terminal can still hide a broken pipeline in a script. The earlier stage might fail while the last command in the chain still exits successfully. When to truncate a target file, where standard error should go, how to check the status of each stage, and how to show output on screen while also saving it to a file — these are separate questions, and getting them wrong is how "the script looked fine yesterday" incidents happen.

This article builds on the standard-stream basics from Viewing Text Files and the searching techniques from Finding Files and Searching Text. The examples assume Bash on Ubuntu Server LTS; where portable POSIX shell behavior differs, that is called out explicitly. You will see why pipeline stages run concurrently, how a plain pipeline's exit status differs from one running under pipefail, how to read every stage's status from PIPESTATUS, how tee shows output while writing it to a file, and how to replace a report file through a temporary file instead of overwriting it directly.

Three Standard Streams, One Byte Pipe

Every process starts with three open file descriptors: 0 (standard input), 1 (standard output), and 2 (standard error). By default, input comes from the keyboard and both output streams go to the terminal. Linux exposes the terminal, a pipe, and a regular file through the same file-descriptor interface, which is exactly what makes redirecting one process's output into another process's input possible.

Lab data:

mkdir -p ~/lab/pipelines
cd ~/lab/pipelines
printf '%s\n' \
    '2026-07-29T10:00:00Z level=INFO service=orders-api' \
    '2026-07-29T10:00:02Z level=ERROR service=orders-api' \
    '2026-07-29T10:00:03Z level=ERROR service=billing-db' \
    '2026-07-29T10:00:04Z level=WARN service=orders-api' > app.log

A pipeline:

grep -F 'level=ERROR' app.log |
    cut -d' ' -f3 |
    sort |
    uniq -c

      1 service=billing-db
      1 service=orders-api

The shell connects every stage, and they normally run at the same time. A pipe carries a stream of bytes through a kernel buffer; it knows nothing about records, JSON structure, or line meaning. Each program in the chain is responsible for interpreting boundaries and encoding on its own.

cut -d' ' -f3 treats a single space as the delimiter. If the log uses variable spacing or quoted fields, this parser breaks. For structured JSON logs, reach for jq; for a stable fixed-field format, awk is usually a better fit, as covered in Rewriting Text: sed and awk.

Where Standard Error Goes: 2>, 2>&1, and Ordering

Because standard output and standard error are separate descriptors, they are redirected separately. > (short for 1>) redirects stdout; 2> redirects stderr:

grep -F 'level=ERROR' app.log missing.log > found.txt 2> errors.txt

Matches land in found.txt; the "No such file or directory" complaint about missing.log lands in errors.txt. Sending both to one place is where order becomes a trap:

grep -F 'level=ERROR' app.log missing.log > combined.txt 2>&1

2>&1 means "make descriptor 2 point at wherever descriptor 1 points right now." Redirections are applied left to right, so > combined.txt must come first: by the time 2>&1 runs, stdout already points at the file, and stderr is duplicated onto that same target. Reverse the two and the result changes:

grep -F 'level=ERROR' app.log missing.log 2>&1 > combined.txt   # errors still hit the terminal

Here 2>&1 copies the terminal (where stdout pointed at that instant), and the later > combined.txt re-points only stdout — so stderr keeps going to the screen while stdout goes to the file. Being able to explain why the two forms differ, rather than reciting one, is the point of this classic screen question.

Bash also offers &> file as a shorthand for "both streams to this file" (equivalent to > file 2>&1), and /dev/null is the kernel's discard sink — 2>/dev/null throws errors away, convenient and dangerous for the same reason: a muted error hides a real failure.

The Hidden Trap in Pipeline Exit Status

In plain Bash, a pipeline's exit status is normally the exit status of its last command:

grep -F 'does-not-exist' app.log | sort
printf 'pipeline status=%s\n' "$?"

grep returns 1 because nothing matched, but sort returns 0 because sorting an empty input is a success. The pipeline as a whole reports 0. This is not automatically a bug — "no matches" can be an entirely expected outcome of a search.

Bash's pipefail option changes that: the pipeline's status becomes the rightmost non-zero stage status, or 0 if every stage succeeded:

(
    set -o pipefail
    grep -F 'does-not-exist' app.log | sort
    printf 'pipefail status=%s\n' "$?"
)

The subshell keeps the option from leaking into your interactive session. pipefail exists in Bash, ksh, and some other shells, but it is not part of mandatory POSIX sh. Know which shell your script's shebang actually targets before relying on it.

Every Stage, Individually: PIPESTATUS

Right after a pipeline finishes, Bash records each stage's exit status in the PIPESTATUS array:

grep -F 'does-not-exist' app.log | sort | wc -l
statuses=( "${PIPESTATUS[@]}" )
printf 'grep=%s sort=%s wc=%s\n' "${statuses[0]}" "${statuses[1]}" "${statuses[2]}"

Approximate output:

0
grep=1 sort=0 wc=0

Copy PIPESTATUS immediately — the very next ordinary command overwrites it. The leading 0 above is wc -l's line count, not a status.

set -e Is Not a Substitute for Checking a Pipeline

set -e (errexit) has notoriously context-dependent rules: its behavior differs inside if, while, &&, ||, pipelines, and command substitution. Do not treat it as "stop on any error, guaranteed."

Checking a critical step explicitly is clearer. Note that $? right after ! command reflects the logically inverted status, not the command's own status — capture the real value immediately after the command instead:

grep -F 'level=ERROR' app.log > errors.log
grep_status=$?
case $grep_status in
    0) echo "matching lines saved" ;;
    1) : > errors.log; echo "no matching lines" ;;
    *) printf 'grep failed: status=%s\n' "$grep_status" >&2; exit "$grep_status" ;;
esac

Here : > errors.log deliberately records an empty result. In a real production report, writing to a temporary file first, as shown later in this article, is safer.

tee: Watch Output and Save It

grep -F 'level=ERROR' app.log | tee errors.log

tee copies its standard input to standard output and to one or more files at once. An operator watching the terminal still sees the result, and the next pipeline stage can consume it too:

grep -F 'level=ERROR' app.log |
    tee errors.log |
    wc -l

Appending instead of truncating:

printf 'manual note\n' | tee -a errors.log

By default tee also truncates an existing target file. sudo command > /root/file fails with a permission error because the plain shell, not sudo, opens that redirection. The common workaround is:

printf '%s\n' '<verified-content>' | sudo tee /etc/<app>/<file> >/dev/null

Grouping and Redirection Scope

Writing several commands' output to one file:

{
    printf 'host=%s\n' "$(hostname)"
    printf 'kernel=%s\n' "$(uname -r)"
    date -u +'generated=%FT%TZ'
} > system-report.txt

{ ...; } groups commands in the current shell; the closing } needs a preceding ; or newline. The redirection applies once, not per line.

A subshell:

(
    cd /etc || exit
    printf 'cwd=%s\n' "$PWD"
    find . -maxdepth 1 -type f -printf '%f\n' | sort
) > etc-report.txt
printf 'outer cwd=%s\n' "$PWD"

(...) runs in a separate subshell, so the cd inside it never changes the parent shell's working directory.

Pipelines and Variable Scope

In Bash, pipeline stages usually run in subshells:

count=0
printf 'a\nb\n' | while IFS= read -r line; do
    ((count++))
done
printf 'count=%s\n' "$count"

count is often still 0 here, because the loop updated its own copy in a child environment. To keep the loop in the current shell, redirect input from a file instead of piping into it:

count=0
while IFS= read -r line; do
    ((count++))
done < app.log
printf 'count=%s\n' "$count"

When you need a command's output rather than a file, Bash process substitution avoids the subshell problem:

count=0
while IFS= read -r line; do
    ((count++))
done < <(grep -F 'level=ERROR' app.log)
printf 'ERROR lines=%s\n' "$count"

<(...) provides a file-like path and is not portable to POSIX sh. Checking the inner grep's exit status is awkward in this form; for anything that must be checked reliably, use a separate check or a temporary file instead.

Here-Documents and Here-Strings

Multi-line standard input:

sort <<'EOF'
gamma
alpha
beta
EOF

Because the delimiter is quoted, $variable, $(command), and backslash expansions inside the body are not performed. That protects a configuration template from unexpected command substitution.

A Bash here-string:

grep -F 'ERROR' <<< 'level=ERROR service=orders-api'

<<< is Bash-specific and appends a trailing newline. For a large amount of data, use a file or a pipe instead of holding everything in a variable.

SIGPIPE and an Early-Exiting Consumer

find /usr/bin -maxdepth 1 -type f -print | head -n 5

head closes after five lines. If the left-hand process then writes to a pipe with no reader, it can receive SIGPIPE or report a "Broken pipe" error. With pipefail enabled, this pipeline may report a non-zero status even though the user got exactly the five lines they wanted.

In other words, pipefail makes the condition visible, but it does not automatically decide whether every non-zero result is an incident. Test whether an early-exiting consumer is expected and how the producer's exit status behaves in that case. When only the first result is needed, GNU find ... -print -quit avoids the extra work entirely:

find /usr/bin -maxdepth 1 -type f -print -quit

Building a Report Near-Atomically

command > report.txt discards the previous report the instant it opens the file. If the command then fails partway through, a consumer may read a half-written file and assume it is complete. Writing to a temporary file in the same file system, validating it, and only then renaming it into place is safer:

report=$HOME/lab/pipelines/error-summary.txt
tmp=$(mktemp "$HOME/lab/pipelines/.error-summary.XXXXXX") || exit 1

if grep -F 'level=ERROR' app.log |
    cut -d' ' -f3 |
    sort |
    uniq -c > "$tmp"; then
    test -s "$tmp" || echo "report is empty" >&2
    mv -f -- "$tmp" "$report"
else
    status=$?
    rm -f -- "$tmp"
    printf 'report was not generated, status=%s\n' "$status" >&2
    exit "$status"
fi

cat "$report"

pipefail is not enabled in this example; if you also need to catch a grep or intermediate-stage failure, enable set -o pipefail inside a controlled Bash subshell. mv is normally an atomic rename within a single file system, but durability, permissions, and other processes' locking expectations are a separate concern.

A rollback is restoring the previously saved copy of the report. For anything that matters:

1. inspect the existing file's metadata and content;
2. take a timestamped backup with cp -a;
3. write to a temporary file;
4. run the application's own validator;
5. replace the file within the same file system;
6. reload and run a health check;
7. restore the backup if any step fails.

Practical Scenario: A Verifiable Log Summary

Problem: count ERROR events by service, show the result on screen, and save it to a file.

log=$HOME/lab/pipelines/app.log
report=$HOME/lab/pipelines/service-errors.txt
test -r "$log" || {
    echo "cannot read log: $log" >&2
    exit 1
}

(
    set -o pipefail
    grep -F 'level=ERROR' "$log" |
        cut -d' ' -f3 |
        sort |
        uniq -c |
        tee "$report"
)
status=$?
printf 'pipeline status=%s\n' "$status"

Verification:

test -s "$report" && echo "report is not empty"
awk 'NF != 2 { bad=1 } END { exit bad }' "$report"
printf 'format status=%s\n' "$?"

If the real log format is not simple space-delimited text, replace the parser accordingly. Check that no sensitive value remains in the report before sharing it.

Common Mistakes

• Assuming a pipeline's exit status reflects every stage's success.
• Enabling pipefail without also analyzing what grep status 1 or an expected SIGPIPE actually means.
• Reading PIPESTATUS after running another command that overwrites it.
• Treating set -e as a full substitute for explicit status checks.
• Forgetting that tee truncates an existing file by default.
• Expecting a variable changed inside a piped while loop to survive in the parent shell.
• Leaving a here-document delimiter unquoted and triggering unwanted expansion.
• Writing a critical report directly to its final destination.
• Never checking that a parser's assumed delimiter actually matches the log format.

Exercises

1. Build a three-stage pipeline, make the middle stage fail deliberately, and compare the default status against the status under pipefail.
2. Save all three stage statuses into an array using PIPESTATUS immediately after the pipeline runs.
3. Use tee to show a result on screen while saving it to a file, then verify the difference -a makes on a second run.
4. Compare the final value of a counter variable between a piped while read loop and one using input redirection.
5. Update the lab report using the mktemp → validate → mv flow, then deliberately break the generation step and confirm the old file is preserved.

Verification criterion: you should be able to show the normal output, each stage's exit status, the target file's integrity, and the rollback path, all at once.

References

• GNU Bash Reference Manual: Pipelines: https://www.gnu.org/software/bash/manual/html_node/Pipelines.html
• GNU Bash Reference Manual: The Set Builtin: https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
• GNU Bash Reference Manual: Bash Variables (PIPESTATUS): https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html
• GNU Bash Reference Manual: Process Substitution: https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html
• GNU Bash Reference Manual: Here Documents: https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Here-Documents
• GNU Coreutils manual: tee invocation: https://www.gnu.org/software/coreutils/manual/html_node/tee-invocation.html
• POSIX: Pipes: https://pubs.opengroup.org/onlinepubs/9799919799/functions/pipe.html