Pipes and IO redirection in Linux and Unix

In this tutorial we will talk about redirection of standard streams, stdin, stdout, stderr in Unix-like operating systems. The Unix philosophy is to write programs that do one thing, but do it well and write them to work together, so the output of a program can be used as input to another, this is great because we can transfer data continuously from a program to another without manually feeding data into each one.

A real life example of the diagram 1.0 using the ps and grep commands.

The ps aux will show every process running on your system, while the grep bash will print the lines that matches the “bash” string. The pipe operator “|” will feed the output of the ps command (stdout) into the input of the grep command (stdin), so the result will be every process that includes the “bash” string.

tester     949  0.0  0.1  20920  4720 pts/0    S    12:00   0:00 bash
tester     956  0.0  0.0  12784   956 pts/0    S+   12:00   0:00 grep bash

But wait.. what about stderr?

Run the following find command as normal user, we will search for the ip_forward file in the /proc directory and match the “ipv4” string with grep.

...
find: `/proc/3/fdinfo': Permission denied
find: `/proc/3/ns': Permission denied
/proc/sys/net/ipv4/ip_forward
find: `/proc/94/task/94/fd': Permission denied
find: `/proc/94/task/94/fdinfo': Permission denied
...

The result is not what we might expect, because we used grep to match only the lines with the “ipv4” string. The explanation is that the lines containing the “Permission denied” string don’t reach the grep command, they are errors which by default goes to display, through the stderr stream (see diagram 1.0). This example can be fixed by redirecting the stderr to stdout using the redirection operator “>“.

/proc/sys/net/ipv4/ip_forward

Back to the basics.

Standard streams can be found in many environments, shortly they are input and output connections between a process and a device or pseudo-device. Each stream is mapped to a file descriptor (FD), a non-negative integer assigned by the operating system. Linux and Unix handles the IO using three streams, check out the table from below.

Standard Streams
FD	Name	Short name	Description
0	Standard Input	stdin	The program reads data from a physical device, usually a keyboard or redirection.
1	Standard Output	stdout	Where a program writes its output data.
2	Standard Error	stderr	Where a program writes its error messages.

Redirect to another program via pipeline, run the commands as normal user.

| Redirect stdout to grep's stdin, stderr goes to display.

...
/proc/sys/net/ipv4/ip_forward
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/fdinfo': Permission denied
...

|& Redirect stdout and stderr to grep, this is a shorthand for "2>&1 |".

/proc/sys/net/ipv4/ip_forward

2>&1 1>/dev/null Redirect stderr to stdout and send find's stdout to /dev/null, literally send stderr to grep and throw find's stdout away.

...
find: `/proc/29353/map_files': Permission denied
find: `/proc/33761/map_files': Permission denied
find: `/proc/33898/map_files': Permission denied
find: `/proc/33902/map_files': Permission denied
find: `/proc/33971/map_files': Permission denied
find: `/proc/37080/map_files': Permission denied

Read data from file using the “<” operator.

Redirect to file.

> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

&> Redirect stdout and stderr to a file, creates the file or overwrites it.

&>> Redirect stdout and stderr to a file, creates the file or appends to it.

Redirect to file, explicitly.

1> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

1>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

2> Redirect stderr to a file, creates the file or overwrites it, stdout goes to display.

2>> Redirect stderr to a file, creates the file or appends to it, stdout goes to display.

Redirect to file, explicitly, combined.

Redirect stdout and stderr to different files, creates the files or overwrites it.

Redirect stdout and stderr to different files, creates the files or appends to it.