Bash - input/output redirections and pipes
Bash (Bourne Again Shell) - Written by Brian Fox with a help of amazing volunteers from the GNU project was created as a open-source, GNU alternative for, now quite dated "sh - shell" (which is still used to this day, but this is topic for another issue). First released in 1989, in the 30 years history of this software it has become the de facto standard where it comes to shell running inside of linux terminal. Now, in year 2020 Bash is pretty much everywhere where we hear the term linux shell. Its popularity comes from it being the default shell in most of the linux distributions these days and this is not without a reason. Many people like bash for its being an easy to use and learn shell with good potential for advance users. That doesn't mean that Bash is the best shell.
There are many alternatives to Bash promissing more advance functionality and all around friendliness than Bash and I'm not saying that those alternatives are lying about that so you should also check them out and maybe learn them as well. Here I'm covering topics that are related to Bash and not the alternatives, just because Bash is the most popular shell and you will be using this shell most of the time you use terminal. Bash is not just a shell, but it is also a scripting language which can be used for creating advance scripts that can do multiple things at once and this is partly thanks to things called pipes and redirections which are the topic of today's issue.
The "shell"
Let's start with what the "shell" is and how it actuallly works. Shell is the underlying command interpreter which reads the input from the user through a piece of software called the terminal, parses the input, talks to the OS which runs an appropriate program with passed arguments and at the end sends output again to the terminal. To do all this things shell uses three channels of a process (every process running inside your linux box has them) to communicate with it, these channels are called streams. Streams are used to connect and interact with a terminal or a process. These are the three main streams:
- Standard input (is eg. your keyboard) - has a file descriptor 0,
- Standard output (is the stream which is used to transfer everything you see in the terminal) - has a file descriptor 1,
- Standard error (output) (same as standard output but used only when something went wrong. If so, it returns a value of 1 to indicate to bash that command failed and it shouldn't continue running a script in this case. This behaviour is present when we use logical AND, again topic for another issue.) - has a file descriptor 2.
Terminal has an main task of reading input from the user, passing the user input to the shell and displaying what it gets from the shell.
How command line works
When you type a command into your terminal emulator of choice, for example:
echo "Hello, World!"
It then sends this command to the shell you are running (eg. Bash) in a form of an standard input. Bash then talks to the Operating System in order to initiate new process running echo binary, provides command's parameters and waits. Echo binary then executes your command and as in this example, returns what you give as a parameter together with an integer called a return value. Bash then checks what return value was returned and, taking this to account, returns what it got from echo to one of the output streams:
- standard output (when return value was 0), or
- standard error (output) (when return value was 1).
Then the terminal's job is to display you the message.
Now that you roughly know how command line works, we can go on and talk about Bash pipes and redirections.
Bash pipes
Pipes in Bash are represented by the | character. Pipes are used to redirect output (standard output and standard error) from one command to (standard input of) another. Here is an example:
Let's say you want to get information about a file you know is in your cwd (current working directory, you can check it using: pwd; command.) and you know the name for. In this case you can use the ls command with a few parameters to list all files in detailed way
[username@username directory]$ ls -lah
total 1,9M
drwxr-xr-x 2 username username 4,0K 10-05 22:35 .
drwxr-xr-x 5 username username 24K 10-05 22:34 ..
-rw-r--r-- 1 username username 40 10-05 22:35 asdf.py
-rw-r--r-- 1 username username 24K 10-05 22:35 logo1.png
-rw-r--r-- 1 username username 1,8M 10-05 22:35 test1.wav
and then you can search those information using pipes and for example grep.
[username@username directory]$ ls -lah | grep asdf.py
-rw-r--r-- 1 username username 40 10-05 22:35 asdf.py
Quick overview what happened here:
ls -lah displayed a whole list of information about every file in this directory, it then output the result into the standard output, so file descriptor 1, which was then piped into the standard input, so file descriptor 0, of grep which then searched what it was given through pipe and returned only line which contained phrase: asdf.py
You can also combine couple of pipes into one command. Here is an example of how to display only the permissions of an asdf.py file:
[username@username directory]$ ls -lah | grep asdf.py | awk '{print $1}'
-rw-r--r--
Bash input/output redirections
Redirections, on the other hand, are represented by the < or > character. They are used to, as the name suggests, redirect output from a command to a file or from a file to a command. You can think of those characters as arrows pointing the way which the data is flowing.
And now some examples:
Let's start and say you want to get a list of files and directories in your current working directory and you want to save them into a file. Here is how to do that:
[username@username directory]$ ls > files.txt
[username@username directory]$ cat files.txt
files.txt
file.txt
image.jpg
sound.mp3
As you can see, names of all the files, generated by the ls command to a standard output - file descriptor 1, were redirected to a file called files.txt and then showed to us by using the cat (concatenate) command.
The next example will be as follows: let's suppose you also want to have the same file structure list but with more information and you want it to be in the same file as the "simple" list is. To do that we need to use a "trick" called appending to a file. To append standard output to a file you will use the " >> " character. The syntax is the same as if you were using redirects. Here is the solution:
[username@username directory]$ ls -lah >> files.txt
[username@username directory]$ cat files.txt
files.txt
file.txt
image.jpg
sound.mp3
total 20K
drwxr-xr-x. 2 username username 73 Oct 7 21:11 .
drwx------. 16 username username 4.0K Oct 7 21:03 ..
-rw-rw-r--. 1 username username 39 Oct 7 21:11 files.txt
-rw-rw-r--. 1 username username 47 Oct 7 21:08 file.txt
-rw-rw-r--. 1 username username 404 Oct 7 21:09 image.jpg
-rw-rw-r--. 1 username username 354 Oct 7 21:09 sound.mp3
Just remember, that if you want to add something to already existing and not empty file, you must use the append " >> " symbol instead of the redirect " > " one. If you were by mistake use the wrong one, you could easly empty an important file, so be very careful with that.
Now I will show you another "trick" called the streams redirections. In this example we'll suppose that you have a command that returns an error but also a bunch of useful information as well. In this case we'll use the file descriptor number to tell Bash to only redirect single stream to a file. Here is the command:
[username@username directory]$ find / -name usr 2>/dev/null
/usr
/usr/lib/debug/usr
/usr/share/doc/oddjob/sample/usr
/usr/share/gdb/auto-load/usr
/usr/src/kernels/4.18.0-193.19.1.el8_2.x86_64/usr
I know that if you are not familiar with this stuff, this example can be weird to you. So here is the explanation: Find command searches in the whole file system (/) for the file or directory which contains the word "usr" and shows us the findings. So why we need the stream redirections here you may ask. That's because if we are loged in as normal user we do not have permissions to access some of the directories on the file system and that will generate an error which we deal with using the stream redirection. Just to ensure you, number 2 is for file descriptor 2 which is the file descriptor of standard error. But what is the /dev/null part you may ask? Good question. That means that we redirect that error output to an special file which is throwing what we give to it to the void.
I just want to point out, that command like:
find / -name usr 1> output.txt`
is doing the same thing as the same command without the "1":
find / -name usr > output.txt
and that's because the > character redirects standard output just like the number 1, so the standard output redirection will.
Now I'll explain how the other way redirects work. First I'll give you the command and then explain what happened.
[username@username directory]$ grep word < file.txt
word1
word2
Here we have the file.txt file which contains couple of lines with the "word1" and "word2" within them. Grep command as you know searches for a string in a given input. Using the "<" character (arrow pointing to the left) redirects content of the file.txt file to the grep program so it can do it's job. That's it.
And the last example will be how to use couple of redirects at once (in one command - "one liner"). Command:
grep word < file.txt > result.txt
First part is the same as before, we search for "word" phrase inside of a file.txt file. The only thing that has been changed is the last part, which redirects what grep found into the result.txt file.
Ending
So, there we go! You now understand what the Bash (shell) and command line is and how it works, you know what the pipes are and how they work as well as what the input/output redirections in bash are and how to use them! That's quite a lot of knowledge I'd say. Just know that I did not exahusted those topics, and there is way more to learn here, which I personally like to hear! Thank you for reading and I hope you learned something from this. Don't forget to check out my other posts too! :)