The sed command in Unix and Linux and its common usage

In my last post on shell programming, we have seen how to use grep command to search patterns in files and print them, coupled with some useful text manipulation commands with grep with other commands like sed. The grep command used with other commands or piped to other commands is something we should avoid as we become a mature UNIX programmer because when the pattern matching task becomes more sophisticated, one should use specialized commands like sed to accomplish the task.

I always wanted to explore sed as using grep command beyond whats it there for always resulted in an unreadable code, therefore I explored sed for sometime and decided to list some common applications of sed command in pattern matching and text manipulation which I hope will be useful for other UNIX programmers also.

As always, for some customary introduction about sed (stream editor), its a command used to read text files line by line and search, substitute, remove lines matching a pattern and does many other interesting text manipulation functions. It outputs the transformed text to the console by default, although you can redirect it to another file or selectively supress it using one of the sed options. Most applications of sed command were in text substitution, but its not limited to it.

The sed command ....

sed options sed-script filename

Though a man page will help you in this, some common options in sed are listed below

sed -e filename - Executes an inline sed-script on the input file (although it can be used to execute multiple line sed commands by using \ separator, the wise way is to use a sed script file for that).
sed -f filename - Execute the series of sed commands in sed-script-file on the input file
sed -n filename - Doesn't print the result of the sed command to the console by default, instead one should use the "/p" flag inside the sed command/script to print the desired output.

Applications of sed command:

sed command can be used to search and manipulate input text in many ways, though mainly used for substitution, it can also do numerous other pattern matching tasks like searching and printing lines which matches a pattern, removing lines which matches a pattern, deleting specific lines in the input file, etc, some common uses of sed command were shown below.

For the sake of simplicity and to make life easy, we won't be using any of the sed command line options described above, starting from simple ones, here we go.

The text file used to illustrate the below sed commands is shown below.

bash-3.00# cat textfile.txt
This is line 1
This is line 2
This is line 3
This is line 4
This is line 5
bash-3.00#
 

1. Deleting lines in a file using sed command

1.1. Deleting the first line in the input file

bash-3.00# sed '1d' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5

1.2 Deleting the last line in the input file

bash-3.00# sed '$d' textfile.txt
This is line 1
This is line 2
This is line 3
This is line 4
bash-3.00#

1.3 Deleting arbitary lines in the input file

To delete the lines 2 through 4 in the above file, use the following sed command

bash-3.00# sed '2,4d' textfile.txt
This is line 1
This is line 5

the 'd' flag is used to delete lines, though the single quote is not mandatory in the above commands (you can simply use sed 1d textfile.txt to delete the first line in the input file), its advisable to use it as a good practice and will be useful when you use the -e option at times to execute a series of sed commands inline.

2. Replace or substitute all occurrences of a pattern using sed command

In the above text file, if you need to replace all occurrences of the pattern "line" with the text "line number", the following sed command can be used (the 's' flag in the sed command is used to substitute text which matches a pattern).

bash-3.00# sed 's/line/line number/g' textfile.txt        
This is line number 1
This is line number 2
This is line number 3
This is line number 4
This is line number 5

Note: The /g flag in the above command is used to instruct sed to substitute globally i.e all occurrences of the input pattern in the line will be replaced and as a result all words matching the pattern in the input file will be substituted by the other, without /g only the first word matching the pattern in a line will be replaced.


3. Delete all occurrences of a pattern in a file using sed command

This is a simple application of the /s flag where the target string is none.

For example, to remove all occurrences of word "line" in the above file, the sed command would be

bash-3.00# sed 's/line//' textfile.txt
This is  1
This is  2
This is  3
This is  4
This is  5

Note that the word to be replaced upon matching the pattern is none in the above command ('/s/line//')

To remove all occurrences of the character space in the above input file, the sed command would be

bash-3.00# sed 's/ //g' textfile.txt
Thisisline1
Thisisline2
Thisisline3
Thisisline4
Thisisline5

Also note that if you don't use single quotes in the above sed command, then the space pattern should be enclosed in quotes for the command to work.

bash-3.00# sed s/' '//g textfile.txt
Thisisline1
Thisisline2
Thisisline3
Thisisline4
Thisisline5

4. Deleting lines which matches a pattern using sed command

To delete lines which matches a pattern, use the /d flag along with the pattern to be matched, for example, to delete lines matching the pattern "line 1" in the above file, the command would be

bash-3.00# sed '/line 1/d' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5

Now using the above commands, one can also delete the first and last line in a file which matches a pattern.

5. Deleting the first line which matches a pattern using sed command

bash-3.00# sed '/line/{1d;}' textfile.txt
This is line 2
This is line 3
This is line 4
This is line 5

In the above command, '1d' is used in braces to indicate that its a command (to delete the specific line, which matches a pattern), this is the way to separate sed flags in command/scripts.

6. Deleting the last line which matches a pattern using sed command

bash-3.00# sed '/line/{$d;}' textfile.txt
This is line 1
This is line 2
This is line 3
This is line 4

Hope you got to know some common sed command functionality, the next time if you think about sed instead of grep/sed pipe for text processing, you have improved a level in UNIX shell scripting.