Category: Computer Programming

Discuss Sed ”; Previous Next This tutorial takes you through all about Stream EDitor (Sed), one of the most prominent text-processing utilities on GNU/Linux. Similar to many other GNU/Linux utilities, it is stream-oriented and uses simple programming language. It is capable of solving complex text processing tasks with few lines of code. This easy, yet powerful utility makes GNU/Linux more interesting. Print Page Previous Next Advertisements ”;

Stream Editor – Quick Guide ”; Previous Next Stream Editor – Overview The acronym SED stands for Stream EDitor. It is a simple yet powerful utility that parses the text and transforms it seamlessly. SED was developed during 1973–74 by Lee E. McMahon of Bell Labs. Today, it runs on all major operating systems. McMahon wrote a general-purpose line-oriented editor, which eventually became SED. SED borrowed syntax and many useful features from ed editor. Since its beginning, it has support for regular expressions. SED accepts inputs from files as well as pipes. Additionally, it can also accept inputs from standard input streams. SED is written and maintained by the Free Software Foundation (FSF) and it is distributed by GNU/Linux. Hence it is often referred to as GNU SED. To a novice user, the syntax of SED may look cryptic. However, once you get familiar with its syntax, you can solve many complex tasks with a few lines of SED script. This is the beauty of SED. Typical Uses of SED SED can be used in many different ways, such as: Text substitution, Selective printing of text files, In-a-place editing of text files, Non-interactive editing of text files, and many more. Stream Editor – Environment This chapter describes how to set up the SED environment on your GNU/Linux system. Installation Using Package Manager Generally, SED is available by default on most GNU/Linux distributions. Use which command to identify whether it is present on your system or not. If not, then install SED on Debian based GNU/Linux using apt package manager as follows: [jerry]$ sudo apt-get install sed After installation, ensure that SED is accessible via command line. [jerry]$ sed –versio On executing the above code, you get the following result: sed (GNU sed) 4.2.2 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Jay Fenlason, Tom Lord, Ken Pizzini, and Paolo Bonzini. GNU sed home page: . General help using GNU software: . E-mail bug reports to: . Be sure to include the word “sed” somewhere in the “Subject:” field. Similarly, to install SED on RPM based GNU/Linux, use yum package manager as follows: [root]# yum -y install sed After installation, ensure that SED is accessible via command line. [root]# sed –version On executing the above code, you get the following result: GNU sed version 4.2.1 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE, to the extent permitted by law. GNU sed home page: . General help using GNU software: . E-mail bug reports to: . Be sure to include the word “sed” somewhere in the “Subject:” field. Installation from Source Code As GNU SED is a part of the GNU project, its source code is available for free download. We have already seen how to install SED using package manager. Let us now understand how to install SED from its source code. The following installation is applicable to any GNU/Linux software, and for most other freely-available programs as well. Here are the installation steps: Download the source code from an authentic place. The command-line utility wget serves this purpose. [jerry]$ wget ftp://ftp.gnu.org/gnu/sed/sed-4.2.2.tar.bz2 Decompress and extract the downloaded source code. [jerry]$ tar xvf sed-4.2.2.tar.bz2 Change into the directory and run configure. [jerry]$ ./configure Upon successful completion, the configure generates Makefile. To compile the source code, issue a make command. [jerry]$ make You can run the test suite to ensure the build is clean. This is an optional step. [jerry]$ make check Finally, install the SED utility. Make sure you have superuser privileges. [jerry]$ sudo make install That is it! You have successfully compiled and installed SED. Verify it by executing the sed command as follows: [jerry]$ sed –version On executing the above code, you get the following result: sed (GNU sed) 4.2.2 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Jay Fenlason, Tom Lord, Ken Pizzini, and Paolo Bonzini. GNU sed home page: . General help using GNU software: . E-mail bug reports to: . Be sure to include the word “sed” somewhere in the “Subject:” field. Stream Editor – Workflow In this chapter, we will explore how SED exactly works. To become an expert SED user, one needs to know its internals. SED follows a simple workflow: Read, Execute, and Display. The following diagram depicts the workflow. Read: SED reads a line from the input stream (file, pipe, or stdin) and stores it in its internal buffer called pattern buffer. Execute: All SED commands are applied sequentially on the pattern buffer. By default, SED commands are applied on all lines (globally) unless line addressing is specified. Display: Send the (modified) contents to the output stream. After sending the data, the pattern buffer will be empty. The above process repeats until the file is exhausted. Points to Note Pattern buffer is a private, in-memory, volatile storage area used by the SED. By default, all SED commands are applied on the pattern buffer, hence the input file remains unchanged. GNU SED provides a way to modify the input file in-a-place. We will explore about it in later sections. There is another memory area called hold buffer which is also private, in- memory, volatile storage area. Data can be stored in a hold buffer for later retrieval. At the end of each cycle, SED removes the contents of the pattern buffer but the contents of the hold buffer remains persistent between SED cycles. However SED commands cannot be directly executed on hold buffer, hence SED allows data movement between the hold buffer and

Stream Editor – Special Characters ”; Previous Next SED provides two special characters which are treated as commands. This chapter illustrates the usage of these two special characters. = Command The “=” command deals with line numbers. Given below is the syntax of the “=” command: [/pattern/]= [address1[,address2]]= The = command writes the line number followed by its contents on the standard output stream. The following example illustrates this. [jerry]$ sed ”=” books.txt On executing the above code, you get the following result: 1 1) A Storm of Swords, George R. R. Martin, 1216 2 2) The Two Towers, J. R. R. Tolkien, 352 3 3) The Alchemist, Paulo Coelho, 197 4 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 5 5) The Pilgrimage, Paulo Coelho, 288 6 6) A Game of Thrones, George R. R. Martin, 864 Let us print the line numbers and the contents of the first four lines. The following command prints the first four lines with line numbers and the remaining without line numbers. [jerry]$ sed ”1, 4=” books.txt On executing the above code, you get the following result: 1 1) A Storm of Swords, George R. R. Martin, 1216 2 2) The Two Towers, J. R. R. Tolkien, 352 3 3) The Alchemist, Paulo Coelho, 197 4 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 5) The Pilgrimage, Paulo Coelho, 288 6) A Game of Thrones, George R. R. Martin, 864 Additionally, we can instruct the SED to print line numbers when a pattern match succeeds. The following example prints the line number that contains the pattern “Paulo”. [jerry]$ sed ”/Paulo/ =” books.txt On executing the above code, you get the following result: 1) A Storm of Swords, George R. R. Martin, 1216 2) The Two Towers, J. R. R. Tolkien, 352 3 3) The Alchemist, Paulo Coelho, 197 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 5 5) The Pilgrimage, Paulo Coelho, 288 6) A Game of Thrones, George R. R. Martin, 864 Can you guess what the following SED command does? [jerry]$ sed -n ”$ =” books.txt On executing the above code, you get the following result: 6 Yes, you are right. It counts the total number of lines present in the file. Let us demystify the code. In the command section, we used “$ =” which prints the line number of the last line followed by its contents. But we also provided the -n flag which suppresses the default printing of the pattern buffer. Hence, only the last line number is displayed. & Command SED supports the special character &. Whenever a pattern match succeeds, this special character stores the matched pattern. It is often used with the substitution command. Let us see how we can leverage this efficient feature. Each line in the book.txt file is numbered. Let us add the words Book number at the beginning of each line. The following example illustrates this. [jerry]$ sed ”s/[[:digit:]]/Book number &/” books.txt On executing the above code, you get the following result: Book number 1) A Storm of Swords, George R. R. Martin, 1216 Book number 2) The Two Towers, J. R. R. Tolkien, 352 Book number 3) The Alchemist, Paulo Coelho, 197 Book number 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 Book number 5) The Pilgrimage, Paulo Coelho, 288 Book number 6) A Game of Thrones, George R. R. Martin, 864 This example is very simple. First, we search for the first occurrence of a digit, which is the line number (that is why we used [[:digit:]]) and the SED automatically stores the matched pattern in the special character &. In the second step, we insert the words Book number before each matched pattern, i.e., before every line. Let us take another example. In the book.txt file, the last digit implies the number of pages of the book. Let us add “Pages =” before that. To do this, find the last occurrence of the digit and replace it with “Pages = &”. Here, & stores the matched pattern, i.e., the number of pages [jerry]$ sed ”s/[[:digit:]]*$/Pages = &/” books.txt On executing the above syntax, you get the following result: 1) A Storm of Swords, George R. R. Martin, Pages = 1216 2) The Two Towers, J. R. R. Tolkien, Pages = 352 3) The Alchemist, Paulo Coelho, Pages = 197 4) The Fellowship of the Ring, J. R. R. Tolkien, Pages = 432 5) The Pilgrimage, Paulo Coelho,Pages = 288 6) A Game of Thrones, George R. R. Martin, Pages = 864 For the time being, just remember that [[:digit:]]*$ finds the last occurrence of the digit. In the chapter “Regular Expressions, we will explore more about regular expressions. Print Page Previous Next Advertisements ”;

Stream Editor – Managing Patterns ”; Previous Next We have already discussed the use of pattern and hold buffer. In this chapter, we are going to explore more about their usage. Let us discuss the n command which prints the pattern space. It will be used in conjunction with other commands. Given below is the syntax of then command. [address1[,address2]]n Let us take an example. [jerry]$ sed ”n” books.txt When the above code is executed, it will produce the following result: 1) A Storm of Swords, George R. R. Martin, 1216 2) The Two Towers, J. R. R. Tolkien, 352 3) The Alchemist, Paulo Coelho, 197 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 5) The Pilgrimage, Paulo Coelho, 288 6) A Game of Thrones, George R. R. Martin, 864 The n command prints the contents of the pattern buffer, clears the pattern buffer, fetches the next line into the pattern buffer, and applies commands on it. Let us consider there are three SED commands before n and two SED commands after n as follows: Sed command #1 Sed command #2 Sed command #3 n command Sed command #4 Sed command #5 In this case, SED applies the first three commands on the pattern buffer, clears the pattern buffer, fetches the next line into the pattern buffer, and thereafter applies the fourth and fifth commands on it. This is a very important concept. Do not go ahead without having a clear understanding of this. The hold buffer holds data, but SED commands cannot be applied directly on the hold buffer. Hence, we need to bring the hold buffer data into the pattern buffer. SED provides the x command to exchange the contents of pattern and hold buffers. The following commands illustrate the x command. Let us slightly modify the books.txt file. Say, the file contains book titles followed by their author names. After modification, the file should look like this: [jerry]$ cat books.txt On executing the above code, you get the following result: A Storm of Swords George R. R. Martin The Two Towers J. R. R. Tolkien The Alchemist Paulo Coelho The Fellowship of the Ring J. R. R. Tolkien The Pilgrimage Paulo Coelho A Game of Thrones George R. R. Martin Let us exchange the contents of the two buffers. For instance, the following example prints only the names of authors. [jerry]$ sed -n ”x;n;p” books.txt On executing the above code, you get the following result: George R. R. Martin J. R. R. Tolkien Paulo Coelho J. R. R. Tolkien Paulo Coelho George R. R. Martin Let us understand how this command works. Initially, SED reads the first line, i.e., A Storm of Swords into the pattern buffer. x command moves this line to the hold buffer. n fetches the next line, i.e., George R. R. Martin into the pattern buffer. The control passes to the command followed by n which prints the contents of the pattern buffer. The process repeats until the file is exhausted. Now let us exchange the contents of the buffers before printing. Guess, what happens? Yes, it prints the titles of books. [jerry]$ sed -n ”x;n;x;p” books.txt On executing the above code, you get the following result: A Storm of Swords The Two Towers The Alchemist The Fellowship of the Ring The Pilgrimage A Game of Thrones The h command deals with the hold buffer. It copies data from the pattern buffer to the hold buffer. Existing data from the hold buffer gets overwritten. Note that the h command does not move data, it only copies data. Hence, the copied data remains as it is in the pattern buffer. Given below is the syntax of the h command. [address1[,address2]]h The following command prints only the titles of the author Paulo Coelho. [jerry]$ sed -n ”/Paulo/!h; /Paulo/{x;p}” books.txt On executing the above code, you get the following result: The Alchemist The Pilgrimage Let us understand how the above command works. The contents of books.txt follow a specific format. The first line is the book title followed by the author of the book. In the above command, “!” is used to reverse the condition, i.e., line is copied to the hold buffer only when a pattern match does not succeed. And curly braces {} are used to group multiple SED commands In the first pass of the command, SED reads the first line, i.e., A Storm of Swords into the pattern buffer and checks whether it contains the pattern Paulo or not. As the pattern match does not succeed, it copies this line to the hold buffer. Now both the pattern buffer and the hold buffer contain the same line i.e., A Storm of Swords. In the second step, it checks whether the line contains the pattern Paulo or not. As the pattern does not match, it does not do anything. In second pass, it reads the next line George R. R. Martin into the pattern buffer and applies the same steps. For the next three lines, it does the same thing. At the end of the fifth pass, both the buffers contain The Alchemist. At the start of the sixth pass, it reads the line Paulo Coelho and as the pattern matches, it does not copy this line into the hold buffer. Hence, the pattern buffer contains Paulo Coelho, and the hold buffer contains The Alchemist. Thereafter, it checks whether the pattern buffer contains the pattern Paulo. As the pattern match succeeds, it exchanges the contents of the pattern buffer with the hold buffer. Now the pattern buffer contains The Alchemist and the hold buffer contains Paulo Coelho. Finally, it prints the contents of the pattern buffer. The same steps are applied to the pattern The Pilgrimage. The h command destroys the previous contents of the hold buffer. This is not always acceptable, as sometimes we need to preserve the contents. For this purpose, SED provides the H command which appends the contents to the hold buffer

Stream Editor – Strings ”; Previous Next Substitute Command Text substitution operations like “find and replace” are common in any text editor. In this section, we illustrate how SED performs text substitution. Given below is the syntax of the substitution command. [address1[,address2]]s/pattern/replacement/[flags] Here, address1 and address2 are the starting and ending addresses respectively, which can be either line numbers or pattern strings. Both these addresses are optional parameters. The pattern is the text which we want to replace with the replacement string. Additionally, we can specify optional flags with the SED. In the books.txt file, we have used comma(,) to separate each column. Let us use vertical bar(|) to separate each column. To do this, replace comma(,) with vertical bar(|). [jerry]$ sed ”s/,/ | /” books.txt On executing the above code, you get the following result: 1) A Storm of Swords | George R. R. Martin, 1216 2) The Two Towers | J. R. R. Tolkien, 352 3) The Alchemist | Paulo Coelho, 197 4) The Fellowship of the Ring | J. R. R. Tolkien, 432 5) The Pilgrimage | Paulo Coelho, 288 6) A Game of Thrones | George R. R. Martin, 864 If you observe carefully, only the first comma is replaced and the second remains as it is. Why? As soon as the pattern matches, SED replaces it with the replacement string and moves to the next line. By default, it replaces only the first occurrence. To replace all occurrences, use the global flag (g) with SED as follows: [jerry]$ sed ”s/,/ | /g” books.txt On executing the above code, you get the following result: 1) A Storm of Swords | George R. R. Martin | 1216 2) The Two Towers | J. R. R. Tolkien | 352 3) The Alchemist | Paulo Coelho | 197 4) The Fellowship of the Ring | J. R. R. Tolkien | 432 5) The Pilgrimage | Paulo Coelho | 288 6) A Game of Thrones | George R. R. Martin | 864 Now all occurrences of commas(,) are replaced with vertical bar(|). We can instruct the SED to perform text substitution only when a pattern match succeeds. The following example replaces comma(,) with vertical bar(|) only when a line contains the pattern The Pilgrimage. [jerry]$ sed ”/The Pilgrimage/ s/,/ | /g” books.txt On executing the above code, you get the following result: 1) A Storm of Swords, George R. R. Martin, 1216 2) The Two Towers, J. R. R. Tolkien, 352 3) The Alchemist, Paulo Coelho, 197 4) The Fellowship of the Ring, J. R. R. Tolkien, 432 5) The Pilgrimage | Paulo Coelho | 288 6) A Game of Thrones, George R. R. Martin, 864 In addition to this, SED can replace a specific occurrence of the pattern. Let us replace only the second instance of comma(,) with vertical bar(|). [jerry]$ sed ”s/,/ | /2” books.txt On executing the above code, you get the following result: 1) A Storm of Swords, George R. R. Martin | 1216 2) The Two Towers, J. R. R. Tolkien | 352 3) The Alchemist, Paulo Coelho | 197 4) The Fellowship of the Ring, J. R. R. Tolkien | 432 5) The Pilgrimage,Paulo Coelho | 288 6) A Game of Thrones, George R. R. Martin | 864 In the above example, the number at the end of the SED command (or at the place of flag) implies the 2nd occurrence. SED provides an interesting feature. After performing substitution, SED provides an option to show only the changed lines. For this purpose, SED uses the p flag which refers to print. The following example lists only changed lines. [jerry]$ sed -n ”s/Paulo Coelho/PAULO COELHO/p” books.txt On executing the above code, you get the following result: 3) The Alchemist, PAULO COELHO, 197 5) The Pilgrimage, PAULO COELHO, 288 We can store changed lines in another file as well. To achieve this result, use the w flag. The following example shows how to do it. [jerry]$ sed -n ”s/Paulo Coelho/PAULO COELHO/w junk.txt” books.txt We used the same SED command. Let us verify the contents of the junk.txt file. [jerry]$ cat junk.txt On executing the above code, you get the following result: 3) The Alchemist, PAULO COELHO, 197 5) The Pilgrimage, PAULO COELHO, 288 To perform case-insensitive substitution, use the i flag which implies ignore case. The following example performs case-insensitive substitution. [jerry]$ sed -n ”s/pAuLo CoElHo/PAULO COELHO/pi” books.txt On executing the above code, you get the following result: 3) The Alchemist, PAULO COELHO, 197 5) The Pilgrimage, PAULO COELHO, 288 So far, we have used only the foreslash(/) character as a delimiter, but we can also use vertical bar(|), at sign(@), caret(^), exclamation mark(!) as a delimiter. The following example shows how to use other characters as a delimiter. Let us assume you need to replace the path /bin/sed with /home/jerry/src/sed/sed-4.2.2/sed. Hence, your SED command looks like this: [jerry]$ echo “/bin/sed” | sed ”s//bin/sed//home/jerry/src/sed/sed-4.2.2/sed/” On executing the above code, you get the following result: /home/jerry/src/sed/sed-4.2.2/sed We can make this command more readable and easy to understand. Let us use vertical bar(|) as delimiter and see the result. [jerry]$ echo “/bin/sed” | sed ”s|/bin/sed|/home/jerry/src/sed/sed-4.2.2/sed|” On executing the above code, you get the following result: /home/jerry/src/sed/sed-4.2.2/sed Indeed! We got the same result and the syntax is more readable. Similarly, we can use the “at” sign (@) as a delimiter as follows: [jerry]$ echo “/bin/sed” | sed ”s@/bin/sed@/home/jerry/src/sed/sed-4.2.2/sed@” On executing the above code, you get the following result: /home/jerry/src/sed/sed-4.2.2/sed In addition to this, we can use caret(^) as a delimiter. [jerry]$ echo “/bin/sed” | sed ”s^/bin/sed^/home/jerry/src/sed/sed-4.2.2/sed^” On executing the above code, you get the following result: /home/jerry/src/sed/sed-4.2.2/sed We can also use exclamation mark (!) as a delimiter as follows: [jerry]$ echo “/bin/sed” | sed ”s!/bin/sed!/home/jerry/src/sed/sed-4.2.2/sed!” On executing the above code, you get the following result: /home/jerry/src/sed/sed-4.2.2/sed Generally, backslash(/) is used as a delimiter but sometimes it is more convenient to use other supported delimiters with SED. Creating a Substring We learnt the powerful substitute command.