Category: Computer Programming

AWK – Output Redirection ”; Previous Next So far, we displayed data on standard output stream. We can also redirect data to a file. A redirection appears after the print or printf statement. Redirections in AWK are written just like redirection in shell commands, except that they are written inside the AWK program. This chapter explains redirection with suitable examples. Redirection Operator The syntax of the redirection operator is − Syntax print DATA > output-file It writes the data into the output-file. If the output-file does not exist, then it creates one. When this type of redirection is used, the output-file is erased before the first output is written to it. Subsequent write operations to the same output-file do not erase the output-file, but append to it. For instance, the following example writes Hello, World !!! to the file. Let us create a file with some text data. Example [jerry]$ echo “Old data” > /tmp/message.txt [jerry]$ cat /tmp/message.txt On executing this code, you get the following result − Output Old data Now let us redirect some contents into it using AWK”s redirection operator. Example [jerry]$ awk ”BEGIN { print “Hello, World !!!” > “/tmp/message.txt” }” [jerry]$ cat /tmp/message.txt On executing this code, you get the following result − Output Hello, World !!! Append Operator The syntax of append operator is as follows − Syntax print DATA >> output-file It appends the data into the output-file. If the output-file does not exist, then it creates one. When this type of redirection is used, new contents are appended at the end of file. For instance, the following example appends Hello, World !!! to the file. Let us create a file with some text data. Example [jerry]$ echo “Old data” > /tmp/message.txt [jerry]$ cat /tmp/message.txt On executing this code, you get the following result − Output Old data Now let us append some contents to it using AWK”s append operator. Example [jerry]$ awk ”BEGIN { print “Hello, World !!!” >> “/tmp/message.txt” }” [jerry]$ cat /tmp/message.txt On executing this code, you get the following result − Output Old data Hello, World !!! Pipe It is possible to send output to another program through a pipe instead of using a file. This redirection opens a pipe to command, and writes the values of items through this pipe to another process to execute the command. The redirection argument command is actually an AWK expression. Here is the syntax of pipe − Syntax print items | command Let us use tr command to convert lowercase letters to uppercase. Example [jerry]$ awk ”BEGIN { print “hello, world !!!” | “tr [a-z] [A-Z]” }” On executing this code, you get the following result − Output HELLO, WORLD !!! Two way communication AWK can communicate to an external process using |&, which is two-way communication. For instance, the following example uses tr command to convert lowercase letters to uppercase. Our command.awk file contains − Example BEGIN { cmd = “tr [a-z] [A-Z]” print “hello, world !!!” |& cmd close(cmd, “to”) cmd |& getline out print out; close(cmd); } On executing this code, you get the following result − Output HELLO, WORLD !!! Does the script look cryptic? Let us demystify it. The first statement, cmd = “tr [a-z] [A-Z]”, is the command to which we establish the two-way communication from AWK. The next statement, i.e., the print command provides input to the tr command. Here &| indicates two-way communication. The third statement, i.e., close(cmd, “to”), closes the to process after competing its execution. The next statement cmd |& getline out stores the output into out variable with the aid of getline function. The next print statement prints the output and finally the close function closes the command. Print Page Previous Next Advertisements ”;

AWK – Built-in Functions ”; Previous Next AWK has a number of functions built into it that are always available to the programmer. This chapter describes Arithmetic, String, Time, Bit manipulation, and other miscellaneous functions with suitable examples. S.No. Built in functions & Description 1 Arithmetic Functions AWK has the following built-in arithmetic functions. 2 String Functions AWK has the following built-in String functions. 3 Time Functions AWK has the following built-in time functions. 4 Bit Manipulation Functions AWK has the following built-in bit manipulation functions. 5 Miscellaneous Functions AWK has the following miscellaneous functions. Print Page Previous Next Advertisements ”;

AWK – Quick Guide ”; Previous Next AWK – Overview AWK is an interpreted programming language. It is very powerful and specially designed for text processing. Its name is derived from the family names of its authors − Alfred Aho, Peter Weinberger, and Brian Kernighan. The version of AWK that GNU/Linux distributes is written and maintained by the Free Software Foundation (FSF); it is often referred to as GNU AWK. Types of AWK Following are the variants of AWK − AWK − Original AWK from AT & T Laboratory. NAWK − Newer and improved version of AWK from AT & T Laboratory. GAWK − It is GNU AWK. All GNU/Linux distributions ship GAWK. It is fully compatible with AWK and NAWK. Typical Uses of AWK Myriad of tasks can be done with AWK. Listed below are just a few of them − Text processing, Producing formatted text reports, Performing arithmetic operations, Performing string operations, and many more. AWK – Environment This chapter describes how to set up the AWK environment on your GNU/Linux system. Installation Using Package Manager Generally, AWK is available by default on most GNU/Linux distributions. You can use which command to check whether it is present on your system or not. In case you don’t have AWK, then install it on Debian based GNU/Linux using Advance Package Tool (APT) package manager as follows − [jeryy]$ sudo apt-get update [jeryy]$ sudo apt-get install gawk Similarly, to install AWK on RPM based GNU/Linux, use Yellowdog Updator Modifier yum package manager as follows − [root]# yum install gawk After installation, ensure that AWK is accessible via command line. [jerry]$ which awk On executing the above code, you get the following result − /usr/bin/awk Installation from Source Code As GNU AWK is a part of the GNU project, its source code is available for free download. We have already seen how to install AWK using package manager. Let us now understand how to install AWK from its source code. The following installation is applicable to any GNU/Linux software, and for most other freely-available programs as well. Here are the installation steps − Step 1 − Download the source code from an authentic place. The command-line utility wget serves this purpose. [jerry]$ wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.1.tar.xz Step 2 − Decompress and extract the downloaded source code. [jerry]$ tar xvf gawk-4.1.1.tar.xz Step 3 − Change into the directory and run configure. [jerry]$ ./configure Step 4 − Upon successful completion, the configure generates Makefile. To compile the source code, issue a make command. [jerry]$ make Step 5 − You can run the test suite to ensure the build is clean. This is an optional step. [jerry]$ make check Step 6 − Finally, install AWK. Make sure you have super-user privileges. [jerry]$ sudo make install That is it! You have successfully compiled and installed AWK. Verify it by executing the awk command as follows − [jerry]$ which awk On executing this code, you get the following result − /usr/bin/awk AWK – Workflow To become an expert AWK programmer, you need to know its internals. AWK follows a simple workflow − Read, Execute, and Repeat. The following diagram depicts the workflow of AWK − Read AWK reads a line from the input stream (file, pipe, or stdin) and stores it in memory. Execute All AWK commands are applied sequentially on the input. By default AWK execute commands on every line. We can restrict this by providing patterns. Repeat This process repeats until the file reaches its end. Program Structure Let us now understand the program structure of AWK. BEGIN block The syntax of the BEGIN block is as follows − Syntax BEGIN {awk-commands} The BEGIN block gets executed at program start-up. It executes only once. This is good place to initialize variables. BEGIN is an AWK keyword and hence it must be in upper-case. Please note that this block is optional. Body Block The syntax of the body block is as follows − Syntax /pattern/ {awk-commands} The body block applies AWK commands on every input line. By default, AWK executes commands on every line. We can restrict this by providing patterns. Note that there are no keywords for the Body block. END Block The syntax of the END block is as follows − Syntax END {awk-commands} The END block executes at the end of the program. END is an AWK keyword and hence it must be in upper-case. Please note that this block is optional. Let us create a file marks.txt which contains the serial number, name of the student, subject name, and number of marks obtained. 1) Amit Physics 80 2) Rahul Maths 90 3) Shyam Biology 87 4) Kedar English 85 5) Hari History 89 Let us now display the file contents with header by using AWK script. Example [jerry]$ awk ”BEGIN{printf “Sr NotNametSubtMarksn”} {print}” marks.txt When this code is executed, it produces the following result − Output Sr No Name Sub Marks 1) Amit Physics 80 2) Rahul Maths 90 3) Shyam Biology 87 4) Kedar English 85 5) Hari History 89 At the start, AWK prints the header from the BEGIN block. Then in the body block, it reads a line from a file and executes AWK”s print command which just prints the contents on the standard output stream. This process repeats until file reaches the end. AWK – Basic Syntax AWK is simple to use. We can provide AWK commands either directly from the command line or in the form of a text file containing AWK commands. AWK Command Line We can specify an AWK command within single quotes at command line as shown − awk [options] file … Example Consider a text file marks.txt with the following content − 1) Amit Physics 80 2) Rahul Maths 90 3) Shyam Biology 87 4) Kedar English 85 5) Hari History 89 Let us display the complete content of the file using AWK as follows − Example [jerry]$ awk ”{print}” marks.txt On executing this code, you get the following result −

AWK – Useful Resources ”; Previous Next The following resources contain additional information on AWK. Please use them to get more in-depth knowledge on this. Useful Links on AWK AWK Wiki − Wikipedia Reference for AWK. Useful Books on AWK To enlist your site on this page, please drop an email to [email protected] Print Page Previous Next Advertisements ”;

AWK – Built-in Variables ”; Previous Next AWK provides several built-in variables. They play an important role while writing AWK scripts. This chapter demonstrates the usage of built-in variables. Standard AWK variables The standard AWK variables are discussed below. ARGC It implies the number of arguments provided at the command line. Example [jerry]$ awk ”BEGIN {print “Arguments =”, ARGC}” One Two Three Four On executing this code, you get the following result − Output Arguments = 5 But why AWK shows 5 when you passed only 4 arguments? Just check the following example to clear your doubt. ARGV It is an array that stores the command-line arguments. The array”s valid index ranges from 0 to ARGC-1. Example [jerry]$ awk ”BEGIN { for (i = 0; i < ARGC – 1; ++i) { printf “ARGV[%d] = %sn”, i, ARGV[i] } }” one two three four On executing this code, you get the following result − Output ARGV[0] = awk ARGV[1] = one ARGV[2] = two ARGV[3] = three CONVFMT It represents the conversion format for numbers. Its default value is %.6g. Example [jerry]$ awk ”BEGIN { print “Conversion Format =”, CONVFMT }” On executing this code, you get the following result − Output Conversion Format = %.6g ENVIRON It is an associative array of environment variables. Example [jerry]$ awk ”BEGIN { print ENVIRON[“USER”] }” On executing this code, you get the following result − Output jerry To find names of other environment variables, use env command. FILENAME It represents the current file name. Example [jerry]$ awk ”END {print FILENAME}” marks.txt On executing this code, you get the following result − Output marks.txt Please note that FILENAME is undefined in the BEGIN block. FS It represents the (input) field separator and its default value is space. You can also change this by using -F command line option. Example [jerry]$ awk ”BEGIN {print “FS = ” FS}” | cat -vte On executing this code, you get the following result − Output FS = $ NF It represents the number of fields in the current record. For instance, the following example prints only those lines that contain more than two fields. Example [jerry]$ echo -e “One TwonOne Two ThreenOne Two Three Four” | awk ”NF > 2” On executing this code, you get the following result − Output One Two Three One Two Three Four NR It represents the number of the current record. For instance, the following example prints the record if the current record number is less than three. Example [jerry]$ echo -e “One TwonOne Two ThreenOne Two Three Four” | awk ”NR < 3” On executing this code, you get the following result − Output One Two One Two Three FNR It is similar to NR, but relative to the current file. It is useful when AWK is operating on multiple files. Value of FNR resets with new file. OFMT It represents the output format number and its default value is %.6g. Example [jerry]$ awk ”BEGIN {print “OFMT = ” OFMT}” On executing this code, you get the following result − Output OFMT = %.6g OFS It represents the output field separator and its default value is space. Example [jerry]$ awk ”BEGIN {print “OFS = ” OFS}” | cat -vte On executing this code, you get the following result − Output OFS = $ ORS It represents the output record separator and its default value is newline. Example [jerry]$ awk ”BEGIN {print “ORS = ” ORS}” | cat -vte On executing the above code, you get the following result − Output ORS = $ $ RLENGTH It represents the length of the string matched by match function. AWK”s match function searches for a given string in the input-string. Example [jerry]$ awk ”BEGIN { if (match(“One Two Three”, “re”)) { print RLENGTH } }” On executing this code, you get the following result − Output 2 RS It represents (input) record separator and its default value is newline. Example [jerry]$ awk ”BEGIN {print “RS = ” RS}” | cat -vte On executing this code, you get the following result − Output RS = $ $ RSTART It represents the first position in the string matched by match function. Example [jerry]$ awk ”BEGIN { if (match(“One Two Three”, “Thre”)) { print RSTART } }” On executing this code, you get the following result − Output 9 SUBSEP It represents the separator character for array subscripts and its default value is 34. Example [jerry]$ awk ”BEGIN { print “SUBSEP = ” SUBSEP }” | cat -vte On executing this code, you get the following result − Output SUBSEP = ^$ $0 It represents the entire input record. Example [jerry]$ awk ”{print $0}” marks.txt On executing this code, you get the following result − Output 1) Amit Physics 80 2) Rahul Maths 90 3) Shyam Biology 87 4) Kedar English 85 5) Hari History 89 $n It represents the nth field in the current record where the fields are separated by FS. Example [jerry]$ awk ”{print $3 “t” $4}” marks.txt On executing this code, you get the following result − Output Physics 80 Maths 90 Biology 87 English 85 History 89 GNU AWK Specific Variables GNU AWK specific variables are as follows − ARGIND It represents the index in ARGV of the current file being processed. Example [jerry]$ awk ”{ print “ARGIND = “, ARGIND; print “Filename = “, ARGV[ARGIND] }” junk1 junk2 junk3 On executing this code, you get the following result − Output ARGIND = 1 Filename = junk1 ARGIND = 2 Filename = junk2 ARGIND = 3 Filename = junk3 BINMODE It is used to specify binary mode for all file I/O on non-POSIX systems. Numeric values of 1, 2, or 3 specify that input files, output files, or all files, respectively, should use binary I/O. String values of r or w specify that input files or output files, respectively, should use binary I/O. String values of rw or wr specify that all files should use binary I/O.

Discuss AWK ”; Previous Next This tutorial takes you through AWK, one of the most prominent text-processing utility on GNU/Linux. It is very powerful and uses simple programming language. It can solve complex text processing tasks with a few lines of code. Starting with an overview of AWK, its environment, and workflow, the tutorial proceeds to explain the syntax, variables, operators, arrays, loops, and functions used in AWK. It also covers topics such as output redirection and pretty printing. Print Page Previous Next Advertisements ”;

AWK – Pretty Printing ”; Previous Next So far we have used AWK”s print and printf functions to display data on standard output. But printf is much more powerful than what we have seen before. This function is borrowed from the C language and is very helpful while producing formatted output. Below is the syntax of the printf statement − Syntax printf fmt, expr-list In the above syntax fmt is a string of format specifications and constants. expr-list is a list of arguments corresponding to format specifiers. Escape Sequences Similar to any string, format can contain embedded escape sequences. Discussed below are the escape sequences supported by AWK − New Line The following example prints Hello and World in separate lines using newline character − Example [jerry]$ awk ”BEGIN { printf “HellonWorldn” }” On executing this code, you get the following result − Output Hello World Horizontal Tab The following example uses horizontal tab to display different field − Example [jerry]$ awk ”BEGIN { printf “Sr NotNametSubtMarksn” }” On executing the above code, you get the following result − Output Sr No Name Sub Marks Vertical Tab The following example uses vertical tab after each filed − Example [jerry]$ awk ”BEGIN { printf “Sr NovNamevSubvMarksn” }” On executing this code, you get the following result − Output Sr No Name Sub Marks Backspace The following example prints a backspace after every field except the last one. It erases the last number from the first three fields. For instance, Field 1 is displayed as Field, because the last character is erased with backspace. However, the last field Field 4 is displayed as it is, as we did not have a b after Field 4. Example [jerry]$ awk ”BEGIN { printf “Field 1bField 2bField 3bField 4n” }” On executing this code, you get the following result − Output Field Field Field Field 4 Carriage Return In the following example, after printing every field, we do a Carriage Return and print the next value on top of the current printed value. It means, in the final output, you can see only Field 4, as it was the last thing to be printed on top of all the previous fields. Example [jerry]$ awk ”BEGIN { printf “Field 1rField 2rField 3rField 4n” }” On executing this code, you get the following result − Output Field 4 Form Feed The following example uses form feed after printing each field. Example [jerry]$ awk ”BEGIN { printf “Sr NofNamefSubfMarksn” }” On executing this code, you get the following result − Output Sr No Name Sub Marks Format Specifier As in C-language, AWK also has format specifiers. The AWK version of the printf statement accepts the following conversion specification formats − %c It prints a single character. If the argument used for %c is numeric, it is treated as a character and printed. Otherwise, the argument is assumed to be a string, and the only first character of that string is printed. Example [jerry]$ awk ”BEGIN { printf “ASCII value 65 = character %cn”, 65 }” Output On executing this code, you get the following result − ASCII value 65 = character A %d and %i It prints only the integer part of a decimal number. Example [jerry]$ awk ”BEGIN { printf “Percentags = %dn”, 80.66 }” On executing this code, you get the following result − Output Percentags = 80 %e and %E It prints a floating point number of the form [-]d.dddddde[+-]dd. Example [jerry]$ awk ”BEGIN { printf “Percentags = %En”, 80.66 }” On executing this code, you get the following result − Output Percentags = 8.066000e+01 The %E format uses E instead of e. Example [jerry]$ awk ”BEGIN { printf “Percentags = %en”, 80.66 }” On executing this code, you get the following result − Output Percentags = 8.066000E+01 %f It prints a floating point number of the form [-]ddd.dddddd. Example [jerry]$ awk ”BEGIN { printf “Percentags = %fn”, 80.66 }” On executing this code, you get the following result − Output Percentags = 80.660000 %g and %G Uses %e or %f conversion, whichever is shorter, with non-significant zeros suppressed. Example [jerry]$ awk ”BEGIN { printf “Percentags = %gn”, 80.66 }” Output On executing this code, you get the following result − Percentags = 80.66 The %G format uses %E instead of %e. Example [jerry]$ awk ”BEGIN { printf “Percentags = %Gn”, 80.66 }” On executing this code, you get the following result − Output Percentags = 80.66 %o It prints an unsigned octal number. Example [jerry]$ awk ”BEGIN { printf “Octal representation of decimal number 10 = %on”, 10}” On executing this code, you get the following result − Output Octal representation of decimal number 10 = 12 %u It prints an unsigned decimal number. Example [jerry]$ awk ”BEGIN { printf “Unsigned 10 = %un”, 10 }” On executing this code, you get the following result − Output Unsigned 10 = 10 %s It prints a character string. Example [jerry]$ awk ”BEGIN { printf “Name = %sn”, “Sherlock Holmes” }” On executing this code, you get the following result − Output Name = Sherlock Holmes %x and %X It prints an unsigned hexadecimal number. The %X format uses uppercase letters instead of lowercase. Example [jerry]$ awk ”BEGIN { printf “Hexadecimal representation of decimal number 15 = %xn”, 15 }” On executing this code, you get the following result − Output Hexadecimal representation of decimal number 15 = f Now let use %X and observe the result − Example [jerry]$ awk ”BEGIN { printf “Hexadecimal representation of decimal number 15 = %Xn”, 15 }” On executing this code, you get the following result − Output Hexadecimal representation of decimal number 15 = F %% It prints a single % character and no argument is converted. Example [jerry]$ awk ”BEGIN { printf “Percentags = %d%%n”, 80.66 }” On executing this code, you get the following result − Output Percentags = 80% Optional Parameters with % With % we can use following optional parameters