Unix Editing Utilities Sed and Awk

For a good reference:
UNIX Made Easy, Second Edition, John Muster and Associates
sed & awk, Dougherty & Robbins, O'Reilly

Comparing sed to vi

When using the vi editor, the entire file is read into buffer space on disk.
You can make changes anywhere in the file.
An editor that works on the lines of a file one at a time is a stream editor, sed.

Sed

allows you to make a series of changes to a number of file from the command line
allows you to edit a very large file from the command line without reading it into a buffer area

Using sed to substitute a pattern

Exercise:

Create a sample file using the passwd file:

$ cp /etc/passwd /tmp/sample

Change the default shells in this file from tcsh to bash using sed's substitute
Notice you must create a new file

$ sed 's/tcsh/bash/' sample > sample.new

Compare the two files using the command diff

$ diff sample sample.new

Change occurrences of mail in your sample file to junkmail

$ sed 's/mail/junkmail/' sample > sample.new

Notice that only the first instance of the target word is matched on each line.
Search for a line with the string mail in it.

$ grep mail sample.new
junkmail:x:8:12:mail:/var/spool/mail:/sbin/nologin
junkmailnull:x:47:47::/var/spool/mqueue:/dev/null

You can instruct sed to change multiple instances on a single line by making a global request

$ sed 's/mail/junkmail/g' sample > sample.new

$ grep mail sample.new
junkmail:x:8:12:junkmail:/var/spool/junkmail:/sbin/nologin
junkmailnull:x:47:47::/var/spool/mqueue:/dev/null

Use grep to filter out the mailnull account entry (line 2 above) in the sample.new file

$ grep 'mail\>' sample.new

Return the file back to its original form, pipe output to grep

$ sed 's/junkmail/mail/g' sample.new | grep mail
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/dev/null

Request that sed act only on lines that meet a specified criteria

Here is a sample fstab file which contains mount requests for the file server fs1.
Change the direcory to be mounted from usr2 to usr3 on the NFS server fs1.
Copy /etc/fstab to the tmp directory and modify it in order to do this lab.

/dev/sda8    /        ext2     defaults        1 1
/dev/sda5    /boot    ext2     defaults        1 2
/dev/cdrom   /mnt/cdrom iso9660 noauto,owner,ro 0 0
/dev/sda6    /var     ext2     defaults        1 2
/dev/sda7    swap     swap     defaults        0 0
/proc        /proc    proc     defaults        0 0
fs1:/usr2/jack /home/jack nfs rw,auto,nouser
fs1:/usr2/jill /home/jill nfs rw,auto,nouser
fs1:/usr2/peter /home/peter nfs rw,auto,nouser
fs2:/usr2/redhat /usr2/redhat nfs rw,auto,nouser

Select lines containing the string "fs1" and substitute the string "usr2" with "usr3":

$ sed '/fs1/s/usr2/usr3/g' /tmp/fstab | grep fs[1,2]

fs1:/usr3/jack /home/jack nfs rw,auto,nouser
fs1:/usr3/jill /home/jill nfs rw,auto,nouser
fs1:/usr3/peter /home/peter nfs rw,auto,nouser
fs2:/usr2/redhat /usr2/redhat nfs rw,auto,nouser

fs2's /usr2 entry is unchanged
This would come in handy if you moved all the users off one partition onto another.
You would need to update your auto.home map

which is where user home directory data is stored if you are using NIS

or in the case of this example the /etc/fstab file on client machines

Exercise:

copy /etc/fstab /tmp and change to the /tmp directory

$ cp /etc/fstab /tmp; cd /tmp

change the root filesystem type from ext3 to ext2

$ sed '/LABEL=\//s/ext3/ext2/' fstab | less

notice how the / (representing the root filesystem) had to be escaped so as not to confuse the / with the substitude delimeter
Does this command change only the / file system?
Now add a space after "LABEL=\/", this will allow you to only modify the / file system

$ sed '/LABEL=\/ /s/ext3/ext2/' fstab | less

^^^

Deleting a line by number

Delete the second line of your sample file:

$ sed '2 d' /tmp/sample

Note: sed instructions are enclosed in single quotes to avoid being interpreted by the shell.
What other command uses single quotes for the same reason?

Delete a range of lines:

$ sed '2,10d' /tmp/sample

Delete any lines containing the string "news"

$ sed '/news/d' /tmp/sample | grep news

Why does this command line should return nothing?

Using the quit command to terminate sed

Exercise:

Create a shell script to replace the head command
The quit command (q) tells sed to stop reading input lines and sending them to output
First make a file called topp containing one line

# cat >> topp

sed '10q' $1

^d

Make the file executable

$ chmod +x topp

Run it on using any text file as as argument

$ ./topp /etc/passwd

The shell replaces the $1 in the script with the first argument from the command line (/etc/passwd).

Combining sed commands

Exercise:

Delete the comment lines and blank lines in the /etc/inittab file to view just the code
use the -e option preceding each instruction to inform sed that more than one set of instructions is being used

$ sed -e '/^#/d' -e '/^$/d' /etc/inittab

You can also do this with grep

$ grep '^[^#$]' /etc/inittab

The ^ indicates the beginning of the line
The [^ tells grep to not match the following characters inside the square brackets

The characters not matched are # and $

Combine the beginning-of-the-line symbol with both # and $
This grep expression matches a line that

doesn't start with a # (^#)
and isn't a blank line (^$)

Transforming Characters with sed

Exercise:

Replace all lowercase vowels to their uppercase forms

$ sed 'y/aeiou/AEIOU/' /etc/fstab

Using a sed command file

Exercises:

Create a sed command file containing these two lines and name it sed.cmd

$ cat >> sed.cmd

s/ext3/ext2/g

/cdrom/,$d

^d

Have sed read a command file for instructions and an input file on which to perform the commands
What does this command do?

$ sed -f sed.cmd fstab

Append Text using a command file
Create a file containing these three lines and name it sed.cmd.2

/^root/a \

newroot:x:0:0:New Root:/newroot:/bin/bash

/uucp/,$ d

a - append
d - delete, used this way it will delete uucp and everything after it to the end of the file
Enter this command:

$ sed -f sed.cmd.2 sample > sample.new

$ diff sample.new sample

What happens in the above example?

The line following /^root/a is appended below any line which starts with the string root.
All lines following and including the line containing uucp are deleted

Review

What do the following commands accomplish?

sed 's/fries/chips/g' file1
sed '/start/,$ d' file2
sed -e 's/initdefault/mychange/g' -e '/^$/ d' /etc/inittab
sed -f changes projectA
sed '/^$/q' /etc/inittab

Exercise:

Assume you have just updated your kernel and want the new kernel to be your default kernel in lilo.conf.

To be safe, first copy/etc/lilo.confto /tmp
Modify /tmp/lilo.conf so it contains two linux versions (see below)
Often times there is a copy of lilo.conf.anaconda in /etc which can be used

prompt
timeout=50
default=linux
boot=/dev/hda
map=/boot/map
install=/boot/boot.b
message=/boot/message
linear

image=/boot/vmlinuz-2.4.18
        label=linux-new
        initrd=/boot/initrd-2.4.18.img
        read-only
        root=/dev/hda1

image=/boot/vmlinuz-2.4.19
        label=linux
        initrd=/boot/initrd-2.4.19.img
        read-only
        root=/dev/hda1

Create the following shell script (you can cut and past it into a vi or pico session) and name it lilomod
The script will modify your /tmp/lilo.conf file and save the changes to /tmp/lilo.conf.new

#! /bin/bash

LOGFILE=/tmp/sedlog

if sed -e 's/label=linux$/label=linux-old/' \
       -e 's/label=linux-new/label=linux/' \
            /tmp/lilo.conf > /tmp/lilo.conf.new ; then
          :
else
      echo Sed command failed... |tee -a $LOGFILE 2>&1
      echo Check /tmp/lilo.conf for errors... |tee -a $LOGFILE 2>&1
fi

Make your script executable

$ chmod +x /tmp/lilomod

Execute it and look at the differences between the two files

$ diff lilo.conf lilo.conf.new

What does the tee command and the redirector 2>&1 do in the above script?

Exercise:

1. Create the directory /tmp/sysconfig

$ mkdir /tmp/sysconfig

2. Copy /etc/sysconfig/network file to /tmp/sysconfig

$ cp /etc/sysconfig/network /tmp/sysconfig

3. Create a shell script named /tmp/namehost that looks like the following:

#! /bin/bash

CUSTOM_NAME=stationXX

if [ $CUSTOM_NAME != "" ]; then
   cp /tmp/sysconfig/network /tmp/sysconfig/network.svd
   sed '/HOSTNAME/d' /tmp/sysconfig/network \
        > /tmp/sysconfig/network.new
   mv /tmp/sysconfig/network.new /tmp/sysconfig/network
   echo "HOSTNAME=$CUSTOM_NAME" >> /tmp/sysconfig/network
fi

4. Make the script executable

$ chmod +x /tmp/namehost

5. Execute the script

$ /tmp/namehost

6. Explain what happened

Extracting a value from a line of text using sed

Extract a value on the end of a line

1. Look at the contents of /proc/cpuinfo

$ less /proc/cpuinfo

2. Extract the model name and assign it the variable name model

The cat and grep are easy
the sed command is substituting nothing for the regular expression that

starts with model name
followed by zero or other characters
followed by a semi colon
followed by zero of more spaces

that leaves the value which is put into the variable model

$ model=`cat /proc/cpuinfo | grep 'model name' \
| sed 's/^model name.*: *//'`

3. check the value of the variable model

$ echo $model
Pentium II (Klamath)

Extract a value surrounded by other text

1. look at the output of ifconfig

$ ifconfig eth0 | grep 'inet addr'

2. Extract the IP number of your machine from this output

Here is the line in question:

inet addr:192.168.1.20 Bcast:192.168.1.255 Mask:255.255.255.0

$ ipaddress=`ifconfig eth0 | grep 'inet addr' \

| sed -e 's/.*inet addr://' -e 's/ .*$//'`

Starting with sed:
the first substitution removes the beginning of the line up to the :
The second substitution removes the end of the line starting with the space after the actual dotted decimal IP number.

3. check the value of ipaddress

$ echo $ipaddress
192.168.1.20

Saving user input to a file using sed

many lines

$ sed -e 's/^//' > /tmp/input
stuff
more stuff
my input lines here
^d
$ cat /tmp/input
stuff
more stuff
my input lines here

The more familiar way

$ cat > /tmp/input
stuff
more stuff
my input lines here
^d

one line

$ sed -e 's/^//' -e 'q' > /tmp/input
one line of input
$ cat /tmp/input
one line of input

The more familiar way

$ echo "one line of input" > /tmp/input

Awk

The name 'awk' is derived from the names of the programmers who worte it,:

Aho, Weinberger, and Kernighan.

Awk is useful for information retrieval
- data manipulation
- formatting output for report writing
By default awk uses white space to separate fields in a record

Awk does its work by selecting records based on a pattern or field, and then performing some action on the selected record.

Instruct awk to select lines that have the string home in any field:

$ awk '/home/' /etc/fstab

Have awk print only the third column of matching records:

$ awk '/home/ {print $3}' /etc/fstab

Print every line in the file.

$ awk '{print $0}' /etc/passwd

Print the 3rd column of every line in the file

$ awk '{print $3}' /etc/passwd

Change the field delimiter

Change the delimeter from the default "white space" to a colon (:).
Notice that you must separate each argument you print with a comma

or the fields will all run together.

Try it with and without the commas

$ awk -F: '{print $1, $3, $4}' /etc/passwd | less

In the above example, the output consists of just the first, third, and fourth fields of all records in the password file.

Relational and Boolean Operators in Awk

== equal to

!= not equal to

< less than

<= less than or equal to

> greater than

>= greater than or equal to

&& and

|| or

~ a match for (regular expression)

!~ not a match for (regular expression)

Selecting Multiple Patterns with awk

The | | tells awk to print lines that contain either uucp or daemon.

$ awk '/uucp/ || /daemon/ {print}' /etc/passwd

Selecting lines by field value

Here we are selecting files based on file size which is field 5 in 'ls -l'.
Only files less then 1K in size will be displayed.

$ ls -l /etc | awk '$5 < 1024'

Match if two conditions are met using &&

This rule matches file less then 1024 bytes in size but greater than 500 bytes.

$ ls -l /etc | awk '$5 < 1024 && $5 > 500'

Here we match entries in the passwd file which have the string home in field 6 and also have the string bash in field 7.
Notice we use -F to change the default delimeter

$ awk -F: '$6 ~/home/ && $7 ~/bash/' /etc/passwd

The ~ operator tells awk to search the 6th field of the records and select the record if the string home is anywhere in that field.
If you want an exact match you would use ==
Notice that double quotes must be used around the string

$ awk -F: '$6 == "/home/mcc" && $7 ~/bash/' /etc/passwd

mcc:x:5560:5560::/home/mcc:/bin/bash

Using logical negation

What happens to the proc line in /etc/fstab when you use negation?

$ awk '!($2 == "/proc")' /etc/fstab

Here is a way to sum the file sizes in a directory.

$ ls -l | awk '{x+=$5} END {print x}'
2036654

Using the same technique, how would you sum the amount of memory being used by all the processes running on your system?

$ ps aux | less
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 1372 432 ? S Nov12 0:04 init [3]

Notice the RSS field is column 6
RSS is the resident set size of your application's text and data segments (the number of pages of memory)
A page is 4K or 4096 bytes
Note that RSS does not account for

shared libraries and data segments
files that are open and cached in memory
kernel segments

This command gives you an idea of the amount of memory used by all the processes running on your system, but it is not conclusive

$ ps aux | awk '{x+=$6} END {print x}'
300448

Using Special Variables in Awk

NR	Number of current record
$NR	Meaningless
NF	Number of field in current record
$NF	prints last field in a line

Exercise:

What does the following commands return?

$ ls -l | awk '{print $NF}'
$ ls -l | awk '{print NF}'
$ ls -l | awk '{print NR}'

Extracting records (lines) from a file

Lets say you want to save the last 200 lines in a very large log file to another file.
You could use vi, but the log file is 40MB in size and you don't enough system resources to open it.
The awk command can be used to cut records from a file using the NR variable.

NR: Number of the current record

First, locate the biggest log file in /var/log

$ ls -l | sort -nr +4 | head

Copy that file to /tmp/test
You will probably have to be the root user to read this file

$ cp /var/log/messages /tmp/test

Find out how many lines are in the file.

$ wc -l /tmp/test
231635 /tmp/test

Simple math tells us where to begin grabbing lines
Remember, we want to save the last 200 lines

231635 - 200 = 231435

Use awk to grab the last 200 records and redirect to /tmp/test2

$ awk 'NR>231435' /tmp/test > /tmp/test2

$ wc -l /tmp/test2
200 test2

Homework:

1. Write a command line using sed and awk that prints the name of any partition (not device) on your system who's capacity is at 95% or greater. Make sure your command line works. Test it!
One possible solution:
$ df | sed 's/%//'| awk '$5>95 && $5 ~/[0-9]/ {print $6,$5"%"}'
Be prepared to write a similar command next Wednesday in class for a quiz.

==	equal to
!=	not equal to
<	less than
<=	less than or equal to
>	greater than
>=	greater than or equal to
&&	and
\|\|	or
~	a match for (regular expression)
!~	not a match for (regular expression)