Locating Data using the grep Command

Commands covered in this section:

grep
egrep
ps

grep: global regular expression printing

grep gives you the ability to search the contents of files or a directory of files without reading the file into a buffer or memory.

It searches one line at at time for the regular expression specified

Examples:

Seaching for text in the output of another command

Lets say you need to update a configuration file in the /etc/directory but you forget the name of the file. All you remember is that it contains the string "conf".

This command will search the output of ls for files that contain the string "conf".

$ ls /etc | grep confhost.conf adduser.conf brltty.conf compizconfig debconf.conf deluser.conf discover.conf discover.conf-2.6 fdmount.conf fuse.conf gai.conf ...

As you can see there are many files in the /etc directory that have the string "conf" in their names.

Many Linux/UNIX versions will fail to find anything if you use this command. We'll discuss why below when we talk about metacharacters and their meaning to the shell versus their meaning in the grep command.

Searching for text in a file

You can also use grep to search through files for information.
Here we will use grep to find all occurances of the string "swap" in the /etc/fstab file.

$ grep swap /etc/fstab

/dev/sdb5 swap swap defaults 0 0

/dev/sda5 swap swap defaults 0 0

The syntax for the grep command:

grep [options] regular expression [file1 file2 ... fileN]

What is a Regular Expression?

A regular expression is a pattern of text
grep searches for regular expressions
A regex can contain metacharacters which have special meaning to the grep command

The shell interprets metacharaters differently then grep.

Examples of a simple regualar expression

^p is used to find files in the current directory witch begin with the letter 'p'

and count the number of lines output

The metacharacter '^' means the beginning of a line.

$ ls /etc | grep '^p' | wc -l

Search for files that contain the string 'dev' in all the files in the /etc directory.

$ grep dev /etc/*

Options to the grep command

The '-l' tells grep to print out the name of the file the regex was found in but not the match.

$ grep -l dev /etc/*

NOTE: You will also get errors if you don't have permission to read the files or if the file is a directory.

Screen out errors by redirecting stderr to /dev/null

$ grep -l dev /etc/* 2> /dev/null

Where does the stdout go in the above command?
Do you know the command to redirect both stderr and stdout to the same file?

$ grep -l dev /etc/* > /tmp/myout 2>&1

This command will print the file name and the number of occurances of dev even if there are zero matches.
This is useful if there are many occurances in the files

$ grep -c dev /etc/*

...
/etc/cron.daily:0
/etc/cron.deny:0
/etc/cron.hourly:0
/etc/cron.monthly:0
/etc/cron.weekly:0
...

If you don't want to see the zero matches use grep with the -v option
The -v tells grep to select non-matchinglines.
print everything but ':0'.

$ grep -c dev /etc/* | grep -v ':0'

Display all matches and their line number using the output of the ps command, which lists process status.

$ ps aux | grep -n pattyo
2:pattyo   19876 0.0 0.4 2792 748 pts/6 S Nov14 0:02 bash
3:pattyo   32228 0.0 0.5 2720 812 pts/6 R 22:20 0:00 ps aux
4:pattyo   32229 0.0 0.3 1728 584 pts/6 S 22:20 0:00 grep -n pattyo
                                                        ^^^^^^^^^^^^^^^^
Note: In the output above, we also see the grep command itself in this process listing.

Another example of grep printing line numbers.
$ grep -n pattyo /etc/passwd
32:pattyo:x:500:500:Patty O'Reilly:/home/pattyo:/bin/bash

Metacharacters

In one of the examples above, we searched for filenames beginning with the letter p using the regex (regular expression) 'grep ^p'. The '^' is a metacharacter.

Metacharacter may have a different meaning to grep then they do to your shell
This confusion will cause bad results!
Surround grep expressions containing metacharacters with quotes

Metacharacter	Meaning in grep	Meaning in the shell
.	match any character; `grep -c '.' somefile`; counts the number of non-blank lines in the file	if followed by filename, exectute filename
*	match zero or more preceding characters; *`grep -n '.' somefile;`** number all lines (includeing blanks) in file	match zero or more
^	match beginning of line; `ls /etc \| grep '^p';` also used for negation; `grep 'q[^u]' filel;` matches q not followed by u	bourne shell pipe symbol
$	match end of line; `ps -aux \| grep 'd]$'`	shell variable, also the user prompt
\	escape the character following; *`grep '\' /etc/crontab;`** matches all the lines containing a *	escape the character following
[ ]	match one from this set or range; `grep '[Yy]ou' somefile` `ls /etc/rc.d \| grep 'rc[0-6]'`	match from this set or range
{ }	matches a specific number of characters or between min and max number; `ls \| egrep '^K[0-9]{3}';` would match files starting with K followed by 3 numbers
+	match one or more preceding characters; `ls \| egrep 'K[0-9]+';`	match one or more preceding
?	match zero or one preceding characters; `egrep 'colou?r' somefile;`	match one character

So in the previous example it would have been safer to do the following:

$ ls /etc | grep '^p' | wc -l

Find filenames with patterns at the end of their names.

$ ls /etc | grep 'p$'

Find lines with patterns at the end of lines

$ grep 'bash$' /etc/passwd

What would the following command print to the screen?

$ grep -v 'bash$' /etc/passwd

The shell and metacharacters

The shell interprets the $USER as the value of the shell variable.

$ who | grep $USER
pattyo   tty1     Nov 3 01:02
pattyo   pts/0    Nov 3 01:03 (:0)
pattyo   pts/1    Nov 3 01:03 (:0)
pattyo   pts/2    Nov 3 01:03 (:0)

Why does this command not return the processes that the user jars owns?

$ ps aux | grep '^$USER'

How can we fix it?
- Shell Quoting

Instructing grep to treat a metacharacter as ordinary.

Create a file containing this line exactly: $USER$

cat >> junk

$USER$

^d

Experiment with the following commands:

grep '$USER' junk

$USER$

grep '$USER$' junk
returns no match, because the $ is interpreted as the end of line character

grep '^$USER$' junk
returns no match

grep '^$USER' junk
$USER$

grep '^\$USER\$$' junk
$USER$

The last example is the most reliable way to use the grep command.
Use the '\' character to escape the character following it
This instucts grep to interpret the dollar sign ($) literally and not as a special character

Selecting only blank lines from a file:

$ grep '^$' /etc/hosts

Printing out a file without the comment lines:

$ grep -v '^#' /etc/hosts

Searching for the word 'the' or the word 'The'

The [ ] instructs grep to match any one of the enclosed characters.

$ grep '[Tt]he' /etc/*

Locating a range of characters

These two examples produce the same results

$ grep 'hda[0123456789]' /etc/fstab

$ grep 'hda[0-9]' /etc/fstab

The dash (-) is a metacharacter that tells grep to create a range. You don't need to type in each number.

$ ls /etc/rc.d | grep 'rc[0-6].d'
rc0.d
rc1.d
rc2.d
rc3.d
rc4.d
rc5.d
rc6.d

You can use letter ranges as well.

$ egrep 'sd[a-z][0-9]+' /etc/fstab
/dev/sdb5    swap      swap    defaults   0 0
/dev/sda5    swap      swap    defaults   0 0
/dev/sdb1   /home      ext3    defaul     1 2

NOTE: If we used a star (*) instead of a plus (+) we will match even if there are no numbers after after the [a-z] character. The plus (+) matches only if at least one number exists. The star (*) matches zero or more numbers.

Using brackets to search for literal metacharacter.

$ grep '[\2^]' filename

Lines containing the characters \, 2, or ^ are returned.
All the characters enclosed in the brackets are treated as ordinary characters.
The only time the caret is not treated as ordinary is inside brackets is when it appears first!

$ grep '[^\2]' filename

Then it is treated as the negation metacharacter.

Searching for a specific word.

$ grep '\<[Tt]he\>' file

In this example, grep will match both capital or lower case T.
Grep will match exact, literal occurrences of the string enclosed in the \< \>. (NOTE: The brackets had to be escaped.)
The string will not be part of another larger word.
The advantage is that you can match a single word followed by punctuation or an end or beginning of line.
Below, lines that contain the string the at the beginning of a word are selected.

$ grep '\<the' file

Matching any Character.

The dot (.) metacharacter will match any single character.
The following command will match every line containing any character.
If the line is blank (no characters) it is not matched.

$ grep '.' file

In this example lines that do NOT have a character in them are selected.

$ grep -v '.' file

Match lines that contain a three-letter word ending in he.
The . matches any single character

$ grep '\<.he\>' file

This command locates all lines that contain an s followed by zero or more characters, followed in turn by an s.

$ grep 's.*s' file

egrep

This utilizes the egrep command to combine the above two searches

egrep (extended grep)
allows OR searches by letting you put a pipe symbol between search expressions
attempts to match the regular expression on each line of the file
displays only the lines that match

Exmples:

Print out the /etc/syslog.conf config file without comment lines or blank lines.

$ egrep -v '^#|^$' /etc/syslog.conf
*.err;kern.debug;daemon.info;user.none;local3.none /var/log/syslog *.alert;kern.err;daemon.err root *.emerg * mail.debug /var/log/syslogs/sendmail auth.info /var/log/syslogs/authlog authpriv.debug /var/log/secure *.info @nloghost local0.info /var/log/syslogs/poplog local1.info /var/log/syslogs/local1 local2.info /var/log/syslogs/sudo.log local3.info /var/log/syslogs/local3 local4.info /var/log/syslogs/local4 local5.info /var/log/syslogs/local5 local6.debug /var/log/syslogs/local6 local7.debug /var/log/syslogs/local7

Printing only the From and Subject lines in a users mail box.

$ cd /var/mail
$ egrep '^(From|Subject):' jars

Looking for system daemons

$ ps aux | egrep 'd$|d]$'