The UNIX/Linux Shell

The shell's job is to translate the user's command lines into operating system instructions
It is an interface to the Operating System
Similar to command.com in DOS
Knowing how the shell works will make you a far more effective system's administrator!

What we'll cover:

Available shells on your system
Changing shells
Command-line parsing
Metacharacters and Wildcards
Redirection: stdin, stout, stderr
Pipeing output of one command into another to create complex commands
Shell history

Commands (nothing new here)

env chsh grep chsh alias whereis echo mail date sort

What happens when you login?

After you login (typing your username and password), the operating system starts your first shell.
The passwd file stores your default shell as the last field in the user record.
The shell responds by giving you a prompt:
If you are the root user the prompt looks like this:

[root@ponto root]#

Changing your default shell

$ chsh pattyo

Changing shell for pattyo.

Password:

New shell [/bin/bash]: /bin/tcsh

Shell changed.

$ grep pattyo /etc/passwd
pattyo:x:500:500:PattyO:/home/pattyo:/bin/tcsh

How the shell interprets your commands

The shell interprets your commands in a specific order

1. checks to see if it matches an internal shell command
Every shell has a few built in commands.
2. checks to see if the command is an alias for another command
The alias command will list all the aliases defined in your environment.
$ alias
alias mv='mv -i'
alias new='ls -lt|head'
3. searches the hard disk for the program based on your PATH
What is your path?
$ env
...
PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:\
/home/pattyo/bin:/sbin:/usr/sbin
...
The env command displays your environment.
This is also the environment you get when you su - to some other user on the system.
What happens when a command is not in your path?
$ fortune
bash: fortune: command not found
Use the whereis command to locate the command.
$ whereis fortune
fortune: /usr/games/fortune /usr/share/man/man6/fortune.6.gz
Now use the absolute (full) path name to execute the program.
$ /usr/games/fortune
Art is a lie which makes us realize the truth.
-- Picasso

Shell Types

Major UNIX shells:

Bourne shell (sh)

Similar to first UNIX shell developed by Bell Labs

C shell (csh, tcsh)

Developed at UC Berkeley, similar format to C programming language

Korn shell (ksh, pdksh)

AT&T Bell Labs, based on Bourne shell but some C shell functionality

Bash shell (bash)

Bourne Again Shell, the default Linux shell
Features of Korn and C shell
sh-compatible interpreter

$ file /etc/init.d/*
...
/etc/init.d/sshd: Bourne-Again shell script text executable
...
$ head -1 /etc/init.d/sshd
#!/bin/bash

Find out what shells are available on your system:

$ cat /etc/shells

bin/bash bin/sh bin/tcsh bin/csh bin/ash bin/bsh bin/bash2

Metacharacters and Windcards

Metacharacters have special meaning to the shell
Should never use them as part of a file name

! # $ % ^ & * } ( ) [ ] | \ ; < > ? /

Wildcards and pattern matching

Wildcards are a subset of metacharacters used in pattern matching

* ? ! [ ]

Why are they so powerful?

Unix/Linux shells can reference more than one file name using wildcards.

The process is called pattern-matching

Asterisk *

The shell performs wildcard expansion when the * is used
The shell substitutes all possible file names that match the characters included before or after the asterisk.

zero to any number of additional characters
as long as the specified string or character is also present

Example:

Create some test files

Copy all files whose names begin with n to the doc directory:


$ touch news new nest net ne nac pack n
$ cp n* doc
$ ls doc 
 n  nac  ne  nest  new  news

Echo all files whose file names begin wiht test1 to the screen:


$ touch test1 test2 test1.1 test1.2 test1.3
$ echo test1* 
 test1 test1.2 test1.3

How do you match characters in the middle and beginning of files?


$ touch baba aba
$ ls *a* 
 aba baba nac pack
$ ls a*
 aba
$ ls *a
 aba  baba
$ ls a*a
 aba

The exception is that you won't get a match with files that begin with a dot (.)
If you want to match hidden files (beginning with '.') you need to put '.' in the search string:


$ cd
$ echo .bash* 
.bash_history  .bash_logout  .bash_profile  .bashrc

Note: You can use the echo command to list the files in a directory. The echo command is built into the shell.

Question Mark ?

The shell substitutes file names that have only a single character in the same position as the question mark.
The shell is still doing wildcard expansion but only to a single character


$ mkdir test
$ cd test
$ touch net new few news
$ ls ne?  
 net new

$ rm -i ?e?
 rm: remove `few'? y
 rm: remove `net'? y
 rm: remove `new'? y

Warning: Using wildcards with certain command like rm, can be dangerous!
```
$ ls /etc/????wd
 /etc/passwd
```
Why does this command only echo the input back to the screen?
```
$ echo ?bash*
?bash*
```

Lists [ ]

Used for matching one of several characters

$ touch net new few myfile top

$ ls ne[stw]

net new

Note: Only one of s, t, or w is mached

Exclamation Point !

Used within square brackets as a wildcard to exclude possibilites
! means not in programming or shell scripting
Below we are looking for files that don't begin with a t or an n
$ echo [!tn]*
myfile few

Square Brackets for Ranges

The position defined by the location of the square brackets is expanded by the shell but to only one character.

$ ls /etc/rc.d/rc3.d/K[1-5]* K20bootparamd K20rstatd K20rusersd K20rwhod K28amd K35dhcpd K55routed

Remember, the symbolic links found in this directory are executed based on the run-level your system is moving to (S for start, K for kill).

Quoting Metacharacters

to disable shell interpretation of metacharacter and variables

allows metacharacters to be interpreted literally


$ echo "*.*"
  *.*

$ echo *.*
    prints a listing of the files in your current directory

always use single and double quotes in pairs

Single quotes (' ')	ignore all metacharacters between quotes	`echo '$HOME'` `$HOME`
Double quotes (" ")	ignore all metacharacters except $, `, and \	`echo "$HOME"` `/home/pattyo`
Backslash (\)	ignore the special meaning of the next metacharacter	`echo \$HOME` `$HOME`

Shell Redirection and Piping

The shell figures out which part of the command line is the command and which are the arguments.
The shell sets up I/O according to the command line.

File Descriptors

Automatically opened for each process in the system.
Integers starting with 0 that index an array of file information within a process.

When a process starts it has three file descriptors:

input (0):

where the command expects to find its input
usually, the keyboard

output (1)

where the command expects to send its results
usually, the screen

standard error (2)

where the command expects to send its error messages
usually, the screen or the system console

Input Redirection

The mail program is instructed to take someletter as the standard input:

$ mail someuser < somefile
0 stdin: somefile (changed) 1 stdout (unchanged)-- the screen 2 stderr (unchanged)-- the screen
$ mail someuser -s "my subject" < somefile

Output Redirection

$ who > filename

The shell creates the new file, filename, and connects the output of the who utility to the new file.

NOTE: If filename already existed, it will be deleted as soon as the who command executes!

$ cal 10 2001 > thismonth $ cat thismonth October 2001 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

The cal command's descriptor table follows:

0 stdin (unchanged)-- the keyboard 1 stdout: thismonth (changed) 2 stderr (unchanged)-- the screen

>> Redirecting output and appending to an existing file

To add data to a file or avoid overwirting existing data

command >> filename

$ who > whoisthere $ date >> whoisthere $ who >> whoisthere $ cat whoisthere

Output redirection, following by input redirection

$ ls -l > ls.out $ sort -n +4 < ls.out

The sort command is followed by the -n option for numerical data and the "+4" option which tells sort to sort the file based on the 4th column (*starting from 0).

What is contained in the 5th column in the ls -l command?

What will this command do? Try it and find out!

$ cd /tmp
$ echo 2 * 3 > 6 more stuff here

Now examine the contents of the file, 6, you just created:

$ cat 6

You instructed the shell to echo the number 2, list all the files in the current directory, echo the number 3, then redirect everything to the file, 6.

What would be the effect of quoting this command the way you see below?

$ echo '2 * 3 > 6 more stuff here'

The shell simply echos your string to the terminal because it is ignoring the metacharacters in the command line between the quotes.

Look at the following three commands. What do they have in common?

$ cat < file1 > file2 $ cp file1 file2 $ cat file1 > file2

Using cat and Redirecting Output

The cat command is normally used to list the contents of a file

$ cat /etc/passwd

You can use cat with redirection to create a new file:

$ cat > newfilesome text here some more text here <Ctrl>d

Here are the descriptores for the cat > newfile example:

0 stdin (unchanged)-- keyboard 1 stdout: newfile (changed) 2 stderr (unchanged)-- screen

This example creates a file called file3 which contains the combined contents of file1 and file2.

$ cat file1 file2 > file3<

Is it possible to concatinate two files into one file and retain the same name as one of the original files?

example 1:

$ cat file1 file2 > file2

example 2:

$ cat file2 >> file1

Example 1 won't work because the shell overwrites file2 as soon as the cat command is interpretted.
- file2 will only contain the contents of file1 since you have overwritten file2 with the destructrive '>'.

2> Error Redirection

By default, the standard error ouput of a command goes to the screen or the system console.

If you want to save the error messages you can append the redirector to standard output of your command.

$ command 2> file1

Lets try catting a file that doesn't exist:

$ cat dog
cat: dog: No such file or directory

Now save the error output to a file:

$ cat dog 2> errfile

$ cat errfile

cat: dog: No such file or directory

If errfile doesn't exist, it is created.
If errfile already exists, its contents are overwritten.

0 stdin (unchanged)-- keyboard 1 stdout (unchanged)-- screen 2 stderr: errfile (changed)

2> for Redirecting Error Output to the "Bit Bucket"

Sending standard error to the bit bucket. /dev/null is the bit bucket. What goes in, never comes out. You won't see the error message on the screen.

$ cat dog 2> /dev/null

Redirecting and appending error output

Simply add the '2' file descriptor.

command 2>> filename

Combined Redirection

Combination of two or more redirections of input, output, and errors. Typical Combined Redirection

The order in which the redirections appear doesn't matter if not associated to the same file
command < infile > outfile 2> errfile command >> apendfile 2>> errfile < infile
In the first example, if outfile and errfile already contain information, that infomation is overwirtten.

In the second example, the results from command execution are appended to appendfile, preserving whatever information may already be in the file.
```
$ sort > /tmp/myoutput < /etc/passwd 2> /tmp/myerrs
$ sort >> /tmp/myoutput < /etc/passwd 2>> /tmp/myerrs
```

Association Order does matter sometimes!

Saving standard error and standard output to the same location.

The 2>&1 redirector instructs the shell to send standard error to the same place as standard output. In this case it goes to the file, logfile.

$ command > logfile 2>&1

In this example stderr is redirected to the standard output location, the screen, and then standard output is redirected to outfile.

$ command 2>&1 > outfile

Here the errors go to the screen the output goes to the file (assuming the file junk is not empty).

$ cat dog junk 2>&1 > outfile cat: dog: No such file or directory

Here the errors and output go to the file, logfile:
$ cat dog junk > logfile 2>&1 $ cat logfile cat: dog: No such file or directory
File descriptor defaults: 1> is equivalent to >, and 0< is the same as <.
These commands are essentially the same:
$ date 1> now $ date > now

Connecting Commands with Pipes

Any command to the left of a pipe must send its ouput to standard ouput
Any command to the right of a pipe must take its input from its standard input
The output from the command on the left is given over as input to the command on the right
$ ls -lt | head
$ ls | wc -w

Remember our earlier examples of input and output redirection:

$ ls -l > ls.out
$ sort +4 < ls.out

Here is a way to put those two commands together using a pipe:

$ ls -l | sort +4

Using Filters

A filter reads its input form stdin, changes it in some way, and writes to stdout
This example return the number of files in your home directory that begin with the letter n.

$ ls | grep ^n | wc -l
4

The output of ls is fed into grep
The hat '^' matches the beginning of a line
grep searches for file names that begin with the letter n
That output is fed into wc
wc counts the number of lines it received from grep.

Warning

Regular expressions are not consistent from program to program in Unix
The meaning of the asterisk (*) in filename expansion is different from programs which support regular expressions. Example
- This command doesn't work:
- The shell is not doing file expansion. It is looking for a file named n*

Using the tee Command to Split Output

To see the output on the screen and also save it to a file:

$ date | tee mynow_file

The tee command can be used for capturing a snapshot of information at a specific point in a pipe.

tee puts a copy of the data at that point into a file
then passes the original data to standard output

$ ls | tee ls.save | wc -l

This command creates the file "ls.save" which contains the output of ls. The results of wc -l are displayed on the screen.

You can send both stderr and stdout through a pipe.

$ command 2>&1 | tee logfile

Here both stderr and stdout go to the file logfile as well as to the screen. The tee command writes its output to both a file and the screen.

The Backslash for Escaping Special Characters

Since the single quote is a special character, how do you echo a string that contains a quote such as this:

$ echo my name's patty

You would have to escape the quote.

$ echo my name\'s patty
my name's patty

Line Continuations

This shell feature is used when the options and arguments appended to a single command cause you to type past the lenght of the line.

It is typically used in shell scripting to make the script more readable.

$ cmd continued_cmd \
> continued_cmd

Have you ever wondered why your prompt looked like this after typing a command?

>

The return key is just another character that can be escaped. You might have hit the backslash by accident. In this case, the shell is waiting for more input.
You escaped the return character.

What if you really want the backslash in your output.

$ echo i like the \ character
$ echo i like the \\ character