Shell for Dummies

...an introduction to basic Linux/Unix shell concepts

With desktop environments, like KDE and Gnome, it is possible to do almost anything without ever touching a text console or even a shell. But why then are all the "gurus" so hung up on this outdated concept? The answer is simple: you can easily create two or three users per week on a graphical terminal using the mouse. But if you have to create 500 student accounts within two days before the new term starts, the task becomes a bit more tricky.

We will use this example to walk through a few shell commands, of course all the commands shown here should enable the reader to perform other tasks as well - if only used with a bit of creativity.

A little word of caution: although I list the names of real system files here, you should try these operations on copies of these files and not on the originals themselves. Furthermore: work as a normal user, not as root - otherwise you might inadvertently destroy something important.

The task

You are the new intern in the IT department. Of course the staff already sent you to get the bogus harddisk rewinder, empty the null-device and similiar jokes.

With that in mind let me show you first how to handle this:
wc /dev/null will immediately show everybody that the null device is indeed already empty an there is no need to send you away really.

But the command is useful for serious tasks as well: wc /etc/passwd will count the characters, words and lines of the file containing all user accounts. wc -l /etc/passwd will only show the lines. Each line is one account and we immediately realize that there are already 6789 accounts. Most of them students.

But how many students exactly? Staff accounts use the name of the staff member, while student accounts use the first letter of the first name and the first letter of the family name plus a number. The layout of the /etc/passwd file is this:
operator:x:237:37:Operator:/home/drive23/operator:/bin/sh
all fields are separated with a colon, the first field contains the user name, the second one a "!" for no login permitted and a "x" for a reference into /etc/shadow, which contains the encrypted password. The third and fourth field contain the user ID (237) and the users primary group ID(37). Unix and Linux can differentiate users only by a number, the names are there only for the convenience of the user. The fifth field contains the users real name, the sixth the home directory and the last one the default shell of the user.

To answer our question, we need to filter out all lines that contain two letters and a number in the first field. It is easy to filter out a line that contains a specific word:
grep rator /etc/passwd
will filter out any line that contains the letters "rator" in this order (eg. the line of the user "operator". But that doesn't help us much. We need more complicated patterns:
egrep '^[a-z][a-z][0-9]+:' /etc/passwd
will search at the beginning of each line (^) for two letters ([a-z]), followed by digits ([0-9]), but there must be at least one digit (+), followed by a colon. How did we find out how to construct this pattern? Thankfully Unix and Linux have the manual always available: man egrep.

But how do we count those lines? We have two possibilities. First we can store those lines in a new file (students) and then count them:
egrep '^[a-z][a-z][0-9]+:' /etc/passwd >students wc -l students
The >-sign redirects the output of a command into a file. Likewise the 2>-sign redirects the error messages of the command into the file, >> and 2>> append to the file, and < redirects the content of the file into the input of the command. But we can also count the lines by simply redirecting the output of the first command into the input of the second command:
egrep '^[a-z][a-z][0-9]+:' /etc/passwd | wc -l
the pipe-sign (|) does this. This works because most Unix-programs that expect a filename as argument use their standard input instead if no filename was given. So we know now that we have exactly 6178 students.

The immatriculation department sent us the names and science-department of about 500 new students in an excel file. This is extremely unhelpful, so we convert the excel file into CSV-format using OpenOffice and go on from here:
Bell,Alexander-Graham,et Neuman,John von,cs ...

Our task is to create the necessary user groups and users.

Groups

Let's find out which departments exist this term. We want the third field of a comma-delimited list:
cut -d , -f 3 students.csv
but this list is yet unsorted and very redundant:
cut -d , -f 3 students.csv | sort |uniq
sort will sort the lines in ASCII order (other orders can be used using additional arguments) and uniq will delete any redundant lines.

With this more reasonable list we can create the user groups. We could now type addgroup et05 etc.pp. for each group we want to create. Or we could even automate it a bit with:
for i in et cs mech ...; do addgroup ${i}05 done
the for loop can be used to go through lists (like file names) and do things with them. The IT department defined that this years groups are in the range of 430-449, so we need to work a bit more complicated. First off, we can get the list of departments directly from our line above using either the italic tickmarks (`command`) or the $()-expression. Next, we add the lines to /etc/group ourselves:
gid=430
foreach i in $(cut -d , -f 3 students.csv | sort |uniq) ; do
echo ${i}05:x:$gid: >>/etc/group
echo ${i}05,$gid >>groups.csv
gid=$(expr $gid + 1)
done
the line gid=430 defines the variable gid to contain the value 430, the {} around the variable i tells the shell where the variable name ends and the text "05" starts, the command expr can be used to calculate. For later convenience we store the new groups in a csv file.

Users

Our UIDs start at 11000 this year. We now need to split the file with the student names into separate lines. We already know that cat outputs a file. We can read it in with the function read, which reads in a line and stores it in a variable. Above we learned what the foreach loop is. The while loop works a bit different - it goes on until the command listed in it returns "false" (a return code different from 0).

cat student.csv | while read line ; do echo another student: $line done

Like above we can split the line into lastname, firstname and group:
lastname=`echo $line | split -d , -f 1`
with -f 2 and -f 3 for firstname and group respectively.

Now we need to find a user name for each student. We start with the first characters:
usrname=`echo $firstname|cut -c 1``echo $lastname|cut -c 1`
as you see cut can be used to split out specific characters as well.

Now we need to find the next free number (see above for how to separate student-accounts from /etc/passwd, I'll assume we stored that in passwd.st):
cat passwd.st|egrep ^$usrname|cut -d : -f 1|cut -c 3- |sort -n|tail -1
the first cut will cut out the account name, the second cut will separate the number of it, sort -n will sort those numerically and tail will give us the last line containing the highest number (head would have given us the first line). Store that value in a variable using `` or $()!

We can calculate the next free number by first checking whether we found anything at all and then increasing it:
test -z "$number" && number=1 || number=`expr $number + 1`
test can be used to perform dozens of different tests (eg. file size, string comparisons, etc.), expr works as above. The && and || constructs are new. The && will execute the command after it, if the command before it ended with success (returns 0), the || will execute the command after it if the commands before it returned no success (returns non-zero).

Now we can easily create the new user name: usrname=$usrname$number.

We still need to find out about the group-ID:
grep $group groups.csv|cut -d , -f 2
by now you will know what that is.

Now we can create the line for /etc/passwd:
uid=11000 cat students.csv |while read line ; do ... echo $usrname:x:$uid:$gid:$firstname $lastname:/home/2005/$usrname:/bin/bash >>newpasswd ... uid=`expr $uid + 1` done

I store the results in a file newpasswd, since it is generally a good idea to not manipulate system files directly, but to test it on another file first.

We still need to create a line for /etc/shadow. We will assume paranoid security and force the students to visit the IT department first time they want to login, otherwise eg. the htpasswd command from Apache could be used to create an encrypted password.
echo $usrname:!:0:0:99999:7::: >>newshadow

Now we only need to create the home directory of the user:
cp -a /etc/skel /home/2005/$usrname chown -r $uid /home/2005/$usrname chgrp -r $gid /home/2005/$usrname
cp -a will copy the system wide template for home directories (/etc/skel) to the homedirectory of our new user and chown -r and chgrp -r will change the user and group ID of it recursively.

Now we only need to put that all into one big script file and let it do its work.

Shell for Dummies

The task

Groups

Users

Further reading