Advanced Bash-Scripting Guide

]> Advanced Bash-Scripting Guide An in-depth exploration of the art of shell scripting Mendel Cooper

thegrendel@theriver.com

5.4.12 23 September 2008 978-1-4357-5219-1 5.2 16 Mar 2008 mc 'SILVERBERRY' release: Important Update. 5.3 05 May 2008 mc 'GOLDENBERRY' release: Minor Update. 5.4 21 July 2008 mc 'ANGLEBERRY' release: Major Update. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX wisdom and lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts. The latest update of this document, as an archived, bzip2-ed tarball including both the SGML source and rendered HTML, may be downloaded from the author's home site. A pdf version is also available ( pdf mirror site). See the change log for a revision history. For Anita, the source of all the magic Introduction The shell is a command interpreter. More than just the insulating layer between the operating system kernel and the user, it's also a fairly powerful programming language. A shell program, called a script, is an easy-to-use tool for building applications by gluing together system calls, tools, utilities, and compiled binaries. Virtually the entire repertoire of UNIX commands, utilities, and tools is available for invocation by a shell script. If that were not enough, internal shell commands, such as testing and loop constructs, lend additional power and flexibility to scripts. Shell scripts are especially well suited for administrative system tasks and other routine repetitive tasks not requiring the bells and whistles of a full-blown tightly structured programming language. Why Shell Programming? No programming language is perfect. There is not even a single best language; there are only languages well suited or perhaps poorly suited for particular purposes. --Herbert Mayer A working knowledge of shell scripting is essential to anyone wishing to become reasonably proficient at system administration, even if they do not anticipate ever having to actually write a script. Consider that as a Linux machine boots up, it executes the shell scripts in /etc/rc.d to restore the system configuration and set up services. A detailed understanding of these startup scripts is important for analyzing the behavior of a system, and possibly modifying it. The craft of scripting is not hard to master, since the scripts can be built in bite-sized sections and there is only a fairly small set of shell-specific operators and options These are referred to as builtins, features internal to the shell. to learn. The syntax is simple and straightforward, similar to that of invoking and chaining together utilities at the command line, and there are only a few rules governing their use. Most short scripts work right the first time, and debugging even the longer ones is straightforward. A shell script is a quick-and-dirty method of prototyping a complex application. Getting even a limited subset of the functionality to work in a script is often a useful first stage in project development. This way, the structure of the application can be tested and played with, and the major pitfalls found before proceeding to the final coding in C, C++, Java, Perl, or Python. Shell scripting hearkens back to the classic UNIX philosophy of breaking complex projects into simpler subtasks, of chaining together components and utilities. Many consider this a better, or at least more esthetically pleasing approach to problem solving than using one of the new generation of high powered all-in-one languages, such as Perl, which attempt to be all things to all people, but at the cost of forcing you to alter your thinking processes to fit the tool. According to Herbert Mayer, a useful language needs arrays, pointers, and a generic mechanism for building data structures. By these criteria, shell scripting falls somewhat short of being useful. Or, perhaps not. . . . When not to use shell scripts Resource-intensive tasks, especially where speed is a factor (sorting, hashing, recursion Although recursion is possible in a shell script, it tends to be slow and its implementation is often an ugly kludge. ...) Procedures involving heavy-duty math operations, especially floating point arithmetic, arbitrary precision calculations, or complex numbers (use C++ or FORTRAN instead) Cross-platform portability required (use C or Java instead) Complex applications, where structured programming is a necessity (type-checking of variables, function prototypes, etc.) Mission-critical applications upon which you are betting the future of the company Situations where security is important, where you need to guarantee the integrity of your system and protect against intrusion, cracking, and vandalism Project consists of subcomponents with interlocking dependencies Extensive file operations required (Bash is limited to serial file access, and that only in a particularly clumsy and inefficient line-by-line fashion.) Need native support for multi-dimensional arrays Need data structures, such as linked lists or trees Need to generate / manipulate graphics or GUIs Need direct access to system hardware Need port or socket I/O Need to use libraries or interface with legacy code Proprietary, closed-source applications (Shell scripts put the source code right out in the open for all the world to see.) If any of the above applies, consider a more powerful scripting language -- perhaps Perl, Tcl, Python, Ruby -- or possibly a compiled language such as C, C++, or Java. Even then, prototyping the application as a shell script might still be a useful development step. We will be using Bash, an acronym for Bourne-Again shell and a pun on Stephen Bourne's now classic Bourne shell. Bash has become a de facto standard for shell scripting on most flavors of UNIX. Most of the principles this book covers apply equally well to scripting with other shells, such as the Korn Shell, from which Bash derives some of its features, Many of the features of ksh88, and even a few from the updated ksh93 have been merged into Bash. and the C Shell and its variants. (Note that C Shell programming is not recommended due to certain inherent problems, as pointed out in an October, 1993 Usenet post by Tom Christiansen.) What follows is a tutorial on shell scripting. It relies heavily on examples to illustrate various features of the shell. The example scripts work -- they've been tested, insofar as was possible -- and some of them are even useful in real life. The reader can play with the actual working code of the examples in the source archive (scriptname.sh or scriptname.bash), By convention, user-written shell scripts that are Bourne shell compliant generally take a name with a .sh extension. System scripts, such as those found in /etc/rc.d, do not conform to this nomenclature. give them execute permission (chmod u+rx scriptname), then run them to see what happens. Should the source archive not be available, then cut-and-paste from the HTML or pdf rendered versions. Be aware that some of the scripts presented here introduce features before they are explained, and this may require the reader to temporarily skip ahead for enlightenment. Unless otherwise noted, the author of this book wrote the example scripts that follow. His countenance was bold and bashed not. --Edmund Spenser Starting Off With a Sha-Bang Shell programming is a 1950s juke box . . . --Larry Wall In the simplest case, a script is nothing more than a list of system commands stored in a file. At the very least, this saves the effort of retyping that particular sequence of commands each time it is invoked. <firstterm>cleanup</firstterm>: A script to clean up the log files in /var/log &ex1; There is nothing unusual here, only a set of commands that could just as easily be invoked one by one from the command line on the console or in an xterm. The advantages of placing the commands in a script go beyond not having to retype them time and again. The script becomes a tool, and can easily be modified or customized for a particular application. <firstterm>cleanup</firstterm>: An improved clean-up script &ex1a; Now that's beginning to look like a real script. But we can go even farther . . . <firstterm>cleanup</firstterm>: An enhanced and generalized version of above scripts. &ex2; Since you may not wish to wipe out the entire system log, this version of the script keeps the last section of the message log intact. You will constantly discover ways of fine-tuning previously written scripts for increased effectiveness. The sha-bang sha-bang ( #! #!) Also seen in the literature as she-bang or sh-bang. This derives from the concatenation of the tokens sharp (#) and bang (!). at the head of a script tells your system that this file is a set of commands to be fed to the command interpreter indicated. The #! is actually a two-byte Some flavors of UNIX (those based on 4.2 BSD) allegedly take a four-byte magic number, requiring a blank after the ! -- #! /bin/sh. According to Sven Mascheck this is probably a myth. magic number magic number, a special marker that designates a file type, or in this case an executable shell script (type man magic for more details on this fascinating topic). Immediately following the sha-bang is a path name. This is the path to the program that interprets the commands in the script, whether it be a shell, a programming language, or a utility. This command interpreter then executes the commands in the script, starting at the top (line following the sha-bang line), ignoring comments. The #! line in a shell script will be the first thing the command interpreter (sh or bash) sees. Since this line begins with a #, it will be correctly interpreted as a comment when the command interpreter finally executes the script. The line has already served its purpose - calling the command interpreter. If, in fact, the script includes an extra #! line, then bash will interpret it as a comment. #!/bin/bash echo "Part 1 of script." a=1 #!/bin/bash # This does *not* launch a new script. echo "Part 2 of script." echo $a # Value of $a stays at 1. #!/bin/sh #!/bin/bash #!/usr/bin/perl #!/usr/bin/tcl #!/bin/sed -f #!/usr/awk -f Each of the above script header lines calls a different command interpreter, be it /bin/sh, the default shell (bash in a Linux system) or otherwise. This allows some cute tricks. #!/bin/rm # Self-deleting script. # Nothing much seems to happen when you run this... except that the file disappears. WHATEVER=65 echo "This line will never print (betcha!)." exit $WHATEVER # Doesn't matter. The script will not exit here. # Try an echo $? after script termination. # You'll get a 0, not a 65. Also, try starting a README file with a #!/bin/more, and making it executable. The result is a self-listing documentation file. (A here document using cat is possibly a better alternative -- see ). Using #!/bin/sh, the default Bourne shell in most commercial variants of UNIX, makes the script portable to non-Linux machines, though you sacrifice Bash-specific features. The script will, however, conform to the POSIX Portable Operating System Interface, an attempt to standardize UNIX-like OSes. The POSIX specifications are listed on the Open Group site. sh standard. Note that the path given at the sha-bang must be correct, otherwise an error message -- usually Command not found. -- will be the only result of running the script. To avoid this possibility, a script may begin with a #!/bin/env bash sha-bang line. This may be useful on UNIX machines where bash is not located in /bin #! can be omitted if the script consists only of a set of generic system commands, using no internal shell directives. The second example, above, requires the initial #!, since the variable assignment line, lines=50, uses a shell-specific construct. If Bash is your default shell, then the #! isn't necessary at the beginning of a script. However, if launching a script from a different shell, such as tcsh, then you will need the #!. Note again that #!/bin/sh invokes the default shell interpreter, which defaults to /bin/bash on a Linux machine. This tutorial encourages a modular approach to constructing a script. Make note of and collect boilerplate code snippets that might be useful in future scripts. Eventually you will build quite an extensive library of nifty routines. As an example, the following script prolog tests whether the script has been invoked with the correct number of parameters. E_WRONG_ARGS=65 script_parameters="-a -h -m -z" # -a = all, -h = help, etc. if [ $# -ne $Number_of_expected_args ] then echo "Usage: `basename $0` $script_parameters" # `basename $0` is the script's filename. exit $E_WRONG_ARGS fi Many times, you will write a script that carries out one particular task. The first script in this chapter is an example of this. Later, it might occur to you to generalize the script to do other, similar tasks. Replacing the literal (hard-wired) constants by variables is a step in that direction, as is replacing repetitive code blocks by functions. Invoking the script Having written the script, you can invoke it by sh scriptname, Caution: invoking a Bash script by sh scriptname turns off Bash-specific extensions, and the script may therefore fail to execute. or alternatively bash scriptname. (Not recommended is using sh <scriptname, since this effectively disables reading from stdin within the script.) Much more convenient is to make the script itself directly executable with a chmod. Either: chmod 555 scriptname (gives everyone read/execute permission) A script needs read, as well as execute permission for it to run, since the shell needs to be able to read it. or chmod +rx scriptname (gives everyone read/execute permission) chmod u+rx scriptname (gives only the script owner read/execute permission) Having made the script executable, you may now test it by ./scriptname. Why not simply invoke the script with scriptname? If the directory you are in ($PWD) is where scriptname is located, why doesn't this work? This fails because, for security reasons, the current directory (./) is not by default included in a user's $PATH. It is therefore necessary to explicitly invoke the script in the current directory with a ./scriptname. If it begins with a sha-bang line, invoking the script calls the correct command interpreter to run it. As a final step, after testing and debugging, you would likely want to move it to /usr/local/bin (as root, of course), to make the script available to yourself and all other users as a systemwide executable. The script could then be invoked by simply typing scriptname [ENTER] from the command line. Preliminary Exercises System administrators often write scripts to automate common tasks. Give several instances where such scripts would be useful. Write a script that upon invocation shows the time and date, lists all logged-in users, and gives the system uptime. The script then saves this information to a logfile. Basics Special Characters What makes a character special? If it has a meaning beyond its literal meaning, a meta-meaning, then we refer to it as a special character. <anchor id="scharlist1">Special Characters Found In Scripts and Elsewhere # # special character # comment Comments Lines beginning with a # (with the exception of #!) are comments and will not be executed. # This line is a comment. Comments may also occur following the end of a command. echo "A comment will follow." # Comment here. # ^ Note whitespace before # Comments may also follow whitespace at the beginning of a line. # A tab precedes this comment. A command may not follow a comment on the same line. There is no method of terminating the comment, in order for live code to begin on the same line. Use a new line for the next command. Of course, a quoted or an escaped # in an echo statement does not begin a comment. Likewise, a # appears in certain parameter-substitution constructs and in numerical constant expressions. echo "The # here does not begin a comment." echo 'The # here does not begin a comment.' echo The \# here does not begin a comment. echo The # here begins a comment. echo ${PATH#*:} # Parameter substitution, not a comment. echo $(( 2#101011 )) # Base conversion, not a comment. # Thanks, S.C. The standard quoting and escape characters (" ' \) escape the #. Certain pattern matching operations also use the #. ; ; special character ; separator Command separator [semicolon] Permits putting two or more commands on the same line. echo hello; echo there if [ -x "$filename" ]; then # Note that "if" and "then" need whitespace #+ separation. Why? echo "File $filename exists."; cp $filename $filename.bak else echo "File $filename not found."; touch $filename fi; echo "File test complete." Note that the ; sometimes needs to be escaped. ;; ;; special character case ;; Terminator in a <link linkend="caseesac1">case</link> option [double semicolon] case "$variable" in abc) echo "\$variable = abc" ;; xyz) echo "\$variable = xyz" ;; esac . . special character . dot command source <quote>dot</quote> command [period] Equivalent to source (see ). This is a bash builtin. . . special character . filename part of a filename <quote>dot</quote>, as a component of a filename When working with filenames, a leading dot is the prefix of a hidden file, a file that an ls will not normally show. bash$ touch .hidden-file bash$ ls -l total 10 -rw-r--r-- 1 bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo 877 Dec 17 2000 employment.addressbook bash$ ls -al total 14 drwxrwxr-x 2 bozo bozo 1024 Aug 29 20:54 ./ drwx------ 52 bozo bozo 3072 Aug 29 20:51 ../ -rw-r--r-- 1 bozo bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo bozo 877 Dec 17 2000 employment.addressbook -rw-rw-r-- 1 bozo bozo 0 Aug 29 20:54 .hidden-file When considering directory names, a single dot represents the current working directory, and two dots denote the parent directory. bash$ pwd /home/bozo/projects bash$ cd . bash$ pwd /home/bozo/projects bash$ cd .. bash$ pwd /home/bozo/ The dot often appears as the destination (directory) of a file movement command, in this context meaning current directory. bash$ cp /home/bozo/current_work/junk/* . Copy all the junk files to $PWD. . . special character . character match match single character <quote>dot</quote> character match When matching characters, as part of a regular expression, a dot matches a single character. " <link linkend="dblquo">partial quoting</link> [double quote] "STRING" preserves (from interpretation) most of the special characters within STRING. See . ' <link linkend="snglquo">full quoting</link> [single quote] 'STRING' preserves all special characters within STRING. This is a stronger form of quoting than "STRING". See . , <link linkend="commaop">comma operator</link> The comma operator An operator is an agent that carries out an operation. Some examples are the common arithmetic operators, + - * /. In Bash, there is some overlap between the concepts of operator and keyword. links together a series of arithmetic operations. All are evaluated, but only the last one is returned. let "t2 = ((a = 9, 15 / 3))" # Set "a = 9" and "t2 = 15 / 3" The comma operator can also concatenate strings. for file in /{,usr/}bin/*calc # ^ Find all executable files ending in "calc" #+ in /bin and /usr/bin directories. do if [ -x "$file" ] then echo $file fi done # /bin/ipcalc # /usr/bin/kcalc # /usr/bin/oidcalc # /usr/bin/oocalc # Thank you, Rory Winston, for pointing this out. \ <link linkend="escp">escape</link> [backslash] A quoting mechanism for single characters. \X escapes the character X. This has the effect of quoting X, equivalent to 'X'. The \ may be used to quote " and ', so they are expressed literally. See for an in-depth explanation of escaped characters. / Filename path separator [forward slash] Separates the components of a filename (as in /home/bozo/projects/Makefile). This is also the division arithmetic operator. ` <link linkend="commandsubref">command substitution</link> The `command` construct makes available the output of command for assignment to a variable. This is also known as backquotes or backticks. : : special character : null command true endless loop null command [colon] This is the shell equivalent of a NOP (no op, a do-nothing operation). It may be considered a synonym for the shell builtin true. The : command is itself a Bash builtin, and its exit status is true (0). : echo $? # 0 Endless loop: while : do operation-1 operation-2 ... operation-n done # Same as: # while true # do # ... # done Placeholder in if/then test: if condition then : # Do nothing and branch ahead else # Or else ... take-some-action fi Provide a placeholder where a binary operation is expected, see and default parameters. : ${username=`whoami`} # ${username=`whoami`} Gives an error without the leading : # unless "username" is a command or builtin... Provide a placeholder where a command is expected in a here document. See . Evaluate string of variables using parameter substitution (as in ). : ${HOSTNAME?} ${USER?} ${MAIL?} # Prints error message #+ if one or more of essential environmental variables not set. Variable expansion / substring replacement. In combination with the > redirection operator, truncates a file to zero length, without changing its permissions. If the file did not previously exist, creates it. : > data.xxx # File "data.xxx" now empty. # Same effect as cat /dev/null >data.xxx # However, this does not fork a new process, since ":" is a builtin. See also . In combination with the >> redirection operator, has no effect on a pre-existing target file (: >> target_file). If the file did not previously exist, creates it. This applies to regular files, not pipes, symlinks, and certain special files. May be used to begin a comment line, although this is not recommended. Using # for a comment turns off error checking for the remainder of that line, so almost anything may appear in a comment. However, this is not the case with :. : This is a comment that generates an error, ( if [ $x -eq 3] ). The : also serves as a field separator, in /etc/passwd, and in the $PATH variable. bash$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin:/usr/games ! ! special character ! not logical not reverse (or negate) the sense of a test or exit status [bang] The ! operator inverts the exit status of the command to which it is applied (see ). It also inverts the meaning of a test operator. This can, for example, change the sense of equal ( = ) to not-equal ( != ). The ! operator is a Bash keyword. In a different context, the ! also appears in indirect variable references. In yet another context, from the command line, the ! invokes the Bash history mechanism (see ). Note that within a script, the history mechanism is disabled. * * special character * wild card globbing regular expression wild card [asterisk] The * character serves as a wild card for filename expansion in globbing. By itself, it matches every filename in a given directory. bash$ echo * abs-book.sgml add-drive.sh agram.sh alias.sh The * also represents any number (or zero) characters in a regular expression. * * special character * multiplication exponentiation arithmetic operator <link linkend="arops1">arithmetic operator</link> In the context of arithmetic operations, the * denotes multiplication. A double asterisk, **, is the exponentiation operator. ? ? special character ? test operator test token test operator Within certain expressions, the ? indicates a test for a condition. In a double-parentheses construct, the ? can serve as an element of a C-style trinary operator, ?:. (( var0 = var1<98?9:21 )) # ^ ^ # if [ "$var1" -lt 98 ] # then # var0=9 # else # var0=21 # fi In a parameter substitution expression, the ? tests whether a variable has been set. ? ? special character ? wild card globbing regular expression wild card The ? character serves as a single-character wild card for filename expansion in globbing, as well as representing one character in an extended regular expression. $ $ special character $ variable substitution <link linkend="varsubn">Variable substitution</link> (contents of a variable) var1=5 var2=23skidoo echo $var1 # 5 echo $var2 # 23skidoo A $ prefixing a variable name indicates the value the variable holds. $ $ special character $ regular expression end of line end-of-line In a regular expression, a $ addresses the end of a line of text. ${} ${} special character ${} parameter substitution <link linkend="paramsubref">Parameter substitution</link> $* $@ $* special character $* $@ positional parameters $@ <link linkend="appref">positional parameters</link> $? $? special character ? exit status variable exit status exit status variable The $? variable holds the exit status of a command, a function, or of the script itself. $$ $$ special character $$ process ID variable process ID process ID variable The $$ variable holds the process ID A PID, or process ID, is a number assigned to a running process. The PIDs of running processes may be viewed with a ps command. Definition: A process is an executing program, sometimes referred to as a job. of the script in which it appears. () command group (a=hello; echo $a) A listing of commands within parentheses starts a subshell. Variables inside parentheses, within the subshell, are not visible to the rest of the script. The parent process, the script, cannot read variables created in the child process, the subshell. a=123 ( a=321; ) echo "a = $a" # a = 123 # "a" within parentheses acts like a local variable. array initialization Array=(element1 element2 element3) {xxx,yyy,zzz..} special character {} brace expansion {xxx,yyy,zzz,...} Brace expansion cat {file1,file2,file3} > combined_file # Concatenates the files file1, file2, and file3 into combined_file. cp file22.{txt,backup} # Copies "file22.txt" to "file22.backup" A command may act upon a comma-separated list of file specs within braces. The shell does the brace expansion. The command itself acts upon the result of the expansion. Filename expansion (globbing) applies to the file specs between the braces. No spaces allowed within the braces unless the spaces are quoted or escaped. echo {file1,file2}\ :{\ A," B",' C'} file1 : A file1 : B file1 : C file2 : A file2 : B file2 : C {a..z} special character {} extended brace expansion {a..z} Extended Brace expansion echo {a..z} # a b c d e f g h i j k l m n o p q r s t u v w x y z # Echoes characters between a and z. echo {0..3} # 0 1 2 3 # Echoes characters between 0 and 3. The {a..z} extended brace expansion construction is a feature introduced in version 3 of Bash. {} {} special character {} block of code Block of code [curly brackets] Also referred to as an inline group, this construct, in effect, creates an anonymous function (a function without a name). However, unlike in a standard function, the variables inside a code block remain visible to the remainder of the script. bash$ { local a; a=123; } bash: local: can only be used in a function a=123 { a=321; } echo "a = $a" # a = 321 (value inside code block) # Thanks, S.C. The code block enclosed in braces may have I/O redirected to and from it. Code blocks and I/O redirection &ex8; Saving the output of a code block to a file &rpmcheck; Unlike a command group within (parentheses), as above, a code block enclosed by {braces} will not normally launch a subshell. Exception: a code block in braces as part of a pipe may run as a subshell. ls | { read firstline; read secondline; } # Error. The code block in braces runs as a subshell, #+ so the output of "ls" cannot be passed to variables within the block. echo "First line is $firstline; second line is $secondline" # Won't work. # Thanks, S.C. {} placeholder for text Used after xargs (replace strings option). The {} double curly brackets are a placeholder for output text. ls . | xargs -i -t cp ./{} $1 # ^^ ^^ # From "ex42.sh" (copydir.sh) example. anchor id="semicolonesc"> {} \; pathname Mostly used in find constructs. This is not a shell builtin. The ; ends the option of a find command sequence. It needs to be escaped to protect it from interpretation by the shell. [ ] [] special character [ ] test test Test expression between [ ]. Note that [ is part of the shell builtin test (and a synonym for it), not a link to the external command /usr/bin/test. [[ ]] [[]] special character [[ ]] test test Test expression between [[ ]]. More flexible than the single-bracket [ ] test, this is a shell keyword. See the discussion on the [[ ... ]] construct. [ ] [ ] special character array_element[ ] array element array element In the context of an array, brackets set off the numbering of each element of that array. Array[1]=slot_1 echo ${Array[1]} [ ] [ ] special character character range regular expression range of characters As part of a regular expression, brackets delineate a range of characters to match. (( )) (( )) special character (( )) integer comparison integer expansion Expand and evaluate integer expression between (( )). See the discussion on the (( ... )) construct. > &> >& >> < <> > >& >> < special character > special character >& special character >> special character < redirection <link linkend="ioredirref">redirection</link> scriptname >filename redirects the output of scriptname to file filename. Overwrite filename if it already exists. command &>filename redirects both the stdout and the stderr of command to filename. command >&2 redirects stdout of command to stderr. scriptname >>filename appends the output of scriptname to file filename. If filename does not already exist, it is created. [i]<>filename opens file filename for reading and writing, and assigns file descriptor i to it. If filename does not exist, it is created. <link linkend="processsubref">process substitution</link> (command)> <(command) In a different context, the < and > characters act as string comparison operators. In yet another context, the < and > characters act as integer comparison operators. See also . << redirection used in a <link linkend="heredocref">here document</link> <<< redirection used in a <link linkend="herestringsref">here string</link> < > < special character < > ASCII comparison > <link linkend="ltref">ASCII comparison</link> veg1=carrots veg2=tomatoes if [[ "$veg1" < "$veg2" ]] then echo "Although $veg1 precede $veg2 in the dictionary," echo -n "this does not necessarily imply anything " echo "about my culinary preferences." else echo "What kind of dictionary are you using, anyhow?" fi \< \> \< regular expression \< > word boundary > <link linkend="anglebrac">word boundary</link> in a <link linkend="regexref">regular expression</link> bash$ grep '\<the\>' textfile | | special character | pipe pipe Passes the output (stdout of a previous command to the input (stdin) of the next one, or to the shell. This is a method of chaining commands together. echo ls -l | sh # Passes the output of "echo ls -l" to the shell, #+ with the same result as a simple "ls -l". cat *.lst | sort | uniq # Merges and sorts all ".lst" files, then deletes duplicate lines. A pipe, as a classic method of interprocess communication, sends the stdout of one process to the stdin of another. In a typical case, a command, such as cat or echo, pipes a stream of data to a filter, a command that transforms its input for processing. Even as in olden times a philtre denoted a potion alleged to have magical transformative powers, so does a UNIX filter transform its target in (rougly) analogous fashion. (The coder who comes up with a love philtre that runs on a Linux machine will likely win accolades and honors.) cat $filename1 $filename2 | grep $search_word For an interesting note on the complexity of using UNIX pipes, see the UNIX FAQ, Part 3. The output of a command or commands may be piped to a script. #!/bin/bash # uppercase.sh : Changes input to uppercase. tr 'a-z' 'A-Z' # Letter ranges must be quoted #+ to prevent filename generation from single-letter filenames. exit 0 Now, let us pipe the output of ls -l to this script. bash$ ls -l | ./uppercase.sh -RW-RW-R-- 1 BOZO BOZO 109 APR 7 19:49 1.TXT -RW-RW-R-- 1 BOZO BOZO 109 APR 14 16:48 2.TXT -RW-R--R-- 1 BOZO BOZO 725 APR 20 20:56 DATA-FILE The stdout of each process in a pipe must be read as the stdin of the next. If this is not the case, the data stream will block, and the pipe will not behave as expected. cat file1 file2 | ls -l | sort # The output from "cat file1 file2" disappears. A pipe runs as a child process, and therefore cannot alter script variables. variable="initial_value" echo "new_value" | read variable echo "variable = $variable" # variable = initial_value If one of the commands in the pipe aborts, this prematurely terminates execution of the pipe. Called a broken pipe, this condition sends a SIGPIPE signal. >| >| special character >| redirection force noclobber force redirection (even if the <link linkend="noclobberref">noclobber option</link> is set) This will forcibly overwrite an existing file. || || special character || or logical operator <link linkend="orref">OR logical operator</link> In a test construct, the || operator causes a return of 0 (success) if either of the linked test conditions is true. & Run job in background A command followed by an & will run in the background. bash$ sleep 10 & [1] 850 [1]+ Done sleep 10 Within a script, commands and even loops may run in the background. Running a loop in the background &bgloop; A command run in the background within a script may cause the script to hang, waiting for a keystroke. Fortunately, there is a remedy for this. && && special character && and logical operator <link linkend="logops1">AND logical operator</link> In a test construct, the && operator causes a return of 0 (success) only if both the linked test conditions are true. - option, prefix Option flag for a command or filter. Prefix for an operator. Prefix for a default parameter in parameter substitution. COMMAND -[Option1][Option2][...] ls -al sort -dfu $filename if [ $file1 -ot $file2 ] then # ^ echo "File $file1 is older than $file2." fi if [ "$a" -eq "$b" ] then ^ echo "$a is equal to $b." fi if [ "$c" -eq 24 -a "$d" -eq 47 ] then ^ ^ echo "$c equals 24 and $d equals 47." fi param2=${param1:-$DEFAULTVAL} # ^ -- The double-dash prefixes long (verbatim) options to commands. sort --ignore-leading-blanks Used with a Bash builtin, it means the end of options to that particular command. This provides a handy means of removing files whose names begin with a dash. bash$ ls -l -rw-r--r-- 1 bozo bozo 0 Nov 25 12:29 -badname bash$ rm -- -badname bash$ ls -l total 0 The double-dash is also used in conjunction with set. set -- $variable (as in ) - - special character - redirection from/to stdin/stdout redirection from/to <filename>stdin</filename> or <filename>stdout</filename> [dash] bash$ cat - abc abc ... Ctl-D As expected, cat - echoes stdin, in this case keyboarded user input, to stdout. But, does I/O redirection using - have real-world applications? (cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -) # Move entire file tree from one directory to another # [courtesy Alan Cox <a.cox@swansea.ac.uk>, with a minor change] # 1) cd /source/directory # Source directory, where the files to be moved are. # 2) && # "And-list": if the 'cd' operation successful, # then execute the next command. # 3) tar cf - . # The 'c' option 'tar' archiving command creates a new archive, # the 'f' (file) option, followed by '-' designates the target file # as stdout, and do it in current directory tree ('.'). # 4) | # Piped to ... # 5) ( ... ) # a subshell # 6) cd /dest/directory # Change to the destination directory. # 7) && # "And-list", as above # 8) tar xpvf - # Unarchive ('x'), preserve ownership and file permissions ('p'), # and send verbose messages to stdout ('v'), # reading data from stdin ('f' followed by '-'). # # Note that 'x' is a command, and 'p', 'v', 'f' are options. # # Whew! # More elegant than, but equivalent to: # cd source/directory # tar cf - . | (cd ../dest/directory; tar xpvf -) # # Also having same effect: # cp -a /source/directory/* /dest/directory # Or: # cp -a /source/directory/* /source/directory/.[^.]* /dest/directory # If there are hidden files in /source/directory. bunzip2 -c linux-2.6.16.tar.bz2 | tar xvf - # --uncompress tar file-- | --then pass it to "tar"-- # If "tar" has not been patched to handle "bunzip2", #+ this needs to be done in two discrete steps, using a pipe. # The purpose of the exercise is to unarchive "bzipped" kernel source. Note that in this context the - is not itself a Bash operator, but rather an option recognized by certain UNIX utilities that write to stdout, such as tar, cat, etc. bash$ echo "whatever" | cat - whatever Where a filename is expected, - redirects output to stdout (sometimes seen with tar cf), or accepts input from stdin, rather than from a file. This is a method of using a file-oriented utility as a filter in a pipe. bash$ file Usage: file [-bciknvzL] [-f namefile] [-m magicfiles] file... By itself on the command line, file fails with an error message. Add a - for a more useful result. This causes the shell to await user input. bash$ file - abc standard input: ASCII text bash$ file - #!/bin/bash standard input: Bourne-Again shell script text executable Now the command accepts input from stdin and analyzes it. The - can be used to pipe stdout to other commands. This permits such stunts as prepending lines to a file. Using diff to compare a file with a section of another: grep Linux file1 | diff file2 - Finally, a real-world example using - with tar. Backup of all files changed in last day &ex58; Filenames beginning with - may cause problems when coupled with the - redirection operator. A script should check for this and add an appropriate prefix to such filenames, for example ./-FILENAME, $PWD/-FILENAME, or $PATHNAME/-FILENAME. If the value of a variable begins with a -, this may likewise create problems. var="-n" echo $var # Has the effect of "echo -n", and outputs nothing. - previous working directory A cd - command changes to the previous working directory. This uses the $OLDPWD environmental variable. Do not confuse the - used in this sense with the - redirection operator just discussed. The interpretation of the - depends on the context in which it appears. - Minus Minus sign in an arithmetic operation. = Equals Assignment operator a=28 echo $a # 28 In a different context, the = is a string comparison operator. + Plus Addition arithmetic operator. In a different context, the + is a Regular Expression operator. + Option Option flag for a command or filter. Certain commands and builtins use the to enable certain options and the to disable them. In parameter substitution, the prefixes an alternate value that a variable expands to. % <link linkend="moduloref">modulo</link> Modulo (remainder of a division) arithmetic operation. let "z = 5 % 3" echo $z # 2 In a different context, the % is a pattern matching operator. ~ home directory [tilde] This corresponds to the $HOME internal variable. ~bozo is bozo's home directory, and ls ~bozo lists the contents of it. ~/ is the current user's home directory, and ls ~/ lists the contents of it. bash$ echo ~bozo /home/bozo bash$ echo ~ /home/bozo bash$ echo ~/ /home/bozo/ bash$ echo ~: /home/bozo: bash$ echo ~nonexistent-user ~nonexistent-user ~+ current working directory This corresponds to the $PWD internal variable. ~- previous working directory This corresponds to the $OLDPWD internal variable. =~ <link linkend="regexmatchref">regular expression match</link> This operator was introduced with version 3 of Bash. ^ ^ special character ^ regular expression beginning of line beginning-of-line In a regular expression, a ^ addresses the beginning of a line of text. Control Characters change the behavior of the terminal or text display. A control character is a CONTROL + key combination (pressed simultaneously). A control character may also be written in octal or hexadecimal notation, following an escape. Control characters are not normally useful inside a script. Ctl-A Moves cursor to beginning of line of text (on the command-line). Ctl-B Backspace (nondestructive). Ctl-C Break. Terminate a foreground job. Ctl-D Log out from a shell (similar to exit). EOF (end-of-file). This also terminates input from stdin. When typing text on the console or in an xterm window, Ctl-D erases the character under the cursor. When there are no characters present, Ctl-D logs out of the session, as expected. In an xterm window, this has the effect of closing the window. Ctl-E Moves cursor to end of line of text (on the command-line). Ctl-F Moves cursor forward one character position (on the command-line). Ctl-G BEL. On some old-time teletype terminals, this would actually ring a bell. In an xterm it might beep. Ctl-H Rubout (destructive backspace). Erases characters the cursor backs over while backspacing. #!/bin/bash # Embedding Ctl-H in a string. a="^H^H" # Two Ctl-H's -- backspaces # ctl-V ctl-H, using vi/vim echo "abcdef" # abcdef echo echo -n "abcdef$a " # abcd f # Space at end ^ ^ Backspaces twice. echo echo -n "abcdef$a" # abcdef # No space at end ^ Doesn't backspace (why?). # Results may not be quite as expected. echo; echo # Constantin Hagemeier suggests trying: # a=$'\010\010' # a=$'\b\b' # a=$'\x08\x08' # But, this does not change the results. Ctl-I Horizontal tab. Ctl-J Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'. Ctl-K Vertical tab. When typing text on the console or in an xterm window, Ctl-K erases from the character under the cursor to end of line. Within a script, Ctl-K may behave differently, as in Lee Lee Maschmeyer's example, below. Ctl-L Formfeed (clear the terminal screen). In a terminal, this has the same effect as the clear command. When sent to a printer, a Ctl-L causes an advance to end of the paper sheet. Ctl-M Carriage return. #!/bin/bash # Thank you, Lee Maschmeyer, for this example. read -n 1 -s -p \ $'Control-M leaves cursor at beginning of this line. Press Enter. \x0d' # Of course, '0d' is the hex equivalent of Control-M. echo >&2 # The '-s' makes anything typed silent, #+ so it is necessary to go to new line explicitly. read -n 1 -s -p $'Control-J leaves cursor on next line. \x0a' # '0a' is the hex equivalent of Control-J, linefeed. echo >&2 ### read -n 1 -s -p $'And Control-K\x0bgoes straight down.' echo >&2 # Control-K is vertical tab. # A better example of the effect of a vertical tab is: var=$'\x0aThis is the bottom line\x0bThis is the top line\x0a' echo "$var" # This works the same way as the above example. However: echo "$var" | col # This causes the right end of the line to be higher than the left end. # It also explains why we started and ended with a line feed -- #+ to avoid a garbled screen. # As Lee Maschmeyer explains: # -------------------------- # In the [first vertical tab example] . . . the vertical tab #+ makes the printing go straight down without a carriage return. # This is true only on devices, such as the Linux console, #+ that can't go "backward." # The real purpose of VT is to go straight UP, not down. # It can be used to print superscripts on a printer. # The col utility can be used to emulate the proper behavior of VT. exit 0 Ctl-N Erases a line of text recalled from history buffer Bash stores a list of commands previously issued from the command-line in a buffer, or memory space, for recall with the builtin history commands. (on the command-line). Ctl-O Issues a newline (on the command-line). Ctl-P Recalls last command from history buffer (on the command-line). Ctl-Q Resume (XON). This resumes stdin in a terminal. Ctl-R Backwards search for text in history buffer (on the command-line). Ctl-S Suspend (XOFF). This freezes stdin in a terminal. (Use Ctl-Q to restore input.) Ctl-T Reverses the position of the character the cursor is on with the previous character (on the command-line). Ctl-U Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position. Ctl-V When inputting text, Ctl-V permits inserting control characters. For example, the following two are equivalent: echo -e '\x0a' echo <Ctl-V><Ctl-J> Ctl-V is primarily useful from within a text editor. Ctl-W When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace. In some settings, Ctl-W erases backwards to first non-alphanumeric character. Ctl-X In certain word processing programs, Cuts highlighted text and copies to clipboard. Ctl-Y Pastes back text previously erased (with Ctl-K or Ctl-U). Ctl-Z Pauses a foreground job. Substitute operation in certain word processing applications. EOF (end-of-file) character in the MSDOS filesystem. Whitespace functions as a separator, separating commands or variables. Whitespace consists of either spaces, tabs, blank lines, or any combination thereof. A linefeed (newline) is also a whitespace character. This explains why a blank line, consisting only of a linefeed, is considered whitespace. In some contexts, such as variable assignment, whitespace is not permitted, and results in a syntax error. Blank lines have no effect on the action of a script, and are therefore useful for visually separating functional sections. $IFS, the special variable separating fields of input to certain commands, defaults to whitespace. To preserve whitespace within a string or in a variable, use quoting. UNIX filters can target and operate on whitespace using the POSIX character class [:space:]. Introduction to Variables and Parameters Variables are how programming and scripting languages represent data. A variable is nothing more than a label, a name assigned to a location or set of locations holding an item of data, in computer memory. Variables appear in arithmetic operations and manipulation of quantities, and in string parsing. Variable Substitution The name of a variable is a placeholder for its value, the data it holds. Referencing (retrieving) its value is called variable substitution. $ $ variable $ variable substitution Let us carefully distinguish between the name of a variable and its value. If variable1 is the name of a variable, then $variable1 is a reference to its value, the data item it contains. Technically, the name of a variable is called an lvalue, meaning that it appears on the left side of an assignment statment, as in VARIABLE=23. A variable's value is an rvalue, meaning that it appears on the right side of an assignment statement, as in VAR2=$VARIABLE. A variable's name is, in fact, a reference, a pointer to the memory location(s) where the actual data associated with that variable is kept. bash$ variable1=23 bash$ echo variable1 variable1 bash$ echo $variable1 23 The only time a variable appears naked -- without the $ prefix -- is when declared or assigned, when unset, when exported, or in the special case of a variable representing a signal (see ). Assignment may be with an = (as in var1=27), in a read statement, and at the head of a loop (for var2 in 1 2 3). Enclosing a referenced value in double quotes (" ... ") does not interfere with variable substitution. This is called partial quoting, sometimes referred to as weak quoting. Using single quotes (' ... ') causes the variable name to be used literally, and no substitution will take place. This is full quoting, sometimes referred to as 'strong quoting.' See for a detailed discussion. Note that $variable is actually a simplified form of ${variable}. In contexts where the $variable syntax causes an error, the longer form may work (see , below). Variable assignment and substitution &ex9; An uninitialized variable has a null value -- no assigned value at all (not zero!). if [ -z "$unassigned" ] then echo "\$unassigned is NULL." fi # $unassigned is NULL. Using a variable before assigning a value to it may cause problems. It is nevertheless possible to perform arithmetic operations on an uninitialized variable. echo "$uninitialized" # (blank line) let "uninitialized += 5" # Add 5 to it. echo "$uninitialized" # 5 # Conclusion: # An uninitialized variable has no value, #+ however it acts as if it were 0 in an arithmetic operation. # This is undocumented (and probably non-portable) behavior, #+ and should not be used in a script. See also . Variable Assignment = = variable assignment the assignment operator (no space before and after) Do not confuse this with = and -eq, which test, rather than assign! Note that = can be either an assignment or a test operator, depending on context. Plain Variable Assignment &ex15; Variable Assignment, plain and fancy &ex16; Variable assignment using the $(...) mechanism (a newer method than backquotes). This is actually a form of command substitution. # From /etc/rc.d/rc.local R=$(cat /etc/redhat-release) arch=$(uname -m) Bash Variables Are Untyped Unlike many other programming languages, Bash does not segregate its variables by type. Essentially, Bash variables are character strings, but, depending on context, Bash permits arithmetic operations and comparisons on variables. The determining factor is whether the value of a variable contains only digits. Integer or string? &intorstring; Untyped variables are both a blessing and a curse. They permit more flexibility in scripting and make it easier to grind out lines of code (and give you enough rope to hang yourself!). However, they likewise permit subtle errors to creep in and encourage sloppy programming habits. To lighten the burden of keeping track of variable types in a script, Bash does permit declaring variables. Special Variable Types local variables variable local variables visible only within a code block or function (see also local variables in functions) environmental variables variable environmental variables that affect the behavior of the shell and user interface In a more general context, each process has an environment, that is, a group of variables that the process may reference. In this sense, the shell behaves like any other process. Every time a shell starts, it creates shell variables that correspond to its own environmental variables. Updating or adding new environmental variables causes the shell to update its environment, and all the shell's child processes (the commands it executes) inherit this environment. The space allotted to the environment is limited. Creating too many environmental variables or ones that use up excessive space may cause problems. bash$ eval "`seq 10000 | sed -e 's/.*/export var&=ZZZZZZZZZZZZZZ/'`" bash$ du bash: /usr/bin/du: Argument list too long Note: this error has been fixed, as of kernel version 2.6.23. (Thank you, Stéphane Chazelas for the clarification, and for providing the above example.) If a script sets environmental variables, they need to be exported, that is, reported to the environment local to the script. This is the function of the export command. A script can export variables only to child processes, that is, only to commands or processes which that particular script initiates. A script invoked from the command line cannot export variables back to the command line environment. Child processes cannot export variables back to the parent processes that spawned them. Definition: A child process is a subprocess launched by another process, its parent. --- positional parameters parameter positional arguments passed to the script from the command line Note that functions also take positional parameters. : $0, $1, $2, $3 . . . $0 is the name of the script itself, $1 is the first argument, $2 the second, $3 the third, and so forth. The process calling the script sets the $0 parameter. By convention, this parameter is the name of the script. See the manpage (manual page) for execv. From the command line, however, $0 is the name of the shell. bash$ echo $0 bash tcsh% echo $0 tcsh After $9, the arguments must be enclosed in brackets, for example, ${10}, ${11}, ${12}. The special variables $* and $@ denote all the positional parameters. Positional Parameters &ex17; Bracket notation for positional parameters leads to a fairly simple way of referencing the last argument passed to a script on the command line. This also requires indirect referencing. args=$# # Number of args passed. lastarg=${!args} # Note: This is an *indirect reference* to $args ... # Or: lastarg=${!#} (Thanks, Chris Monson.) # This is an *indirect reference* to the $# variable. # Note that lastarg=${!$#} doesn't work. Some scripts can perform different operations, depending on which name they are invoked with. For this to work, the script needs to check $0, the name it was invoked by. There must also exist symbolic links to all the alternate names of the script. See . If a script expects a command line parameter but is invoked without one, this may cause a null variable assignment, generally an undesirable result. One way to prevent this is to append an extra character to both sides of the assignment statement using the expected positional parameter. variable1_=$1_ # Rather than variable1=$1 # This will prevent an error, even if positional parameter is absent. critical_argument01=$variable1_ # The extra character can be stripped off later, like so. variable1=${variable1_/_/} # Side effects only if $variable1_ begins with an underscore. # This uses one of the parameter substitution templates discussed later. # (Leaving out the replacement pattern results in a deletion.) # A more straightforward way of dealing with this is #+ to simply test whether expected positional parameters have been passed. if [ -z $1 ] then exit $E_MISSING_POS_PARAM fi # However, as Fabian Kreutz points out, #+ the above method may have unexpected side-effects. # A better method is parameter substitution: # ${1:-$DefaultVal} # See the "Parameter Substition" section #+ in the "Variables Revisited" chapter. --- <firstterm>wh</firstterm>, <firstterm> whois</firstterm> domain name lookup &ex18; --- shift command shift The shift command reassigns the positional parameters, in effect shifting them to the left one notch. $1 <--- $2, $2 <--- $3, $3 <--- $4, etc. The old $1 disappears, but $0 (the script name) does not change. If you use a large number of positional parameters to a script, shift lets you access those past 10, although {bracket} notation also permits this. Using <firstterm>shift</firstterm> &ex19; The shift command can take a numerical parameter indicating how many positions to shift. #!/bin/bash # shift-past.sh shift 3 # Shift 3 positions. # n=3; shift $n # Has the same effect. echo "$1" exit 0 # ======================== # $ sh shift-past.sh 1 2 3 4 5 4 # However, as Eleni Fragkiadaki, points out, #+ attempting a 'shift' past the number of #+ positional parameters ($#) returns an exit status of 1, #+ and the positional parameters themselves do not change. # This means possibly getting stuck in an endless loop. . . . # For example: # until [ -z "$1" ] # do # echo -n "$1 " # shift 20 # If less than 20 pos params, # done #+ then loop never ends! # # When in doubt, add a sanity check. . . . # shift 20 || break # ^^^^^^^^ The shift command works in a similar fashion on parameters passed to a function. See . Quoting " special character " ' special character ' quote \ special character \ escape Quoting means just that, bracketing a string in quotes. This has the effect of protecting special characters in the string from reinterpretation or expansion by the shell or shell script. (A character is special if it has an interpretation other than its literal meaning. For example, the asterisk * represents a wild card character in globbing and Regular Expressions). bash$ ls -l [Vv]* -rw-rw-r-- 1 bozo bozo 324 Apr 2 15:05 VIEWDATA.BAT -rw-rw-r-- 1 bozo bozo 507 May 4 14:25 vartrace.sh -rw-rw-r-- 1 bozo bozo 539 Apr 14 17:11 viewdata.sh bash$ ls -l '[Vv]*' ls: [Vv]*: No such file or directory In everyday speech or writing, when we quote a phrase, we set it apart and give it special meaning. In a Bash script, when we quote a string, we set it apart and protect its literal meaning. Certain programs and utilities reinterpret or expand special characters in a quoted string. An important use of quoting is protecting a command-line parameter from the shell, but still letting the calling program expand it. bash$ grep '[Ff]irst' *.txt file1.txt:This is the first line of file1.txt. file2.txt:This is the First line of file2.txt. Note that the unquoted grep [Ff]irst *.txt works under the Bash shell. Unless there is a file named first in the current working directory. Yet another reason to quote. (Thank you, Harald Koenig, for pointing this out. Quoting can also suppress echo's appetite for newlines. bash$ echo $(ls -l) total 8 -rw-rw-r-- 1 bo bo 13 Aug 21 12:57 t.sh -rw-rw-r-- 1 bo bo 78 Aug 21 12:57 u.sh bash$ echo "$(ls -l)" total 8 -rw-rw-r-- 1 bo bo 13 Aug 21 12:57 t.sh -rw-rw-r-- 1 bo bo 78 Aug 21 12:57 u.sh Quoting Variables When referencing a variable, it is generally advisable to enclose its name in double quotes. This prevents reinterpretation of all special characters within the quoted string -- the variable name It also has side-effects on the value of the variable (see below) -- except $, ` (backquote), and \ (escape). Encapsulating ! within double quotes gives an error when used from the command line. This is interpreted as a history command. Within a script, though, this problem does not occur, since the Bash history mechanism is disabled then. Of more concern is the apparently inconsistent behavior of \ within double quotes. bash$ echo hello\! hello! bash$ echo "hello\!" hello\! bash$ echo -e x\ty xty bash$ echo -e "x\ty" x y What happens is that double quotes normally escape the \ escape character, so that it echoes literally. However, the option to echo changes that. It causes the \t to be interpreted as a tab. (Thank you, Wayne Pollock, for pointing this out, and Geoff Lee for explaining it.) Keeping $ as a special character within double quotes permits referencing a quoted variable ("$variable"), that is, replacing the variable with its value (see , above). Use double quotes to prevent word splitting. Word splitting, in this context, means dividing a character string into a number of separate and discrete arguments. An argument enclosed in double quotes presents itself as a single word, even if it contains whitespace separators. List="one two three" for a in $List # Splits the variable in parts at whitespace. do echo "$a" done # one # two # three echo "---" for a in "$List" # Preserves whitespace in a single variable. do # ^ ^ echo "$a" done # one two three A more elaborate example: variable1="a variable containing five words" COMMAND This is $variable1 # Executes COMMAND with 7 arguments: # "This" "is" "a" "variable" "containing" "five" "words" COMMAND "This is $variable1" # Executes COMMAND with 1 argument: # "This is a variable containing five words" variable2="" # Empty. COMMAND $variable2 $variable2 $variable2 # Executes COMMAND with no arguments. COMMAND "$variable2" "$variable2" "$variable2" # Executes COMMAND with 3 empty arguments. COMMAND "$variable2 $variable2 $variable2" # Executes COMMAND with 1 argument (2 spaces). # Thanks, Stéphane Chazelas. Enclosing the arguments to an echo statement in double quotes is necessary only when word splitting or preservation of whitespace is an issue. Echoing Weird Variables &weirdvars; Single quotes (' ') operate similarly to double quotes, but do not permit referencing variables, since the special meaning of $ is turned off. Within single quotes, every special character except ' gets interpreted literally. Consider single quotes (full quoting) to be a stricter method of quoting than double quotes (partial quoting). Since even the escape character (\) gets a literal interpretation within single quotes, trying to enclose a single quote within single quotes will not yield the expected result. echo "Why can't I write 's between single quotes" echo # The roundabout method. echo 'Why can'\''t I write '"'"'s between single quotes' # |-------| |----------| |-----------------------| # Three single-quoted strings, with escaped and quoted single quotes between. # This example courtesy of Stéphane Chazelas. Escaping Escaping is a method of quoting single characters. The escape (\) preceding a character tells the shell to interpret that character literally. With certain commands and utilities, such as echo and sed, escaping a character may have the opposite effect - it can toggle on a special meaning for that character. <anchor id="spm">Special meanings of certain escaped characters used with echo and sed \n \n escaped character \n newline means newline \r \r escaped character \r carriage return means return \t \t escaped character \t tabulation means tab \v \v escaped character \v vertical tabulation means vertical tab \b \b escaped character \b backspace means backspace \a \a escaped character \a alert beep flash means alert (beep or flash) \0xx \0xx escaped character \0nn octal ASCII translates to the octal ASCII equivalent of 0nn, where nn is a string of digits Escaped Characters &escaped; See for another example of the $' ... ' string-expansion construct. \" \" escaped character \" quote gives the quote its literal meaning echo "Hello" # Hello echo "\"Hello\" ... he said." # "Hello" ... he said. \$ \$ escaped character \$ dollar gives the dollar sign its literal meaning (variable name following \$ will not be referenced) echo "\$variable01" # $variable01 echo "The book cost \$7.98." # The book cost $7.98. \\ \\ escaped character \\ double backslash gives the backslash its literal meaning echo "\\" # Results in \ # Whereas . . . echo "\" # Invokes secondary prompt from the command line. # In a script, gives an error message. # However . . . echo '\' # Results in \ The behavior of \ depends on whether it is escaped, strong-quoted, weak-quoted, or appearing within command substitution or a here document. # Simple escaping and quoting echo \z # z echo \\z # \z echo '\z' # \z echo '\\z' # \\z echo "\z" # \z echo "\\z" # \z # Command substitution echo `echo \z` # z echo `echo \\z` # z echo `echo \\\z` # \z echo `echo \\\\z` # \z echo `echo \\\\\\z` # \z echo `echo \\\\\\\z` # \\z echo `echo "\z"` # \z echo `echo "\\z"` # \z # Here document cat <<EOF \z EOF # \z cat <<EOF \\z EOF # \z # These examples supplied by Stéphane Chazelas. Elements of a string assigned to a variable may be escaped, but the escape character alone may not be assigned to a variable. variable=\ echo "$variable" # Will not work - gives an error message: # test.sh: : command not found # A "naked" escape cannot safely be assigned to a variable. # # What actually happens here is that the "\" escapes the newline and #+ the effect is variable=echo "$variable" #+ invalid variable assignment variable=\ 23skidoo echo "$variable" # 23skidoo # This works, since the second line #+ is a valid variable assignment. variable=\ # \^ escape followed by space echo "$variable" # space variable=\\ echo "$variable" # \ variable=\\\ echo "$variable" # Will not work - gives an error message: # test.sh: \: command not found # # First escape escapes second one, but the third one is left "naked", #+ with same result as first instance, above. variable=\\\\ echo "$variable" # \\ # Second and fourth escapes escaped. # This is o.k. Escaping a space can prevent word splitting in a command's argument list. file_list="/bin/cat /bin/gzip /bin/more /usr/bin/less /usr/bin/emacs-20.7" # List of files as argument(s) to a command. # Add two files to the list, and list all. ls -l /usr/X11R6/bin/xsetroot /sbin/dump $file_list echo "-------------------------------------------------------------------------" # What happens if we escape a couple of spaces? ls -l /usr/X11R6/bin/xsetroot\ /sbin/dump\ $file_list # Error: the first three files concatenated into a single argument to 'ls -l' # because the two escaped spaces prevent argument (word) splitting. The escape also provides a means of writing a multi-line command. Normally, each separate line constitutes a different command, but an escape at the end of a line escapes the newline character, and the command sequence continues on to the next line. (cd /source/directory && tar cf - . ) | \ (cd /dest/directory && tar xpvf -) # Repeating Alan Cox's directory tree copy command, # but split into two lines for increased legibility. # As an alternative: tar cf - -C /source/directory . | tar xpvf - -C /dest/directory # See note below. # (Thanks, Stéphane Chazelas.) If a script line ends with a |, a pipe character, then a \, an escape, is not strictly necessary. It is, however, good programming practice to always escape the end of a line of code that continues to the following line. echo "foo bar" #foo #bar echo echo 'foo bar' # No difference yet. #foo #bar echo echo foo\ bar # Newline escaped. #foobar echo echo "foo\ bar" # Same here, as \ still interpreted as escape within weak quotes. #foobar echo echo 'foo\ bar' # Escape character \ taken literally because of strong quoting. #foo\ #bar # Examples suggested by Stéphane Chazelas. Exit and Exit Status ... there are dark corners in the Bourne shell, and people use all of them. --Chet Ramey The exit command exit exit command terminates a script, just as in a C program. It can also return a value, which is available to the script's parent process. Every command returns an exit status exit status (sometimes referred to as a return status return status or exit code). A successful command returns a 0, while an unsuccessful one returns a non-zero value that usually may be interpreted as an error code. Well-behaved UNIX commands, programs, and utilities return a 0 exit code upon successful completion, though there are some exceptions. Likewise, functions within a script and the script itself return an exit status. The last command executed in the function or script determines the exit status. Within a script, an exit nnn command may be used to deliver an nnn exit status to the shell (nnn must be a decimal number in the 0 - 255 range). When a script ends with an exit that has no parameter, the exit status of the script is the exit status of the last command executed in the script (previous to the exit). #!/bin/bash COMMAND_1 . . . # Will exit with status of last command. COMMAND_LAST exit The equivalent of a bare exit is exit $? or even just omitting the exit. #!/bin/bash COMMAND_1 . . . # Will exit with status of last command. COMMAND_LAST exit $? #!/bin/bash COMMAND1 . . . # Will exit with status of last command. COMMAND_LAST $? variable $? $? reads the exit status of the last command executed. After a function returns, $? gives the exit status of the last command executed in the function. This is Bash's way of giving functions a return value. After a script terminates, a $? from the command line gives the exit status of the script, that is, the last command executed in the script, which is, by convention, 0 on success or an integer in the range 1 - 255 on error. exit / exit status &ex5; $? is especially useful for testing the result of a command in a script (see and ). The !, the logical not qualifier, reverses the outcome of a test or command, and this affects its exit status. Negating a condition using <token>!</token> true # The "true" builtin. echo "exit status of \"true\" = $?" # 0 ! true echo "exit status of \"! true\" = $?" # 1 # Note that the "!" needs a space between it and the command. # !true leads to a "command not found" error # # The '!' operator prefixing a command invokes the Bash history mechanism. true !true # No error this time, but no negation either. # It just repeats the previous command (true). # Thanks, Stéphane Chazelas and Kristopher Newsome. Certain exit status codes have reserved meanings and should not be user-specified in a script. Tests if test if then else else if elif Every reasonably complete programming language can test for a condition, then act according to the result of the test. Bash has the test command, various bracket and parenthesis operators, and the if/then construct. Test Constructs An if/then construct tests whether the exit status of a list of commands is 0 (since 0 means success by UNIX convention), and if so, executes one or more commands. There exists a dedicated command called [ (left bracket special character). It is a synonym for test, and a builtin for efficiency reasons. This command considers its arguments as comparison expressions or file tests and returns an exit status corresponding to the result of the comparison (0 for true, 1 for false). With version 2.02, Bash introduced the [[ ... ]] extended test command, which performs comparisons in a manner more familiar to programmers from other languages. Note that [[ is a keyword, not a command. Bash sees [[ $a -lt $b ]] as a single element, which returns an exit status. The (( ... )) and let ... constructs also return an exit status, according to whether the arithmetic expressions they evaluate expand to a non-zero value. These arithmetic-expansion constructs may therefore be used to perform arithmetic comparisons. (( 0 && 1 )) # Logical AND echo $? # 1 *** # And so ... let "num = (( 0 && 1 ))" echo $num # 0 # But ... let "num = (( 0 && 1 ))" echo $? # 1 *** (( 200 || 11 )) # Logical OR echo $? # 0 *** # ... let "num = (( 200 || 11 ))" echo $num # 1 let "num = (( 200 || 11 ))" echo $? # 0 *** (( 200 | 11 )) # Bitwise OR echo $? # 0 *** # ... let "num = (( 200 | 11 ))" echo $num # 203 let "num = (( 200 | 11 ))" echo $? # 0 *** # The "let" construct returns the same exit status #+ as the double-parentheses arithmetic expansion. An if can test any command, not just conditions enclosed within brackets. if cmp a b &> /dev/null # Suppress output. then echo "Files a and b are identical." else echo "Files a and b differ." fi # The very useful "if-grep" construct: # ----------------------------------- if grep -q Bash file then echo "File contains at least one occurrence of Bash." fi word=Linux letter_sequence=inu if echo "$word" | grep -q "$letter_sequence" # The "-q" option to grep suppresses output. then echo "$letter_sequence found in $word" else echo "$letter_sequence not found in $word" fi if COMMAND_WHOSE_EXIT_STATUS_IS_0_UNLESS_ERROR_OCCURRED then echo "Command succeeded." else echo "Command failed." fi These last two examples courtesy of Stéphane Chazelas. What is truth? &ex10; Exercise Explain the behavior of , above. if [ condition-true ] then command 1 command 2 ... else # Or else ... # Adds default code block executing if original condition tests false. command 3 command 4 ... fi When if and then are on same line in a condition test, a semicolon must terminate the if statement. Both if and then are keywords. Keywords (or commands) begin statements, and before a new statement on the same line begins, the old one must terminate. if [ -x "$filename" ]; then <anchor id="elifref1">Else if and elif elif elif is a contraction for else if. The effect is to nest an inner if/then construct within an outer one. if [ condition1 ] then command1 command2 command3 elif [ condition2 ] # Same as else if then command4 command5 else default-command fi test test test [ special character [ ] special character ] The if test condition-true construct is the exact equivalent of if [ condition-true ]. As it happens, the left bracket, [ , is a token which invokes the test command. The closing right bracket, ] , in an if/test should not therefore be strictly necessary, however newer versions of Bash require it. The test command is a Bash builtin which tests file types and compares strings. Therefore, in a Bash script, test does not call the external /usr/bin/test binary, which is part of the sh-utils package. Likewise, [ does not call /usr/bin/[, which is linked to /usr/bin/test. bash$ type test test is a shell builtin bash$ type '[' [ is a shell builtin bash$ type '[[' [[ is a shell keyword bash$ type ']]' ]] is a shell keyword bash$ type ']' bash: type: ]: not found If, for some reason, you wish to use /usr/bin/test in a Bash script, then specify it by full pathname. Equivalence of <firstterm>test</firstterm>, <filename>/usr/bin/test</filename>, <token>[ ]</token>, and <filename>/usr/bin/[</filename> &ex11; test test test [[ special character [[ ]] special character ]] The [[ ]] construct is the more versatile Bash version of [ ]. This is the extended test command, adopted from ksh88. * * * No filename expansion or word splitting takes place between [[ and ]], but there is parameter expansion and command substitution. file=/etc/passwd if [[ -e $file ]] then echo "Password file exists." fi Using the [[ ... ]] test construct, rather than [ ... ] can prevent many logic errors in scripts. For example, the &&, ||, <, and > operators work within a [[ ]] test, despite giving an error within a [ ] construct. Arithmetic evaluation of octal / hexadecimal constants takes place automatically within a [[ ... ]] construct. # [[ Octal and hexadecimal evaluation ]] # Thank you, Moritz Gronbach, for pointing this out. decimal=15 octal=017 # = 15 (decimal) hex=0x0f # = 15 (decimal) if [ "$decimal" -eq "$octal" ] then echo "$decimal equals $octal" else echo "$decimal is not equal to $octal" # 15 is not equal to 017 fi # Doesn't evaluate within [ single brackets ]! if [[ "$decimal" -eq "$octal" ]] then echo "$decimal equals $octal" # 15 equals 017 else echo "$decimal is not equal to $octal" fi # Evaluates within [[ double brackets ]]! if [[ "$decimal" -eq "$hex" ]] then echo "$decimal equals $hex" # 15 equals 0x0f else echo "$decimal is not equal to $hex" fi # [[ $hexadecimal ]] also evaluates! Following an if, neither the test command nor the test brackets ( [ ] or [[ ]] ) are strictly necessary. dir=/home/bozo if cd "$dir" 2>/dev/null; then # "2>/dev/null" hides error message. echo "Now in $dir." else echo "Can't change to $dir." fi The "if COMMAND" construct returns the exit status of COMMAND. Similarly, a condition within test brackets may stand alone without an if, when used in combination with a list construct. var1=20 var2=22 [ "$var1" -ne "$var2" ] && echo "$var1 is not equal to $var2" home=/home/bozo [ -d "$home" ] || echo "$home directory does not exist." test test test (( special character )) (( special character )) The (( )) construct expands and evaluates an arithmetic expression. If the expression evaluates as zero, it returns an exit status of 1, or false. A non-zero expression returns an exit status of 0, or true. This is in marked contrast to using the test and [ ] constructs previously discussed. Arithmetic Tests using <token>(( ))</token> &arithtests; File test operators <anchor id="rtif">Returns true if... -e file exists -a file exists This is identical in effect to -e. It has been deprecated, Per the 1913 edition of Webster's Dictionary: Deprecate ... To pray against, as an evil; to seek to avert by prayer; to desire the removal of; to seek deliverance from; to express deep regret for; to disapprove of strongly. and its use is discouraged. -f file is a regular file (not a directory or device file) -s file is not zero size -d file is a directory -b file is a block device device="/dev/sda2" # / (root directory) if [ -b "$device" ] then echo "$device is a block device." fi # /dev/sda2 is a block device. -c file is a character device -p file is a pipe -h file is a symbolic link -L file is a symbolic link -S file is a socket -t file (descriptor) is associated with a terminal device This test option may be used to check whether the stdin ([ -t 0 ]) or stdout ([ -t 1 ]) in a given script is a terminal. -r file has read permission (for the user running the test) -w file has write permission (for the user running the test) -x file has execute permission (for the user running the test) -g set-group-id (sgid) flag set on file or directory If a directory has the sgid flag set, then a file created within that directory belongs to the group that owns the directory, not necessarily to the group of the user who created the file. This may be useful for a directory shared by a workgroup. -u set-user-id (suid) flag set on file A binary owned by root with set-user-id flag set runs with root privileges, even when an ordinary user invokes it. Be aware that suid binaries may open security holes. The suid flag has no effect on shell scripts. This is useful for executables (such as pppd and cdrecord) that need to access system hardware. Lacking the suid flag, these binaries could not be invoked by a non-root user. -rwsr-xr-t 1 root 178236 Oct 2 2000 /usr/sbin/pppd A file with the suid flag set shows an s in its permissions. -k sticky bit set Commonly known as the sticky bit, the save-text-mode flag is a special type of file permission. If a file has this flag set, that file will be kept in cache memory, for quicker access. On modern UNIX systems, the sticky bit is no longer used for files, only on directories. If set on a directory, it restricts write permission. Setting the sticky bit adds a t to the permissions on the file or directory listing. drwxrwxrwt 7 root 1024 May 19 21:26 tmp/ If a user does not own a directory that has the sticky bit set, but has write permission in that directory, she can only delete those files that she owns in it. This keeps users from inadvertently overwriting or deleting each other's files in a publicly accessible directory, such as /tmp. (The owner of the directory or root can, of course, delete or rename files there.) -O you are owner of file -G group-id of file same as yours -N file modified since it was last read f1 -nt f2 file f1 is newer than f2 f1 -ot f2 file f1 is older than f2 f1 -ef f2 files f1 and f2 are hard links to the same file ! not -- reverses the sense of the tests above (returns true if condition absent). Testing for broken links &brokenlink; , , , , and also illustrate uses of the file test operators. Other Comparison Operators A binary comparison operator compares two variables or quantities. Note that integer and string comparison use a different set of operators. <anchor id="icomparison1">integer comparison -eq is equal to if [ "$a" -eq "$b" ] -ne is not equal to if [ "$a" -ne "$b" ] -gt is greater than if [ "$a" -gt "$b" ] -ge is greater than or equal to if [ "$a" -ge "$b" ] -lt is less than if [ "$a" -lt "$b" ] -le is less than or equal to if [ "$a" -le "$b" ] < is less than (within double parentheses) (("$a" < "$b")) <= is less than or equal to (within double parentheses) (("$a" <= "$b")) > is greater than (within double parentheses) (("$a" > "$b")) >= is greater than or equal to (within double parentheses) (("$a" >= "$b")) <anchor id="scomparison1">string comparison = is equal to if [ "$a" = "$b" ] == is equal to if [ "$a" == "$b" ] This is a synonym for =. The == comparison operator behaves differently within a double-brackets test than within single brackets. [[ $a == z* ]] # True if $a starts with an "z" (regex pattern matching). [[ $a == "z*" ]] # True if $a is equal to z* (literal matching). [ $a == z* ] # File globbing and word splitting take place. [ "$a" == "z*" ] # True if $a is equal to z* (literal matching). # Thanks, Stéphane Chazelas != is not equal to if [ "$a" != "$b" ] This operator uses pattern matching within a [[ ... ]] construct. < is less than, in ASCII alphabetical order if [[ "$a" < "$b" ]] if [ "$a" \< "$b" ] Note that the < needs to be escaped within a [ ] construct. > is greater than, in ASCII alphabetical order if [[ "$a" > "$b" ]] if [ "$a" \> "$b" ] Note that the > needs to be escaped within a [ ] construct. See for an application of this comparison operator. -z string is null, that is, has zero length String='' # Zero-length ("null") string variable. if [ -z "$String" ] then echo "\$String is null." else echo "\$String is NOT null." fi # $String is null. -n string is not null. The -n test requires that the string be quoted within the test brackets. Using an unquoted string with ! -z, or even just the unquoted string alone within test brackets (see ) normally works, however, this is an unsafe practice. Always quote a tested string. As S.C. points out, in a compound test, even quoting the string variable might not suffice. [ -n "$string" -o "$a" = "$b" ] may cause an error with some versions of Bash if $string is empty. The safe way is to append an extra character to possibly empty variables, [ "x$string" != x -o "x$a" = "x$b" ] (the x's cancel out). Arithmetic and string comparisons &ex13; Testing whether a string is <firstterm>null</firstterm> &strtest; <firstterm>zmore</firstterm> &ex14; <anchor id="ccomparison1">compound comparison -a logical and exp1 -a exp2 returns true if both exp1 and exp2 are true. -o logical or exp1 -o exp2 returns true if either exp1 or exp2 is true. These are similar to the Bash comparison operators && and ||, used within double brackets. [[ condition1 && condition2 ]] The -o and -a operators work with the test command or occur within single test brackets. if [ "$exp1" -a "$exp2" ] Refer to , , and to see compound comparison operators in action. Nested <replaceable>if/then</replaceable> Condition Tests Condition tests using the if/then construct may be nested. The net result is equivalent to using the && compound comparison operator. if [ condition1 ] then if [ condition2 ] then do-something # But only if both "condition1" AND "condition2" valid. fi fi demonstrates a nested if/then condition test. Testing Your Knowledge of Tests The systemwide xinitrc file can be used to launch the X server. This file contains quite a number of if/then tests. The following is excerpted from an ancient version of xinitrc (Red Hat 7.1, or thereabouts). if [ -f $HOME/.Xclients ]; then exec $HOME/.Xclients elif [ -f /etc/X11/xinit/Xclients ]; then exec /etc/X11/xinit/Xclients else # failsafe settings. Although we should never get here # (we provide fallbacks in Xclients as well) it can't hurt. xclock -geometry 100x100-5+5 & xterm -geometry 80x50-50+150 & if [ -f /usr/bin/netscape -a -f /usr/share/doc/HTML/index.html ]; then netscape /usr/share/doc/HTML/index.html & fi fi Explain the test constructs in the above snippet, then examine an updated version of the file, /etc/X11/xinit/xinitrc, and analyze the if/then test constructs there. You may need to refer ahead to the discussions of grep, sed, and regular expressions. Operations and Related Topics Operators <anchor id="asnop1">assignment variable assignment Initializing or changing the value of a variable = operation = = All-purpose assignment operator, which works for both arithmetic and string assignments. var=27 category=minerals # No spaces allowed after the "=". Do not confuse the = assignment operator with the = test operator. # = as a test operator if [ "$string1" = "$string2" ] then command fi # if [ "X$string1" = "X$string2" ] is safer, #+ to prevent an error message should one of the variables be empty. # (The prepended "X" characters cancel out.) expr command expr let command let <anchor id="arops1">arithmetic operators + + operation + addition plus plus - - operation - subtraction minus minus * * operation * multiplication multiplication / / operation / division division ** ** operation ** exponentiation exponentiation # Bash, version 2.02, introduced the "**" exponentiation operator. let "z=5**3" # 5 * 5 * 5 echo "z = $z" # z = 125 % % operation % modulo modulo, or mod (returns the remainder of an integer division operation) bash$ expr 5 % 3 2 5/3 = 1, with remainder 2 This operator finds use in, among other things, generating numbers within a specific range (see and ) and formatting program output (see and ). It can even be used to generate prime numbers, (see ). Modulo turns up surprisingly often in various numerical recipes. Greatest common divisor &gcd; += += operation += plus-equal plus-equal (increment variable by a constant) let "var += 5" results in var being incremented by 5. -= -= operation -= minus-equal minus-equal (decrement variable by a constant) *= *= operation *= times-equal times-equal (multiply variable by a constant) let "var *= 4" results in var being multiplied by 4. /= /= operation /= slash-equal slash-equal (divide variable by a constant) %= %= operation %= mod-equal mod-equal (remainder of dividing variable by a constant) Arithmetic operators often occur in an expr or let expression. Using Arithmetic Operations &arithops; Integer variables in Bash are actually signed long (32-bit) integers, in the range of -2147483648 to 2147483647. An operation that takes a variable outside these limits will give an erroneous result. a=2147483646 echo "a = $a" # a = 2147483646 let "a+=1" # Increment "a". echo "a = $a" # a = 2147483647 let "a+=1" # increment "a" again, past the limit. echo "a = $a" # a = -2147483648 # ERROR (out of range) As of version 2.05b, Bash supports 64-bit integers. Bash does not understand floating point arithmetic. It treats numbers containing a decimal point as strings. a=1.5 let "b = $a + 1.3" # Error. # t2.sh: let: b = 1.5 + 1.3: syntax error in expression # (error token is ".5 + 1.3") echo "b = $b" # b=1 Use bc in scripts that that need floating point calculations or math library functions. bitwise operators The bitwise operators seldom make an appearance in shell scripts. Their chief use seems to be manipulating and testing values read from ports or sockets. Bit flipping is more relevant to compiled languages, such as C and C++, which provide direct access to system hardware. <anchor id="bitwsops1">bitwise operators << << operation << left shift bitwise left shift (multiplies by 2 for each shift position) <<= <<= operation <<= left-shift-equal left-shift-equal let "var <<= 2" results in var left-shifted 2 bits (multiplied by 4) >> >> operation >> right shift bitwise right shift (divides by 2 for each shift position) >>= >>= operation >>= right-shift-equal right-shift-equal (inverse of <<=) & & operation & AND bitwise bitwise AND &= &= operation &= and-equal bitwise AND-equal | | operation | OR bitwise bitwise OR |= |= operation |= OR-equal bitwise OR-equal ~ ~ operation ~ negate bitwise NOT ^ ^ operation ^ XOR bitwise XOR ^= ^= operation ^= XOR-equal bitwise XOR-equal <anchor id="logops1">logical (boolean) operators ! ! operator ! NOT NOT if [ ! -f $FILENAME ] then ... && && operator && AND logical AND if [ $condition1 ] && [ $condition2 ] # Same as: if [ $condition1 -a $condition2 ] # Returns true if both condition1 and condition2 hold true... if [[ $condition1 && $condition2 ]] # Also works. # Note that && operator not permitted within [ ... ] construct. && may also, depending on context, be used in an and list to concatenate commands. || || operator || OR logical OR if [ $condition1 ] || [ $condition2 ] # Same as: if [ $condition1 -o $condition2 ] # Returns true if either condition1 or condition2 holds true... if [[ $condition1 || $condition2 ]] # Also works. # Note that || operator not permitted within [ ... ] construct. Bash tests the exit status of each statement linked with a logical operator. Compound Condition Tests Using && and || &andor; The && and || operators also find use in an arithmetic context. bash$ echo $(( 1 && 2 )) $((3 && 0)) $((4 || 0)) $((0 || 0)) 1 0 1 0 <anchor id="miscop1">miscellaneous operators , , operation , linking Comma operator The comma operator chains together two or more arithmetic operations. All the operations are evaluated (with possible side effects Side effects are, of course, unintended -- and usually undesirable -- consequences. ), but only the last operation is returned. let "t1 = ((5 + 3, 7 - 1, 15 - 4))" echo "t1 = $t1" # t1 = 11 let "t2 = ((a = 9, 15 / 3))" # Set "a" and calculate "t2". echo "t2 = $t2 a = $a" # t2 = 5 a = 9 The comma operator finds use mainly in for loops. See . Numerical Constants A shell script interprets a number as decimal (base 10), unless that number has a special prefix or notation. A number preceded by a 0 is octal (base 8). A number preceded by 0x is hexadecimal (base 16). A number with an embedded # evaluates as BASE#NUMBER (with range and notational restrictions). Representation of numerical constants &numbers; Beyond the Basics Variables Revisited Used properly, variables can add power and flexibility to scripts. This requires learning their subtleties and nuances. Internal Variables Builtin variables variables affecting bash script behavior $BASH $BASH variable $BASH path to bash the path to the Bash binary itself bash$ echo $BASH /bin/bash $BASH_ENV $BASH_ENV variable $BASH_ENV an environmental variable pointing to a Bash startup file to be read when a script is invoked $BASH_SUBSHELL $BASH_SUBSHELL variable subshell a variable indicating the subshell level. This is a new addition to Bash, version 3. See for usage. $BASH_VERSINFO[n] $BASH_VERSINFO variable version information a 6-element array containing version information about the installed release of Bash. This is similar to $BASH_VERSION, below, but a bit more detailed. # Bash version info: for n in 0 1 2 3 4 5 do echo "BASH_VERSINFO[$n] = ${BASH_VERSINFO[$n]}" done # BASH_VERSINFO[0] = 3 # Major version no. # BASH_VERSINFO[1] = 00 # Minor version no. # BASH_VERSINFO[2] = 14 # Patch level. # BASH_VERSINFO[3] = 1 # Build version. # BASH_VERSINFO[4] = release # Release status. # BASH_VERSINFO[5] = i386-redhat-linux-gnu # Architecture # (same as $MACHTYPE). $BASH_VERSION $BASH_VERSION variable $BASH_VERSION the version of Bash installed on the system bash$ echo $BASH_VERSION 3.2.25(1)-release tcsh% echo $BASH_VERSION BASH_VERSION: Undefined variable. Checking $BASH_VERSION is a good method of determining which shell is running. $SHELL does not necessarily give the correct answer. $DIRSTACK $DIRSTACK variable $DIRSTACK directory stack directory stack the top value in the directory stack (affected by pushd and popd) This builtin variable corresponds to the dirs command, however dirs shows the entire contents of the directory stack. $EDITOR $EDITOR variable $EDITOR editor the default editor invoked by a script, usually vi or emacs. $EUID $EUID variable $EUID effective user ID effective user ID number Identification number of whatever identity the current user has assumed, perhaps by means of su. The $EUID is not necessarily the same as the $UID. $FUNCNAME $FUNCNAME variable function name name of the current function xyz23 () { echo "$FUNCNAME now executing." # xyz23 now executing. } xyz23 echo "FUNCNAME = $FUNCNAME" # FUNCNAME = # Null value outside a function. See also . $GLOBIGNORE $GLOBIGNORE variable globbing ignore A list of filename patterns to be excluded from matching in globbing. $GROUPS $GROUPS variable $GROUPS groups groups current user belongs to This is a listing (array) of the group id numbers for current user, as recorded in /etc/passwd and /etc/group. root# echo $GROUPS 0 root# echo ${GROUPS[1]} 1 root# echo ${GROUPS[5]} 6 $HOME $HOME variable $HOME home directory directory home home directory of the user, usually /home/username (see ) $HOSTNAME $HOSTNAME variable $HOSTNAME system name variable name The hostname command assigns the system host name at bootup in an init script. However, the gethostname() function sets the Bash internal variable $HOSTNAME. See also . $HOSTTYPE $HOSTTYPE variable $HOSTTYPE host type host type Like $MACHTYPE, identifies the system hardware. bash$ echo $HOSTTYPE i686 $IFS $IFS variable $IFS internal field separator internal field separator This variable determines how Bash recognizes fields, or word boundaries, when it interprets character strings. $IFS defaults to whitespace (space, tab, and newline), but may be changed, for example, to parse a comma-separated data file. Note that $* uses the first character held in $IFS. See . bash$ echo "$IFS" (With $IFS set to default, a blank line displays.) bash$ echo "$IFS" | cat -vte ^I$ $ (Show whitespace -- here space, ^I [horizontal tab], and newline -- and display "$" at end-of-line.) bash$ bash -c 'set w x y z; IFS=":-;"; echo "$*"' w:x:y:z (Read commands from string and assign any arguments to pos params.) $IFS does not handle whitespace the same as it does other characters. $IFS and whitespace &ifsh; (Many thanks, Stéphane Chazelas, for clarification and examples.) See also , , and for instructive examples of using $IFS. $IGNOREEOF $IGNOREEOF variable $IGNOREEOF Ignore EOF ignore EOF: how many end-of-files (control-D) the shell will ignore before logging out. $LC_COLLATE $LC_COLLATE variable $LC_COLLATE lowercase collate Often set in the .bashrc or /etc/profile files, this variable controls collation order in filename expansion and pattern matching. If mishandled, LC_COLLATE can cause unexpected results in filename globbing. As of version 2.05 of Bash, filename globbing no longer distinguishes between lowercase and uppercase letters in a character range between brackets. For example, ls [A-M]* would match both File1.txt and file1.txt. To revert to the customary behavior of bracket matching, set LC_COLLATE to by an export LC_COLLATE=C in /etc/profile and/or ~/.bashrc. $LC_CTYPE $LC_CTYPE variable $LC_CTYPE lowercase character type This internal variable controls character interpretation in globbing and pattern matching. $LINENO $LINENO variable $LINENO line number This variable is the line number of the shell script in which this variable appears. It has significance only within the script in which it appears, and is chiefly useful for debugging purposes. # *** BEGIN DEBUG BLOCK *** last_cmd_arg=$_ # Save it. echo "At line number $LINENO, variable \"v1\" = $v1" echo "Last command argument processed = $last_cmd_arg" # *** END DEBUG BLOCK *** $MACHTYPE $MACHTYPE variable $MACHTYPE machine type machine type Identifies the system hardware. bash$ echo $MACHTYPE i686 $OLDPWD $OLDPWD variable $OLDPWD previous working directory directory working old working directory (OLD-print-working-directory, previous directory you were in) $OSTYPE $OSTYPE variable $OSTYPE os type operating system type bash$ echo $OSTYPE linux $PATH $PATH variable $PATH path to binaries path to binaries, usually /usr/bin/, /usr/X11R6/bin/, /usr/local/bin, etc. When given a command, the shell automatically does a hash table search on the directories listed in the path for the executable. The path is stored in the environmental variable, $PATH, a list of directories, separated by colons. Normally, the system stores the $PATH definition in /etc/profile and/or ~/.bashrc (see ). bash$ echo $PATH /bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/sbin:/usr/sbin PATH=${PATH}:/opt/bin appends the /opt/bin directory to the current path. In a script, it may be expedient to temporarily add a directory to the path in this way. When the script exits, this restores the original $PATH (a child process, such as a script, may not change the environment of the parent process, the shell). The current working directory, ./, is usually omitted from the $PATH as a security measure. $PIPESTATUS $PIPESTATUS variable pipe Array variable holding exit status(es) of last executed foreground pipe. bash$ echo $PIPESTATUS 0 bash$ ls -al | bogus_command bash: bogus_command: command not found bash$ echo ${PIPESTATUS[1]} 127 bash$ ls -al | bogus_command bash: bogus_command: command not found bash$ echo $? 127 The members of the $PIPESTATUS array hold the exit status of each respective command executed in a pipe. $PIPESTATUS[0] holds the exit status of the first command in the pipe, $PIPESTATUS[1] the exit status of the second command, and so on. The $PIPESTATUS variable may contain an erroneous 0 value in a login shell (in releases prior to 3.0 of Bash). tcsh% bash bash$ who | grep nobody | sort bash$ echo ${PIPESTATUS[*]} 0 The above lines contained in a script would produce the expected 0 1 0 output. Thank you, Wayne Pollock for pointing this out and supplying the above example. The $PIPESTATUS variable gives unexpected results in some contexts. bash$ echo $BASH_VERSION 3.00.14(1)-release bash$ $ ls | bogus_command | wc bash: bogus_command: command not found 0 0 0 bash$ echo ${PIPESTATUS[@]} 141 127 0 Chet Ramey attributes the above output to the behavior of ls. If ls writes to a pipe whose output is not read, then SIGPIPE kills it, and its exit status is 141. Otherwise its exit status is 0, as expected. This likewise is the case for tr. $PIPESTATUS is a volatile variable. It needs to be captured immediately after the pipe in question, before any other command intervenes. bash$ $ ls | bogus_command | wc bash: bogus_command: command not found 0 0 0 bash$ echo ${PIPESTATUS[@]} 0 127 0 bash$ echo ${PIPESTATUS[@]} 0 The pipefail option may be useful in cases where $PIPESTATUS does not give the desired information. $PPID $PPID variable $PPID process ID The $PPID of a process is the process ID (pid) of its parent process. The PID of the currently running script is $$, of course. Compare this with the pidof command. $PROMPT_COMMAND $PROMPT_COMMAND variable prompt A variable holding a command to be executed just before the primary prompt, $PS1 is to be displayed. $PS1 $PS1 variable $PS1 prompt This is the main prompt, seen at the command line. $PS2 $PS2 variable $PS2 prompt secondary The secondary prompt, seen when additional input is expected. It displays as >. $PS3 $PS3 variable $PS3 prompt tertiary The tertiary prompt, displayed in a select loop (see ). $PS4 $PS4 variable $PS4 prompt quartenary The quartenary prompt, shown at the beginning of each line of output when invoking a script with the -x option. It displays as +. $PWD $PWD variable $PWD working directory directory working working directory (directory you are in at the time) This is the analog to the pwd builtin command. &wipedir; $REPLY $REPLY variable $REPLY default value of read reply read The default value when a variable is not supplied to read. Also applicable to select menus, but only supplies the item number of the variable chosen, not the value of the variable itself. &reply; $SECONDS $SECONDS variable $SECONDS seconds execution time runtime seconds The number of seconds the script has been running. &seconds; $SHELLOPTS $SHELLOPTS variable $SHELLOPTS shell options the list of enabled shell options, a readonly variable bash$ echo $SHELLOPTS braceexpand:hashall:histexpand:monitor:history:interactive-comments:emacs $SHLVL $SHLVL variable $SHLVL shell level Shell level, how deeply Bash is nested. Somewhat analogous to recursion, in this context nesting refers to a pattern embedded within a larger pattern. One of the definitions of nest, according to the 1913 edition of Webster's Dictionary, illustrates this beautifully: A collection of boxes, cases, or the like, of graduated size, each put within the one next larger. If, at the command line, $SHLVL is 1, then in a script it will increment to 2. This variable is not affected by subshells. Use $BASH_SUBSHELL when you need an indication of subshell nesting. $TMOUT $TMOUT variable $TMOUT timeout interval If the $TMOUT environmental variable is set to a non-zero value time, then the shell prompt will time out after $time seconds. This will cause a logout. As of version 2.05b of Bash, it is now possible to use $TMOUT in a script in combination with read. # Works in scripts for Bash, versions 2.05b and later. TMOUT=3 # Prompt times out at three seconds. echo "What is your favorite song?" echo "Quickly now, you only have $TMOUT seconds to answer!" read song if [ -z "$song" ] then song="(no answer)" # Default response. fi echo "Your favorite song is $song." There are other, more complex, ways of implementing timed input in a script. One alternative is to set up a timing loop to signal the script when it times out. This also requires a signal handling routine to trap (see ) the interrupt generated by the timing loop (whew!). Timed Input &tmdin; An alternative is using stty. Once more, timed input &timeout; Perhaps the simplest method is using the option to read. Timed <firstterm>read</firstterm> &tout; $UID $UID variable $UID user ID user ID number Current user's user identification number, as recorded in /etc/passwd This is the current user's real id, even if she has temporarily assumed another identity through su. $UID is a readonly variable, not subject to change from the command line or within a script, and is the counterpart to the id builtin. Am I root? &amiroot; See also . The variables $ENV, $LOGNAME, $MAIL, $TERM, $USER, and $USERNAME are not Bash builtins. These are, however, often set as environmental variables in one of the Bash startup files. $SHELL, the name of the user's login shell, may be set from /etc/passwd or in an init script, and it is likewise not a Bash builtin. tcsh% echo $LOGNAME bozo tcsh% echo $SHELL /bin/tcsh tcsh% echo $TERM rxvt bash$ echo $LOGNAME bozo bash$ echo $SHELL /bin/tcsh bash$ echo $TERM rxvt Positional Parameters $0, $1, $2, etc. $0 variable $0 positional parameter parameter positional Positional parameters, passed from command line to script, passed to a function, or set to a variable (see and ) $# $# variable $# positional parameter number of parameter positional number of Number of command line arguments The words argument and parameter are often used interchangeably. In the context of this document, they have the same precise meaning: a variable passed to a script or function. or positional parameters (see ) $* $* variable $* positional parameter all parameter positional all All of the positional parameters, seen as a single word $* must be quoted. $@ $@ variable $* positional parameter all Same as $*, but each parameter is a quoted string, that is, the parameters are passed on intact, without interpretation or expansion. This means, among other things, that each parameter in the argument list is seen as a separate word. Of course, $@ should be quoted. <firstterm>arglist</firstterm>: Listing arguments with $* and $@ &arglist; Following a shift, the $@ holds the remaining command-line parameters, lacking the previous $1, which was lost. #!/bin/bash # Invoke with ./scriptname 1 2 3 4 5 echo "$@" # 1 2 3 4 5 shift echo "$@" # 2 3 4 5 shift echo "$@" # 3 4 5 # Each "shift" loses parameter $1. # "$@" then contains the remaining parameters. The $@ special parameter finds use as a tool for filtering input into shell scripts. The cat "$@" construction accepts input to a script either from stdin or from files given as parameters to the script. See and . The $* and $@ parameters sometimes display inconsistent and puzzling behavior, depending on the setting of $IFS. Inconsistent <varname>$*</varname> and <varname>$@</varname> behavior &incompat; The $@ and $* parameters differ only when between double quotes. <varname>$*</varname> and <varname>$@</varname> when <varname>$IFS</varname> is empty &ifsempty; Other Special Parameters $- $- variable $- flags Flags passed to script (using set). See . This was originally a ksh construct adopted into Bash, and unfortunately it does not seem to work reliably in Bash scripts. One possible use for it is to have a script self-test whether it is interactive. $! $! variable $! PID last job background PID (process ID) of last job run in background LOG=$0.log COMMAND1="sleep 100" echo "Logging PIDs background commands for script: $0" >> "$LOG" # So they can be monitored, and killed as necessary. echo >> "$LOG" # Logging commands. echo -n "PID of \"$COMMAND1\": " >> "$LOG" ${COMMAND1} & echo $! >> "$LOG" # PID of "sleep 100": 1506 # Thank you, Jacques Lederer, for suggesting this. Using $! for job control: possibly_hanging_job & { sleep ${TIMEOUT}; eval 'kill -9 $!' &> /dev/null; } # Forces completion of an ill-behaved program. # Useful, for example, in init scripts. # Thank you, Sylvain Fourmanoit, for this creative use of the "!" variable. Or, alternately: # This example by Matthew Sage. # Used with permission. TIMEOUT=30 # Timeout value in seconds count=0 possibly_hanging_job & { while ((count < TIMEOUT )); do eval '[ ! -d "/proc/$!" ] && ((count = TIMEOUT))' # /proc is where information about running processes is found. # "-d" tests whether it exists (whether directory exists). # So, we're waiting for the job in question to show up. ((count++)) sleep 1 done eval '[ -d "/proc/$!" ] && kill -15 $!' # If the hanging job is running, kill it. } $_ $_ variable $_ underscore last argument Special variable set to last argument of previous command executed. Underscore variable #!/bin/bash echo $_ # /bin/bash # Just called /bin/bash to run the script. du >/dev/null # So no output from command. echo $_ # du ls -al >/dev/null # So no output from command. echo $_ # -al (last argument) : echo $_ # : $? $? variable $? exit status Exit status of a command, function, or the script itself (see ) $$ $$ variable $$ PID of script Process ID of the script itself. The $$ variable often finds use in scripts to construct unique temp file names (see , , , and ). This is usually simpler than invoking mktemp. Manipulating Strings Bash supports a surprising number of string manipulation operations. Unfortunately, these tools lack a unified focus. Some are a subset of parameter substitution, and others fall under the functionality of the UNIX expr command. This results in inconsistent command syntax and overlap of functionality, not to mention confusion. String Length ${#string} string length parameter substitution expr length $string string length expr These are the equivalent of strlen() in C. expr "$string" : '.*' string length expr stringZ=abcABC123ABCabc echo ${#stringZ} # 15 echo `expr length $stringZ` # 15 echo `expr "$stringZ" : '.*'` # 15 Inserting a blank line between paragraphs in a text file ¶graphspace; Length of Matching Substring at Beginning of String expr match "$string" '$substring' substring length expr $substring is a regular expression. expr "$string" : '$substring' substring length expr $substring is a regular expression. stringZ=abcABC123ABCabc # |------| # 12345678 echo `expr match "$stringZ" 'abc[A-Z]*.2'` # 8 echo `expr "$stringZ" : 'abc[A-Z]*.2'` # 8 Index expr index $string $substring substring index expr Numerical position in $string of first character in $substring that matches. stringZ=abcABC123ABCabc # 123456 ... echo `expr index "$stringZ" C12` # 6 # C position. echo `expr index "$stringZ" 1c` # 3 # 'c' (in #3 position) matches before '1'. This is the near equivalent of strchr() in C. Substring Extraction ${string:position} substring extraction Extracts substring from $string at $position. If the $string parameter is * or @, then this extracts the positional parameters, This applies to either command-line arguments or parameters passed to a function. starting at $position. ${string:position:length} substring extraction Extracts $length characters of substring from $string at $position. stringZ=abcABC123ABCabc # 0123456789..... # 0-based indexing. echo ${stringZ:0} # abcABC123ABCabc echo ${stringZ:1} # bcABC123ABCabc echo ${stringZ:7} # 23ABCabc echo ${stringZ:7:3} # 23A # Three characters of substring. # Is it possible to index from the right end of the string? echo ${stringZ:-4} # abcABC123ABCabc # Defaults to full string, as in ${parameter:-default}. # However . . . echo ${stringZ:(-4)} # Cabc echo ${stringZ: -4} # Cabc # Now, it works. # Parentheses or added space "escape" the position parameter. # Thank you, Dan Jacobson, for pointing this out. The position and length arguments can be parameterized, that is, represented as a variable, rather than as a numerical constant. Generating an 8-character <quote>random</quote> string &randstring; If the $string parameter is * or @, then this extracts a maximum of $length positional parameters, starting at $position. echo ${*:2} # Echoes second and following positional parameters. echo ${@:2} # Same as above. echo ${*:2:3} # Echoes three positional parameters, starting at second. expr substr $string $position $length substring extraction expr Extracts $length characters from $string starting at $position. stringZ=abcABC123ABCabc # 123456789...... # 1-based indexing. echo `expr substr $stringZ 1 2` # ab echo `expr substr $stringZ 4 3` # ABC expr match "$string" '$$substring$' substring extraction expr Extracts $substring at beginning of $string, where $substring is a regular expression. expr "$string" : '$$substring$' substring extraction expr Extracts $substring at beginning of $string, where $substring is a regular expression. stringZ=abcABC123ABCabc # ======= echo `expr match "$stringZ" '$.[b-c]*[A-Z]..[0-9]$'` # abcABC1 echo `expr "$stringZ" : '$.[b-c]*[A-Z]..[0-9]$'` # abcABC1 echo `expr "$stringZ" : '$.......$'` # abcABC1 # All of the above forms give an identical result. expr match "$string" '.*$$substring$' substring extraction expr Extracts $substring at end of $string, where $substring is a regular expression. expr "$string" : '.*$$substring$' substring extraction expr Extracts $substring at end of $string, where $substring is a regular expression. stringZ=abcABC123ABCabc # ====== echo `expr match "$stringZ" '.*$[A-C][A-C][A-C][a-c]*$'` # ABCabc echo `expr "$stringZ" : '.*$......$'` # ABCabc Substring Removal ${string#substring} substring removal Deletes shortest match of $substring from front of $string. ${string##substring} substring removal Deletes longest match of $substring from front of $string. stringZ=abcABC123ABCabc # |----| shortest # |----------| longest echo ${stringZ#a*C} # 123ABCabc # Strip out shortest match between 'a' and 'C'. echo ${stringZ##a*C} # abc # Strip out longest match between 'a' and 'C'. ${string%substring} substring removal Deletes shortest match of $substring from back of $string. For example: # Rename all filenames in $PWD with "TXT" suffix to a "txt" suffix. # For example, "file1.TXT" becomes "file1.txt" . . . SUFF=TXT suff=txt for i in $(ls *.$SUFF) do mv -f $i ${i%.$SUFF}.$suff # Leave unchanged everything *except* the shortest pattern match #+ starting from the right-hand-side of the variable $i . . . done ### This could be condensed into a "one-liner" if desired. # Thank you, Rory Winston. ${string%%substring} substring removal Deletes longest match of $substring from back of $string. stringZ=abcABC123ABCabc # || shortest # |------------| longest echo ${stringZ%b*c} # abcABC123ABCa # Strip out shortest match between 'b' and 'c', from back of $stringZ. echo ${stringZ%%b*c} # a # Strip out longest match between 'b' and 'c', from back of $stringZ. This operator is useful for generating filenames. Converting graphic file formats, with filename change &cvt; Converting streaming audio files to <firstterm>ogg</firstterm> &ra2ogg; A simple emulation of getopt using substring-extraction constructs. Emulating <firstterm>getopt</firstterm> &getoptsimple; Substring Replacement ${string/substring/replacement} substring replacement Replace first match of $substring with $replacement. ${string//substring/replacement} substring replacement Replace all matches of $substring with $replacement. stringZ=abcABC123ABCabc echo ${stringZ/abc/xyz} # xyzABC123ABCabc # Replaces first match of 'abc' with 'xyz'. echo ${stringZ//abc/xyz} # xyzABC123ABCxyz # Replaces all matches of 'abc' with # 'xyz'. # What happens if no $replacement string is supplied? echo ${stringZ/abc} # ABC123ABCabc echo ${stringZ//abc} # ABC123ABC # A simple deletion takes place. ${string/#substring/replacement} substring replacement If $substring matches front end of $string, substitute $replacement for $substring. ${string/%substring/replacement} substring replacement If $substring matches back end of $string, substitute $replacement for $substring. stringZ=abcABC123ABCabc echo ${stringZ/#abc/XYZ} # XYZABC123ABCabc # Replaces front-end match of 'abc' with 'XYZ'. echo ${stringZ/%abc/XYZ} # abcABC123ABCXYZ # Replaces back-end match of 'abc' with 'XYZ'. Manipulating strings using awk A Bash script may invoke the string manipulation facilities of awk as an alternative to using its built-in operations. Alternate ways of extracting and locating substrings &substringex; Further Reference For more on string manipulation in scripts, refer to and the relevant section of the expr command listing. Script examples: Parameter Substitution <anchor id="pssub1">Manipulating and/or expanding variables ${parameter} Same as $parameter, i.e., value of the variable parameter. In certain contexts, only the less ambiguous ${parameter} form works. May be used for concatenating variables with strings. your_id=${USER}-on-${HOSTNAME} echo "$your_id" # echo "Old \$PATH = $PATH" PATH=${PATH}:/opt/bin #Add /opt/bin to $PATH for duration of script. echo "New \$PATH = $PATH" ${parameter-default} ${parameter:-default} If parameter not set, use default. echo ${username-`whoami`} # Echoes the result of `whoami`, if variable $username is still unset. ${parameter-default} and ${parameter:-default} are almost equivalent. The extra : makes a difference only when parameter has been declared, but is null. ¶msub; The default parameter construct finds use in providing missing command-line arguments in scripts. DEFAULT_FILENAME=generic.data filename=${1:-$DEFAULT_FILENAME} # If not otherwise specified, the following command block operates #+ on the file "generic.data". # # Commands follow. See also , , and . Compare this method with using an and list to supply a default command-line argument. ${parameter=default} ${parameter:=default} If parameter not set, set it to default. Both forms nearly equivalent. The : makes a difference only when $parameter has been declared and is null, If $parameter is null in a non-interactive script, it will terminate with a 127 exit status (the Bash error code for command not found). as above. echo ${username=`whoami`} # Variable "username" is now set to `whoami`. ${parameter+alt_value} ${parameter:+alt_value} If parameter set, use alt_value, else use null string. Both forms nearly equivalent. The : makes a difference only when parameter has been declared and is null, see below. echo "###### \${parameter+alt_value} ########" echo a=${param1+xyz} echo "a = $a" # a = param2= a=${param2+xyz} echo "a = $a" # a = xyz param3=123 a=${param3+xyz} echo "a = $a" # a = xyz echo echo "###### \${parameter:+alt_value} ########" echo a=${param4:+xyz} echo "a = $a" # a = param5= a=${param5:+xyz} echo "a = $a" # a = # Different result from a=${param5+xyz} param6=123 a=${param6:+xyz} echo "a = $a" # a = xyz ${parameter?err_msg} ${parameter:?err_msg} If parameter set, use it, else print err_msg. Both forms nearly equivalent. The : makes a difference only when parameter has been declared and is null, as above. Using parameter substitution and error messages &ex6; Parameter substitution and <quote>usage</quote> messages &usagemessage; Parameter substitution and/or expansion The following expressions are the complement to the match in expr string operations (see ). These particular ones are used mostly in parsing file path names. <anchor id="psorex1">Variable length / Substring removal ${#var} String length (number of characters in $var). For an array, ${#array} is the length of the first element in the array. Exceptions: ${#*} and ${#@} give the number of positional parameters. For an array, ${#array[*]} and ${#array[@]} give the number of elements in the array. Length of a variable &length; ${var#Pattern} ${var##Pattern} ${var#Pattern} Remove from $var the shortest part of $Pattern that matches the front end of $var. ${var##Pattern} Remove from $var the longest part of $Pattern that matches the front end of $var. A usage illustration from : # Function from "days-between.sh" example. # Strips leading zero(s) from argument passed. strip_leading_zero () # Strip possible leading zero(s) { #+ from argument passed. return=${1#0} # The "1" refers to "$1" -- passed arg. } # The "0" is what to remove from "$1" -- strips zeros. Manfred Schwarb's more elaborate variation of the above: strip_leading_zero2 () # Strip possible leading zero(s), since otherwise { # Bash will interpret such numbers as octal values. shopt -s extglob # Turn on extended globbing. local val=${1##+(0)} # Use local variable, longest matching series of 0's. shopt -u extglob # Turn off extended globbing. _strip_leading_zero2=${val:-0} # If input was 0, return 0 instead of "". } Another usage illustration: echo `basename $PWD` # Basename of current working directory. echo "${PWD##*/}" # Basename of current working directory. echo echo `basename $0` # Name of script. echo $0 # Name of script. echo "${0##*/}" # Name of script. echo filename=test.data echo "${filename##*.}" # data # Extension of filename. ${var%Pattern} ${var%%Pattern} {$var%Pattern} Remove from $var the shortest part of $Pattern that matches the back end of $var. {$var%%Pattern} Remove from $var the longest part of $Pattern that matches the back end of $var. Version 2 of Bash added additional options. Pattern matching in parameter substitution &pattmatching; Renaming file extensions<token>:</token> &rfe; <anchor id="exprepl1">Variable expansion / Substring replacement These constructs have been adopted from ksh. ${var:pos} Variable var expanded, starting from offset pos. ${var:pos:len} Expansion to a max of len characters of variable var, from offset pos. See for an example of the creative use of this operator. ${var/Pattern/Replacement} First match of Pattern, within var replaced with Replacement. If Replacement is omitted, then the first match of Pattern is replaced by nothing, that is, deleted. ${var//Pattern/Replacement} Global replacement All matches of Pattern, within var replaced with Replacement. As above, if Replacement is omitted, then all occurrences of Pattern are replaced by nothing, that is, deleted. Using pattern matching to parse arbitrary strings &ex7; ${var/#Pattern/Replacement} If prefix of var matches Pattern, then substitute Replacement for Pattern. ${var/%Pattern/Replacement} If suffix of var matches Pattern, then substitute Replacement for Pattern. Matching patterns at prefix or suffix of string &varmatch; ${!varprefix*} ${!varprefix@} Matches names of all previously declared variables beginning with varprefix. # This is a variation on indirect reference, but with a * or @. # Bash, version 2.04, adds this feature. xyz23=whatever xyz24= a=${!xyz*} # Expands to *names* of declared variables # ^ ^ ^ + beginning with "xyz". echo "a = $a" # a = xyz23 xyz24 a=${!xyz@} # Same as above. echo "a = $a" # a = xyz23 xyz24 echo "---" abc23=something_else b=${!abc*} echo "b = $b" # b = abc23 c=${!b} # Now, the more familiar type of indirect reference. echo $c # something_else declare typeset command declare command typeset Typing variables: <command>declare</command> or <command>typeset</command> The declare or typeset builtins, which are exact synonyms, permit modifying the properties of variables. This is a very weak form of the typing In this context, typing a variable means to classify it and restrict its properties. For example, a variable declared or typed as an integer is no longer available for string operations. declare -i intvar intvar=23 echo "$intvar" # 23 intvar=stringval echo "$intvar" # 0 available in certain programming languages. The declare command is specific to version 2 or later of Bash. The typeset command also works in ksh scripts. <anchor id="declareopsref1">declare/typeset options -r readonly (declare -r var1 works the same as readonly var1) This is the rough equivalent of the C const type qualifier. An attempt to change the value of a readonly variable fails with an error message. declare -r var1=1 echo "var1 = $var1" # var1 = 1 (( var1++ )) # x.sh: line 4: var1: readonly variable -i integer declare -i number # The script will treat subsequent occurrences of "number" as an integer. number=3 echo "Number = $number" # Number = 3 number=three echo "Number = $number" # Number = 0 # Tries to evaluate the string "three" as an integer. Certain arithmetic operations are permitted for declared integer variables without the need for expr or let. n=6/3 echo "n = $n" # n = 6/3 declare -i n n=6/3 echo "n = $n" # n = 2 -a array declare -a indices The variable indices will be treated as an array. -f function(s) declare -f A declare -f line with no arguments in a script causes a listing of all the functions previously defined in that script. declare -f function_name A declare -f function_name in a script lists just the function named. -x export declare -x var3 This declares a variable as available for exporting outside the environment of the script itself. -x var=$value declare -x var3=373 The declare command permits assigning a value to a variable in the same statement as setting its properties. Using <firstterm>declare</firstterm> to type variables &ex20; Using the declare builtin restricts the scope of a variable. foo () { FOO="bar" } bar () { foo echo $FOO } bar # Prints bar. However . . . foo (){ declare FOO="bar" } bar () { foo echo $FOO } bar # Prints nothing. # Thank you, Michael Iatrou, for pointing this out. Indirect References We have seen that referencing a variable, $var, fetches its value. But, what about the value of a value? What about $$var? The actual notation is \$$var, usually preceded by an eval (and sometimes an echo). This is called an indirect reference. Indirect Variable References &indref; Of what practical use is indirect referencing of variables? It gives Bash a little of the functionality of pointers in C, for instance, in table lookup. And, it also has some other very interesting applications. . . . Nils Radtke shows how to build dynamic variable names and evaluate their contents. This can be useful when sourcing configuration files. #!/bin/bash # --------------------------------------------- # This could be "sourced" from a separate file. isdnMyProviderRemoteNet=172.16.0.100 isdnYourProviderRemoteNet=10.0.0.10 isdnOnlineService="MyProvider" # --------------------------------------------- remoteNet=$(eval "echo \$$(echo isdn${isdnOnlineService}RemoteNet)") remoteNet=$(eval "echo \$$(echo isdnMyProviderRemoteNet)") remoteNet=$(eval "echo \$isdnMyProviderRemoteNet") remoteNet=$(eval "echo $isdnMyProviderRemoteNet") echo "$remoteNet" # 172.16.0.100 # ================================================================ # And, it gets even better. # Consider the following snippet given a variable named getSparc, #+ but no such variable getIa64: chkMirrorArchs () { arch="$1"; if [ "$(eval "echo \${$(echo get$(echo -ne $arch | sed 's/^$.$.*/\1/g' | tr 'a-z' 'A-Z'; echo $arch | sed 's/^.$.*$/\1/g')):-false}")" = true ] then return 0; else return 1; fi; } getSparc="true" unset getIa64 chkMirrorArchs sparc echo $? # 0 # True chkMirrorArchs Ia64 echo $? # 1 # False # Notes: # ----- # Even the to-be-substituted variable name part is built explicitly. # The parameters to the chkMirrorArchs calls are all lower case. # The variable name is composed of two parts: "get" and "Sparc" . . . Passing an indirect reference to <firstterm>awk</firstterm> &coltotaler2; This method of indirect referencing is a bit tricky. If the second order variable changes its value, then the first order variable must be properly dereferenced (as in the above example). Fortunately, the ${!variable} notation introduced with version 2 of Bash (see and ) makes indirect referencing more intuitive. Bash does not support pointer arithmetic, and this severely limits the usefulness of indirect referencing. In fact, indirect referencing in a scripting language is, at best, an afterthought. $RANDOM variable $RANDOM $RANDOM: generate random integer $RANDOM is an internal Bash function (not a constant) that returns a pseudorandom True randomness, insofar as it exists at all, can only be found in certain incompletely understood natural phenomena, such as radioactive decay. Computers only simulate randomness, and computer-generated sequences of random numbers are therefore referred to as pseudorandom. integer in the range 0 - 32767. It should not be used to generate an encryption key. Generating random numbers &ex21; Picking a random card from a deck &pickcard; Brownian Motion Simulation &brownian; Jipe points out a set of techniques for generating random numbers within a range. # Generate random number between 6 and 30. rnumber=$((RANDOM%25+6)) # Generate random number in the same 6 - 30 range, #+ but the number must be evenly divisible by 3. rnumber=$(((RANDOM%30/3+1)*3)) # Note that this will not work all the time. # It fails if $RANDOM%30 returns 0. # Frank Wang suggests the following alternative: rnumber=$(( RANDOM%27/3*3+6 )) Bill Gradwohl came up with an improved formula that works for positive numbers. rnumber=$(((RANDOM%(max-min+divisibleBy))/divisibleBy*divisibleBy+min)) Here Bill presents a versatile function that returns a random number between two specified values. Random between values &randombetween; Just how random is $RANDOM? The best way to test this is to write a script that tracks the distribution of random numbers generated by $RANDOM. Let's roll a $RANDOM die a few times . . . Rolling a single die with RANDOM &randomtest; As we have seen in the last example, it is best to reseed the RANDOM generator each time it is invoked. Using the same seed for RANDOM repeats the same series of numbers. The seed of a computer-generated pseudorandom number series can be considered an identification label. For example, think of the pseudorandom series with a seed of 23 as Series #23. A property of a pseurandom number series is the length of the cycle before it starts repeating itself. A good pseurandom generator will produce series with very long cycles. (This mirrors the behavior of the random() function in C.) Reseeding RANDOM &seedingrandom; The /dev/urandom pseudo-device file provides a method of generating much more random pseudorandom numbers than the $RANDOM variable. dd if=/dev/urandom of=targetfile bs=1 count=XX creates a file of well-scattered pseudorandom numbers. However, assigning these numbers to a variable in a script requires a workaround, such as filtering through od (as in above example, , and ), or even piping to md5sum (see ). There are also other ways to generate pseudorandom numbers in a script. Awk provides a convenient means of doing this. Pseudorandom numbers, using <link linkend="awkref">awk</link> &random2; The date command also lends itself to generating pseudorandom integer sequences. The Double-Parentheses Construct Similar to the let command, the (( ... )) construct permits arithmetic expansion and evaluation. In its simplest form, a=$(( 5 + 3 )) would set a to 5 + 3, or 8. However, this double-parentheses construct is also a mechanism for allowing C-style manipulation of variables in Bash, for example, (( var++ )). C-style manipulation of variables &cvars; See also and . Loops and Branches What needs this iteration, woman? --Shakespeare, Othello Operations on code blocks are the key to structured and organized shell scripts. Looping and branching constructs provide the tools for accomplishing this. Loops A loop is a block of code that iterates Iteration: Repeated execution of a command or group of commands -- usually, but not always -- while a given condition holds, or until a given condition is met. a list of commands as long as the loop control condition is true. <anchor id="forloopref1">for loops for arg in [list] for in do done loop for This is the basic looping construct. It differs significantly from its C counterpart. for arg in list do command(s) done During each pass through the loop, arg takes on the value of each successive variable in the list. for arg in "$var1" "$var2" "$var3" ... "$varN" # In pass 1 of the loop, arg = $var1 # In pass 2 of the loop, arg = $var2 # In pass 3 of the loop, arg = $var3 # ... # In pass N of the loop, arg = $varN # Arguments in [list] quoted to prevent possible word splitting. The argument list may contain wild cards. If do is on same line as for, there needs to be a semicolon after list. for arg in list ; do Simple <firstterm>for</firstterm> loops &ex22; Each [list] element may contain multiple parameters. This is useful when processing parameters in groups. In such cases, use the set command (see ) to force parsing of each [list] element and assignment of each component to the positional parameters. <firstterm>for</firstterm> loop with two parameters in each [list] element &ex22a; A variable may supply the [list] in a for loop. <emphasis>Fileinfo:</emphasis> operating on a file list contained in a variable &fileinfo; If the [list] in a for loop contains wild cards (* and ?) used in filename expansion, then globbing takes place. Operating on files with a <firstterm>for</firstterm> loop &listglob; Omitting the in [list] part of a for loop causes the loop to operate on $@ -- the positional parameters. A particularly clever illustration of this is . See also . Missing <userinput>in [list]</userinput> in a <firstterm>for</firstterm> loop &ex23; It is possible to use command substitution to generate the [list] in a for loop. See also , and . Generating the <userinput>[list]</userinput> in a <firstterm>for</firstterm> loop with command substitution &forloopcmd; Here is a somewhat more complex example of using command substitution to create the [list]. A <firstterm>grep</firstterm> replacement for binary files &bingrep; More of the same. Listing all users on the system &userlist; Yet another example of the [list] resulting from command substitution. Checking all the binaries in a directory for authorship &findstring; A final example of [list] / command substitution, but this time the command is a function. generate_list () { echo "one two three" } for word in $(generate_list) # Let "word" grab output of function. do echo "$word" done # one # two # three The output of a for loop may be piped to a command or commands. Listing the <firstterm>symbolic links</firstterm> in a directory &symlinks; The stdout of a loop may be redirected to a file, as this slight modification to the previous example shows. Symbolic links in a directory, saved to a file &symlinks2; There is an alternative syntax to a for loop that will look very familiar to C programmers. This requires double parentheses. A C-style <firstterm>for</firstterm> loop &forloopc; See also , , and . --- Now, a for loop used in a real-life context. Using <firstterm>efax</firstterm> in batch mode &ex24; while while do done loop while This construct tests for a condition at the top of a loop, and keeps looping as long as that condition is true (returns a 0 exit status). In contrast to a for loop, a while loop finds use in situations where the number of loop repetitions is not known beforehand. while condition do command(s) done The bracket construct in a while loop is nothing more than our old friend, the test brackets used in an if/then test. In fact, a while loop can legally use the more versatile double-brackets construct (while [[ condition ]]). As is the case with for loops, placing the do on the same line as the condition test requires a semicolon. while condition ; do Note that the test brackets are not mandatory in a while loop. See, for example, the getopts construct. Simple <firstterm>while</firstterm> loop &ex25; Another <firstterm>while</firstterm> loop &ex26; A while loop may have multiple conditions. Only the final condition determines when the loop terminates. This necessitates a slightly different loop syntax, however. <firstterm>while</firstterm> loop with multiple conditions &ex26a; As with a for loop, a while loop may employ C-style syntax by using the double-parentheses construct (see also ). C-style syntax in a <firstterm>while</firstterm> loop &whloopc; Inside its test brackets, a while loop can call a function. t=0 condition () { ((t++)) if [ $t -lt 5 ] then return 0 # true else return 1 # false fi } while condition # ^^^^^^^^^ # Function call -- four loop iterations. do echo "Still going: t = $t" done # Still going: t = 1 # Still going: t = 2 # Still going: t = 3 # Still going: t = 4 Similar to the if-test construct, a while loop can omit the test brackets. while condition do ... done By coupling the power of the read command with a while loop, we get the handy while read construct, useful for reading and parsing files. cat $filename | # Supply input from a file. while read line # As long as there is another line to read ... do ... done # =========== Snippet from "sd.sh" example script ========== # while read value # Read one data point at a time. do rt=$(echo "scale=$SC; $rt + $value" | bc) (( ct++ )) done am=$(echo "scale=$SC; $rt / $ct" | bc) echo $am; return $ct # This function "returns" TWO values! # Caution: This little trick will not work if $ct > 255! # To handle a larger number of data points, #+ simply comment out the "return $ct" above. } <"$datafile" # Feed in data file. A while loop may have its stdin redirected to a file by a < at its end. A while loop may have its stdin supplied by a pipe. until until do done loop until This construct tests for a condition at the top of a loop, and keeps looping as long as that condition is false (opposite of while loop). until condition-is-true do command(s) done Note that an until loop tests for the terminating condition at the top of the loop, differing from a similar construct in some programming languages. As is the case with for loops, placing the do on the same line as the condition test requires a semicolon. until condition-is-true ; do <firstterm>until</firstterm> loop &ex27; How to choose between a for loop or a while loop or until loop? In C, you would typically use a for loop when the number of loop iterations is known beforehand. With Bash, however, the situation is fuzzier. The Bash for loop is more loosely structured and more flexible than its equivalent in other languages. Therefore, feel free to use whatever type of loop gets the job done in the simplest way. Nested Loops A nested loop is a loop within a loop, an inner loop within the body of an outer one. How this works is that the first pass of the outer loop triggers the inner loop, which executes to completion. Then the second pass of the outer loop triggers the inner loop again. This repeats until the outer loop finishes. Of course, a break within either the inner or outer loop would interrupt this process. Nested Loop &nestedloop; See for an illustration of nested while loops, and to see a while loop nested inside an until loop. Loop Control Tournez cent tours, tournez mille tours, Tournez souvent et tournez toujours . . . --Verlaine, Chevaux de bois <anchor id="brkcont1">Commands affecting loop behavior break continue loop break loop continue break continue The break and continue loop control commands These are shell builtins, whereas other loop commands, such as while and case, are keywords. correspond exactly to their counterparts in other programming languages. The break command terminates the loop (breaks out of it), while continue causes a jump to the next iteration of the loop, skipping all the remaining commands in that particular loop cycle. Effects of <firstterm>break</firstterm> and <command>continue</command> in a loop &ex28; The break command may optionally take a parameter. A plain break terminates only the innermost loop in which it is embedded, but a break N breaks out of N levels of loop. Breaking out of multiple loop levels &breaklevels; The continue command, similar to break, optionally takes a parameter. A plain continue cuts short the current iteration within its loop and begins the next. A continue N terminates all remaining iterations at its loop level and continues with the next iteration at the loop, levels above. Continuing at a higher loop level &continuelevels; Using <firstterm>continue N</firstterm> in an actual task &continuenex; The continue N construct is difficult to understand and tricky to use in any meaningful context. It is probably best avoided. Testing and Branching The case and select constructs are technically not loops, since they do not iterate the execution of a code block. Like loops, however, they direct program flow according to conditions at the top or bottom of the block. <anchor id="caseesac1">Controlling program flow in a code block case (in) / esac case in esac switch ;; menus The case construct is the shell scripting analog to switch in C/C++. It permits branching to one of a number of code blocks, depending on condition tests. It serves as a kind of shorthand for multiple if/then/else statements and is an appropriate tool for creating menus. case "$variable" in "$condition1" ) command ;; "$condition2" ) command ;; esac Quoting the variables is not mandatory, since word splitting does not take place. Each test line ends with a right paren ). Each condition block ends with a double semicolon ;;. The entire case block terminates with an esac (case spelled backwards). Using <firstterm>case</firstterm> &ex29; Creating menus using <firstterm>case</firstterm> &ex30; An exceptionally clever use of case involves testing for command-line parameters. #! /bin/bash case "$1" in "") echo "Usage: ${0##*/} <filename>"; exit $E_PARAM;; # No command-line parameters, # or first parameter empty. # Note that ${0##*/} is ${var##pattern} param substitution. # Net result is $0. -*) FILENAME=./$1;; # If filename passed as argument ($1) #+ starts with a dash, #+ replace it with ./$1 #+ so further commands don't interpret it #+ as an option. * ) FILENAME=$1;; # Otherwise, $1. esac Here is an more straightforward example of command-line parameter handling: #! /bin/bash while [ $# -gt 0 ]; do # Until you run out of parameters . . . case "$1" in -d|--debug) # "-d" or "--debug" parameter? DEBUG=1 ;; -c|--conf) CONFFILE="$2" shift if [ ! -f $CONFFILE ]; then echo "Error: Supplied file doesn't exist!" exit $E_CONFFILE # File not found error. fi ;; esac shift # Check next set of parameters. done # From Stefano Falsetto's "Log2Rot" script, #+ part of his "rottlog" package. # Used with permission. Using <firstterm>command substitution</firstterm> to generate the <firstterm>case</firstterm> variable &casecmd; A case construct can filter strings for globbing patterns. Simple string matching &matchstring; Checking for alphabetic input &isalpha; select select menus The select construct, adopted from the Korn Shell, is yet another tool for building menus. select variable in list do command break done This prompts the user to enter one of the choices presented in the variable list. Note that select uses the $PS3 prompt (#? ) by default, but this may be changed. Creating menus using <firstterm>select</firstterm> &ex31; If in list is omitted, then select uses the list of command line arguments ($@) passed to the script or to the function in which the select construct is embedded. Compare this to the behavior of a for variable in list construct with the in list omitted. Creating menus using <firstterm>select</firstterm> in a function &ex32; See also . Command Substitution $ special character ` Command substitution reassigns the output of a command For purposes of command substitution, a command may be an external system command, an internal scripting builtin, or even a script function. or even multiple commands; it literally plugs the command output into another context. In a more technically correct sense, command substitution extracts the stdout of a command, then assigns it to a variable using the = operator. The classic form of command substitution uses backquotes (`...`). Commands within backquotes (backticks) generate command line text. script_name=`basename $0` echo "The name of this script is $script_name." The output of commands can be used as arguments to another command, to set a variable, and even for generating the argument list in a <link linkend="forloopref1">for</link> loop. rm `cat filename` # filename contains a list of files to delete. # # S. C. points out that "arg list too long" error might result. # Better is xargs rm -- < filename # ( -- covers those cases where filename begins with a - ) textfile_listing=`ls *.txt` # Variable contains names of all *.txt files in current working directory. echo $textfile_listing textfile_listing2=$(ls *.txt) # The alternative form of command substitution. echo $textfile_listing2 # Same result. # A possible problem with putting a list of files into a single string # is that a newline may creep in. # # A safer way to assign a list of files to a parameter is with an array. # shopt -s nullglob # If no match, filename expands to nothing. # textfile_listing=( *.txt ) # # Thanks, S.C. Command substitution invokes a subshell. Command substitution may result in word splitting. COMMAND `echo a b` # 2 args: a and b COMMAND "`echo a b`" # 1 arg: "a b" COMMAND `echo` # no arg COMMAND "`echo`" # one empty arg # Thanks, S.C. Even when there is no word splitting, command substitution can remove trailing newlines. # cd "`pwd`" # This should always work. # However... mkdir 'dir with trailing newline ' cd 'dir with trailing newline ' cd "`pwd`" # Error message: # bash: cd: /tmp/file with trailing newline: No such file or directory cd "$PWD" # Works fine. old_tty_setting=$(stty -g) # Save old terminal setting. echo "Hit a key " stty -icanon -echo # Disable "canonical" mode for terminal. # Also, disable *local* echo. key=$(dd bs=1 count=1 2> /dev/null) # Using 'dd' to get a keypress. stty "$old_tty_setting" # Restore old setting. echo "You hit ${#key} key." # ${#variable} = number of characters in $variable # # Hit any key except RETURN, and the output is "You hit 1 key." # Hit RETURN, and it's "You hit 0 key." # The newline gets eaten in the command substitution. Thanks, S.C. Using echo to output an unquoted variable set with command substitution removes trailing newlines characters from the output of the reassigned command(s). This can cause unpleasant surprises. dir_listing=`ls -l` echo $dir_listing # unquoted # Expecting a nicely ordered directory listing. # However, what you get is: # total 3 -rw-rw-r-- 1 bozo bozo 30 May 13 17:15 1.txt -rw-rw-r-- 1 bozo # bozo 51 May 15 20:57 t2.sh -rwxr-xr-x 1 bozo bozo 217 Mar 5 21:13 wi.sh # The newlines disappeared. echo "$dir_listing" # quoted # -rw-rw-r-- 1 bozo 30 May 13 17:15 1.txt # -rw-rw-r-- 1 bozo 51 May 15 20:57 t2.sh # -rwxr-xr-x 1 bozo 217 Mar 5 21:13 wi.sh Command substitution even permits setting a variable to the contents of a file, using either redirection or the cat command. variable1=`<file1` # Set "variable1" to contents of "file1". variable2=`cat file2` # Set "variable2" to contents of "file2". # This, however, forks a new process, #+ so the line of code executes slower than the above version. # Note: # The variables may contain embedded whitespace, #+ or even (horrors), control characters. # Excerpts from system file, /etc/rc.d/rc.sysinit #+ (on a Red Hat Linux installation) if [ -f /fsckoptions ]; then fsckoptions=`cat /fsckoptions` ... fi # # if [ -e "/proc/ide/${disk[$device]}/media" ] ; then hdmedia=`cat /proc/ide/${disk[$device]}/media` ... fi # # if [ ! -n "`uname -r | grep -- "-"`" ]; then ktag="`cat /proc/version`" ... fi # # if [ $usb = "1" ]; then sleep 5 mouseoutput=`cat /proc/bus/usb/devices 2>/dev/null|grep -E "^I.*Cls=03.*Prot=02"` kbdoutput=`cat /proc/bus/usb/devices 2>/dev/null|grep -E "^I.*Cls=03.*Prot=01"` ... fi Do not set a variable to the contents of a long text file unless you have a very good reason for doing so. Do not set a variable to the contents of a binary file, even as a joke. Stupid script tricks &stupscr; Notice that a buffer overrun does not occur. This is one instance where an interpreted language, such as Bash, provides more protection from programmer mistakes than a compiled language. Command substitution permits setting a variable to the output of a loop. The key to this is grabbing the output of an echo command within the loop. Generating a variable from a loop &csubloop; Command substitution makes it possible to extend the toolset available to Bash. It is simply a matter of writing a program or script that outputs to stdout (like a well-behaved UNIX tool should) and assigning that output to a variable. #include <stdio.h> /* "Hello, world." C program */ int main() { printf( "Hello, world." ); return (0); } bash$ gcc -o hello hello.c #!/bin/bash # hello.sh greeting=`./hello` echo $greeting bash$ sh hello.sh Hello, world. The $(...) form has superseded backticks for command substitution. output=$(sed -n /"$1"/p $file) # From "grp.sh" example. # Setting a variable to the contents of a text file. File_contents1=$(cat $file1) File_contents2=$(<$file2) # Bash permits this also. The $(...) form of command substitution treats a double backslash in a different way than `...`. bash$ echo `echo \\` bash$ echo $(echo \\) \ The $(...) form of command substitution permits nesting. In fact, nesting with backticks is also possible, but only by escaping the inner backticks, as John Default points out. word_count=` wc -w \`ls -l | awk '{print $9}'\` ` word_count=$( wc -w $(ls -l | awk '{print $9}') ) Or, for something a bit more elaborate . . . Finding anagrams &agram2; Examples of command substitution in shell scripts: Arithmetic Expansion Arithmetic expansion provides a powerful tool for performing (integer) arithmetic operations in scripts. Translating a string into a numerical expression is relatively straightforward using backticks, double parentheses, or let. <anchor id="arithexpvar1">Variations Arithmetic expansion with backticks (often used in conjunction with expr) arithmetic expansion arithmetic expansion z=`expr $z + 3` # The 'expr' command performs the expansion. Arithmetic expansion with double parentheses double parentheses and using let let let The use of backticks (backquotes) in arithmetic expansion has been superseded by double parentheses -- ((...)) and $((...)) -- and also by the very convenient let construction. z=$(($z+3)) z=$((z+3)) # Also correct. # Within double parentheses, #+ parameter dereferencing #+ is optional. # $((EXPRESSION)) is arithmetic expansion. # Not to be confused with #+ command substitution. # You may also use operations within double parentheses without assignment. n=0 echo "n = $n" # n = 0 (( n += 1 )) # Increment. # (( $n += 1 )) is incorrect! echo "n = $n" # n = 1 let z=z+3 let "z += 3" # Quotes permit the use of spaces in variable assignment. # The 'let' operator actually performs arithmetic evaluation, #+ rather than expansion. Examples of arithmetic expansion in scripts: Recess Time This bizarre little intermission gives the reader a chance to relax and maybe laugh a bit.

Fellow Linux user, greetings! You are reading something which will bring you luck and good fortune. Just e-mail a copy of this document to 10 of your friends. Before making the copies, send a 100-line Bash script to the first person on the list at the bottom of this letter. Then delete their name and add yours to the bottom of the list. Don't break the chain! Make the copies within 48 hours. Wilfred P. of Brooklyn failed to send out his ten copies and woke the next morning to find his job description changed to "COBOL programmer." Howard L. of Newport News sent out his ten copies and within a month had enough hardware to build a 100-node Beowulf cluster dedicated to playing Tuxracer. Amelia V. of Chicago laughed at this letter and broke the chain. Shortly thereafter, a fire broke out in her terminal and she now spends her days writing documentation for MS Windows. Don't break the chain! Send out your ten copies today!

Courtesy 'NIX "fortune cookies", with some alterations and many apologies Commands Mastering the commands on your Linux machine is an indispensable prelude to writing effective shell scripts. This section covers the following commands: . (See also source) ac adduser agetty agrep ar arch at autoload awk (See also Using awk for math operations) badblocks banner basename batch bc bg bind bison builtin bzgrep bzip2 cal caller cat cd chattr chfn chgrp chkconfig chmod chown chroot cksum clear clock cmp col colrm column comm command compress cp cpio cron crypt csplit cu cut date dc dd debugfs declare depmod df dialog diff diff3 diffstat dig dirname dirs disown dmesg doexec dos2unix du dump dumpe2fs e2fsck echo egrep enable enscript env eqn eval exec exit (Related topic: exit status) expand export expr factor false fdformat fdisk fg fgrep file find finger flex flock fmt fold free fsck ftp fuser getopt getopts gettext getty gnome-mount grep groff groupmod groups (Related topic: the $GROUPS variable) gs gzip halt hash hdparm head help hexdump host hostid hostname (Related topic: the $HOSTNAME variable) hwclock iconv id (Related topic: the $UID variable) ifconfig info infocmp init insmod install ip ipcalc iwconfig jobs join jot kill killall last lastcomm lastlog ldd less let lex lid ln locate lockfile logger logname logout logrotate look losetup lp ls lsdev lsmod lsof lspci lsusb ltrace lynx lzcat lzma m4 mail mailstats mailto make MAKEDEV man mcookie md5sum merge mesg mimencode mkbootdisk mkdir mke2fs mkfifo mkisofs mknod mkswap mktemp mmencode modinfo modprobe more mount msgfmt mv nc netconfig netstat newgrp nice nl nm nmap nohup nslookup objdump od passwd paste patch (Related topic: diff) pathchk pgrep pidof ping pkill popd pr printenv printf procinfo ps pstree ptx pushd pwd (Related topic: the $PWD variable) quota rcp rdev rdist read readelf readlink readonly reboot recode renice reset resize restore rev rlogin rm rmdir rmmod route rpm rpm2cpio rsh rsync runlevel run-parts rx rz sar scp script sdiff sed seq service set setquota setserial setterm sha1sum shar shopt shred shutdown size skill sleep slocate snice sort source sox split sq ssh stat strace strings strip stty su sudo sum suspend swapoff swapon sx sync sz tac tail tar tbl tcpdump tee telinit telnet Tex texexec time times tmpwatch top touch tput tr traceroute true tset tsort tty tune2fs type typeset ulimit umask umount uname unarc unarj uncompress unexpand uniq units unlzma unrar unset unsq unzip uptime usbmodules useradd userdel usermod users usleep uucp uudecode uuencode uux vacation vdir vmstat vrfy w wait wall watch wc wget whatis whereis which who whoami whois write xargs yacc yes zcat zdiff zdump zegrep zfgrep zgrep zip Internal Commands and Builtins builtin A builtin is a command contained within the Bash tool set, literally built in. This is either for performance reasons -- builtins execute faster than external commands, which usually require forking off a separate process -- or because a particular builtin needs direct access to the shell internals. When a command or the shell itself initiates (or spawns) a new subprocess to carry out a task, this is called forking. This new process is the child, and the process that forked it off is the parent. While the child process is doing its work, the parent process is still executing. Note that while a parent process gets the process ID of the child process, and can thus pass arguments to it, the reverse is not true. This can create problems that are subtle and hard to track down. A script that forks off multiple instances of itself &spawnscr; Generally, a Bash builtin does not fork a subprocess when it executes within a script. An external system command or filter in a script usually will fork a subprocess. A builtin may be a synonym to a system command of the same name, but Bash reimplements it internally. For example, the Bash echo command is not the same as /bin/echo, although their behavior is almost identical. #!/bin/bash echo "This line uses the \"echo\" builtin." /bin/echo "This line uses the /bin/echo system command." A keyword is a reserved word, token or operator. Keywords have a special meaning to the shell, and indeed are the building blocks of the shell's syntax. As examples, for, while, do, and ! are keywords. Similar to a builtin, a keyword is hard-coded into Bash, but unlike a builtin, a keyword is not in itself a command, but a subunit of a command construct. An exception to this is the time command, listed in the official Bash documentation as a keyword (reserved word). <anchor id="intio1">I/O echo echo command echo prints (to stdout) an expression or variable (see ). echo Hello echo $a An echo requires the option to print escaped characters. See . Normally, each echo command prints a terminal newline, but the option suppresses this. An echo can be used to feed a sequence of commands down a pipe. if echo "$VAR" | grep -q txt # if [[ $VAR = *txt* ]] then echo "$VAR contains the substring sequence \"txt\"" fi An echo, in combination with command substitution can set a variable. a=`echo "HELLO" | tr A-Z a-z` See also , , , and . Be aware that echo `command` deletes any linefeeds that the output of command generates. The $IFS (internal field separator) variable normally contains \n (linefeed) as one of its set of whitespace characters. Bash therefore splits the output of command at linefeeds into arguments to echo. Then echo outputs these arguments, separated by spaces. bash$ ls -l /usr/share/apps/kjezz/sounds -rw-r--r-- 1 root root 1407 Nov 7 2000 reflect.au -rw-r--r-- 1 root root 362 Nov 7 2000 seconds.au bash$ echo `ls -l /usr/share/apps/kjezz/sounds` total 40 -rw-r--r-- 1 root root 716 Nov 7 2000 reflect.au -rw-r--r-- 1 root root ... So, how can we embed a linefeed within an echoed character string? # Embedding a linefeed? echo "Why doesn't this string \n split on two lines?" # Doesn't split. # Let's try something else. echo echo $"A line of text containing a linefeed." # Prints as two distinct lines (embedded linefeed). # But, is the "$" variable prefix really necessary? echo echo "This string splits on two lines." # No, the "$" is not needed. echo echo "---------------" echo echo -n $"Another line of text containing a linefeed." # Prints as two distinct lines (embedded linefeed). # Even the -n option fails to suppress the linefeed here. echo echo echo "---------------" echo echo # However, the following doesn't work as expected. # Why not? Hint: Assignment to a variable. string1=$"Yet another line of text containing a linefeed (maybe)." echo $string1 # Yet another line of text containing a linefeed (maybe). # ^ # Linefeed becomes a space. # Thanks, Steve Parker, for pointing this out. This command is a shell builtin, and not the same as /bin/echo, although its behavior is similar. bash$ type -a echo echo is a shell builtin echo is /bin/echo printf printf command printf The printf, formatted print, command is an enhanced echo. It is a limited variant of the C language printf() library function, and its syntax is somewhat different. printf format-string parameter This is the Bash builtin version of the /bin/printf or /usr/bin/printf command. See the printf manpage (of the system command) for in-depth coverage. Older versions of Bash may not support printf. <firstterm>printf</firstterm> in action &ex47; Formatting error messages is a useful application of printf E_BADDIR=85 var=nonexistent_directory error() { printf "$@" >&2 # Formats positional params passed, and sends them to stderr. echo exit $E_BADDIR } cd $var || error $"Can't cd to %s." "$var" # Thanks, S.C. See also . read read command read Reads the value of a variable from stdin, that is, interactively fetches input from the keyboard. The option lets read get array variables (see ). Variable assignment, using <firstterm>read</firstterm> &ex36; A read without an associated variable assigns its input to the dedicated variable $REPLY. What happens when <firstterm>read</firstterm> has no variable &readnovar; Normally, inputting a \ suppresses a newline during input to a read. The option causes an inputted \ to be interpreted literally. Multi-line input to <firstterm>read</firstterm> &readr; The read command has some interesting options that permit echoing a prompt and even reading keystrokes without hitting ENTER. # Read a keypress without hitting ENTER. read -s -n1 -p "Hit a key " keypress echo; echo "Keypress was "\"$keypress\""." # -s option means do not echo input. # -n N option means accept only N characters of input. # -p option means echo the following prompt before reading input. # Using these options is tricky, since they need to be in the correct order. The option to read also allows detection of the arrow keys and certain of the other unusual keys. Detecting the arrow keys &arrowdetect; The option to read will not detect the ENTER (newline) key. The option to read permits timed input (see and ). The read command may also read its variable value from a file redirected to stdin. If the file contains more than one line, only the first line is assigned to the variable. If read has more than one parameter, then each of these variables gets assigned a successive whitespace-delineated string. Caution! Using <firstterm>read</firstterm> with <link linkend="ioredirref">file redirection</link> &readredir; Piping output to a read, using echo to set variables will fail. Yet, piping the output of cat seems to work. cat file1 file2 | while read line do echo $line done However, as Bjön Eriksson shows: Problems reading from a pipe &readpipe; The gendiff script, usually found in /usr/bin on many Linux distros, pipes the output of find to a while read construct. find $1 $ -name "*$2" -o -name ".*$2" $ -print | while read f; do . . . It is possible to paste text into the input field of a read (but not multiple lines!). See . <anchor id="intfilesystem1">Filesystem cd cd command cd The familiar cd change directory command finds use in scripts where execution of a command requires being in a specified directory. (cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -) [from the previously cited example by Alan Cox] The (physical) option to cd causes it to ignore symbolic links. cd - changes to $OLDPWD, the previous working directory. The cd command does not function as expected when presented with two forward slashes. bash$ cd // bash$ pwd // The output should, of course, be /. This is a problem both from the command line and in a script. pwd pwd command pwd $PWD variable $PWD directory working Print Working Directory. This gives the user's (or script's) current directory (see ). The effect is identical to reading the value of the builtin variable $PWD. pushd popd dirs pushd command pushd popd command popd dirs command dirs directory working bookmark This command set is a mechanism for bookmarking working directories, a means of moving back and forth through directories in an orderly manner. A pushdown stack is used to keep track of directory names. Options allow various manipulations of the directory stack. pushd dir-name pushes the path dir-name onto the directory stack and simultaneously changes the current working directory to dir-name popd removes (pops) the top directory path name off the directory stack and simultaneously changes the current working directory to that directory popped from the stack. dirs lists the contents of the directory stack (compare this with the $DIRSTACK variable). A successful pushd or popd will automatically invoke dirs. Scripts that require various changes to the current working directory without hard-coding the directory name changes can make good use of these commands. Note that the implicit $DIRSTACK array variable, accessible from within a script, holds the contents of the directory stack. Changing the current working directory &ex37; <anchor id="intvar1">Variables let let command let The let command carries out arithmetic operations on variables. In many cases, it functions as a less complex version of expr. Letting <firstterm>let</firstterm> do arithmetic. &ex46; eval eval command eval eval arg1 [arg2] ... [argN] Combines the arguments in an expression or list of expressions and evaluates them. Any variables contained within the expression are expanded. The result translates into a command. This can be useful for code generation from the command line or within a script. bash$ process=xterm bash$ show_process="eval ps ax | grep $process" bash$ $show_process 1867 tty1 S 0:02 xterm 2779 tty1 S 0:00 xterm 2886 pts/1 S 0:00 grep xterm Each invocation of eval forces a re-evaluation of its arguments. a='$b' b='$c' c=d echo $a # $b # First level. eval echo $a # $c # Second level. eval eval echo $a # d # Third level. # Thank you, E. Choroba. Showing the effect of <firstterm>eval</firstterm> &ex43; <firstterm>Echoing</firstterm> the <firstterm>command-line parameters</firstterm> &echoparams; Forcing a log-off &ex44; A version of <firstterm>rot13</firstterm> &rot14; Rory Winston contributed the following instance of how useful eval can be. Using <firstterm>eval</firstterm> to force variable substitution in a <firstterm>Perl</firstterm> script &evalex; The eval command occurs in the older version of indirect referencing. eval var=\$$var The eval command can be risky, and normally should be avoided when there exists a reasonable alternative. An eval $COMMANDS executes the contents of COMMANDS, which may contain such unpleasant surprises as rm -rf *. Running an eval on unfamiliar code written by persons unknown is living dangerously. set set command set The set command changes the value of internal script variables/options. One use for this is to toggle option flags which help determine the behavior of the script. Another application for it is to reset the positional parameters that a script sees as the result of a command (set `command`). The script can then parse the fields of the command output. Using <firstterm>set</firstterm> with positional parameters &ex34; More fun with positional parameters. Reversing the positional parameters &revposparams; Invoking set without any options or arguments simply lists all the environmental and other variables that have been initialized. bash$ set AUTHORCOPY=/home/bozo/posts BASH=/bin/bash BASH_VERSION=$'2.05.8(1)-release' ... XAUTHORITY=/home/bozo/.Xauthority _=/etc/bashrc variable22=abc variable23=xzy Using set with the option explicitly assigns the contents of a variable to the positional parameters. If no variable follows the it unsets the positional parameters. Reassigning the positional parameters &setpos; See also and . unset unset command unset The unset command deletes a shell variable, effectively setting it to null. Note that this command does not affect positional parameters. bash$ unset PATH bash$ echo $PATH bash$ <quote>Unsetting</quote> a variable &uns; export export command export The export To Export information is to make it available in a more general context. See also scope. command makes available variables to all child processes of the running script or shell. One important use of the export command is in startup files, to initialize and make accessible environmental variables to subsequent user processes. Unfortunately, there is no way to export variables back to the parent process, to the process that called or invoked the script or shell. Using <firstterm>export</firstterm> to pass a variable to an embedded <firstterm>awk</firstterm> script &coltotaler3; It is possible to initialize and export variables in the same operation, as in export var1=xxx. However, as Greg Keraunen points out, in certain situations this may have a different effect than setting a variable, then exporting it. bash$ export var=(a b); echo ${var[0]} (a b) bash$ var=(a b); export var; echo ${var[0]} a declare typeset declare command declare typeset command typeset The declare and typeset commands specify and/or restrict properties of variables. readonly readonly command readonly Same as declare -r, sets a variable as read-only, or, in effect, as a constant. Attempts to change the variable fail with an error message. This is the shell analog of the C language const type qualifier. getopts getopts command getopts $OPTIND variable $OPTIND $OPTARG variable $OPTARG This powerful tool parses command-line arguments passed to the script. This is the Bash analog of the getopt external command and the getopt library function familiar to C programmers. It permits passing and concatenating multiple options An option is an argument that acts as a flag, switching script behaviors on or off. The argument associated with a particular option indicates the behavior that the option (flag) switches on or off. and associated arguments to a script (for example scriptname -abc -e /usr/local). The getopts construct uses two implicit variables. $OPTIND is the argument pointer (OPTion INDex) and $OPTARG (OPTion ARGument) the (optional) argument attached to an option. A colon following the option name in the declaration tags that option as having an associated argument. A getopts construct usually comes packaged in a while loop, which processes the options and arguments one at a time, then increments the implicit $OPTIND variable to step to the next. The arguments passed from the command line to the script must be preceded by a minus (). It is the prefixed that lets getopts recognize command-line arguments as options. In fact, getopts will not process arguments without the prefixed , and will terminate option processing at the first argument encountered lacking them. The getopts template differs slightly from the standard while loop, in that it lacks condition brackets. The getopts construct is a highly functional replacement for the traditional getopt external command. while getopts ":abcde:fg" Option # Initial declaration. # a, b, c, d, e, f, and g are the options (flags) expected. # The : after option 'e' shows it will have an argument passed with it. do case $Option in a ) # Do something with variable 'a'. b ) # Do something with variable 'b'. ... e) # Do something with 'e', and also with $OPTARG, # which is the associated argument passed with option 'e'. ... g ) # Do something with variable 'g'. esac done shift $(($OPTIND - 1)) # Move argument pointer to next. # All this is not nearly as complicated as it looks <grin>. Using <firstterm>getopts</firstterm> to read the options/arguments passed to a script &ex33; <anchor id="intscrbeh1">Script Behavior source . (dot command) source command source . command . This command, when invoked from the command line, executes a script. Within a script, a source file-name loads the file file-name. Sourcing a file (dot-command) imports code into the script, appending to the script (same effect as the #include directive in a C program). The net result is the same as if the sourced lines of code were physically present in the body of the script. This is useful in situations when multiple scripts use a common data file or function library. <quote>Including</quote> a data file &ex38; File data-file for , above. Must be present in same directory. &ex38bis; If the sourced file is itself an executable script, then it will run, then return control to the script that called it. A sourced executable script may use a return for this purpose. Arguments may be (optionally) passed to the sourced file as positional parameters. source $filename $arg1 arg2 It is even possible for a script to source itself, though this does not seem to have any practical applications. A (useless) script that sources itself &selfsource; exit exit command exit Unconditionally terminates a script. Technically, an exit only terminates the process (or shell) in which it is running, not the parent process. The exit command may optionally take an integer argument, which is returned to the shell as the exit status of the script. It is good practice to end all but the simplest scripts with an exit 0, indicating a successful run. If a script terminates with an exit lacking an argument, the exit status of the script is the exit status of the last command executed in the script, not counting the exit. This is equivalent to an exit $?. An exit command may also be used to terminate a subshell. exec exec command exec This shell builtin replaces the current process with a specified command. Normally, when the shell encounters a command, it forks off a child process to actually execute the command. Using the exec builtin, the shell does not fork, and the command exec'ed replaces the shell. When used in a script, therefore, it forces an exit from the script when the exec'ed command terminates. Unless the exec is used to reassign file descriptors. Effects of <firstterm>exec</firstterm> &ex54; A script that <firstterm>exec's</firstterm> itself &selfexec; An exec also serves to reassign file descriptors. For example, exec <zzz-file replaces stdin with the file zzz-file. The option to find is not the same as the exec shell builtin. shopt shopt command shopt This command permits changing shell options on the fly (see and ). It often appears in the Bash startup files, but also has its uses in scripts. Needs version 2 or later of Bash. shopt -s cdspell # Allows minor misspelling of directory names with 'cd' cd /hpme # Oops! Mistyped '/home'. pwd # /home # The shell corrected the misspelling. caller caller command caller Putting a caller command inside a function echoes to stdout information about the caller of that function. #!/bin/bash function1 () { # Inside function1 (). caller 0 # Tell me about it. } function1 # Line 9 of script. # 9 main test.sh # ^ Line number that the function was called from. # ^^^^ Invoked from "main" part of script. # ^^^^^^^ Name of calling script. caller 0 # Has no effect because it's not inside a function. A caller command can also return caller information from a script sourced within another script. Analogous to a function, this is a subroutine call. You may find this command useful in debugging. <anchor id="intcommand1">Commands true true command true A command that returns a successful (zero) exit status, but does nothing else. bash$ true bash$ echo $? 0 # Endless loop while true # alias for ":" do operation-1 operation-2 ... operation-n # Need a way to break out of loop or script will hang. done false false command false A command that returns an unsuccessful exit status, but does nothing else. bash$ false bash$ echo $? 1 # Testing "false" if false then echo "false evaluates \"true\"" else echo "false evaluates \"false\"" fi # false evaluates "false" # Looping while "false" (null loop) while false do # The following code will not execute. operation-1 operation-2 ... operation-n # Nothing happens! done type [cmd] type command type variable which Similar to the which external command, type cmd identifies cmd. Unlike which, type is a Bash builtin. The useful option to type identifies keywords and builtins, and also locates system commands with identical names. bash$ type '[' [ is a shell builtin bash$ type -a '[' [ is a shell builtin [ is /usr/bin/[ bash$ type type type is a shell builtin hash [cmds] hash command hash $PATH variable $PATH Record the path name of specified commands -- in the shell hash table Hashing is a method of creating lookup keys for data stored in a table. The data items themselves are scrambled to create keys, using one of a number of simple mathematical algorithms (methods, or recipes). An advantage of hashing is that it is fast. A disadvantage is that collisions -- where a single key maps to more than one data item -- are possible. For examples of hashing see and . -- so the shell or script will not need to search the $PATH on subsequent calls to those commands. When hash is called with no arguments, it simply lists the commands that have been hashed. The option resets the hash table. bind bind bind key bindings The bind builtin displays or modifies readline The readline library is what Bash uses for reading input in an interactive shell. key bindings. help help command Gets a short usage summary of a shell builtin. This is the counterpart to whatis, but for builtins. bash$ help exit exit: exit [n] Exit the shell with a status of N. If N is omitted, the exit status is that of the last command executed. Job Control Commands Certain of the following job control commands take a job identifier as an argument. See the table at end of the chapter. jobs jobs command jobs ps command ps Lists the jobs running in the background, giving the job number. Not as useful as ps. It is all too easy to confuse jobs and processes. Certain builtins, such as kill, disown, and wait accept either a job number or a process number as an argument. The fg, bg and jobs commands accept only a job number. bash$ sleep 100 & [1] 1384 bash $ jobs [1]+ Running sleep 100 & 1 is the job number (jobs are maintained by the current shell). 1384 is the PID or process ID number (processes are maintained by the system). To kill this job/process, either a kill %1 or a kill 1384 works. Thanks, S.C. disown disown command disown Remove job(s) from the shell's table of active jobs. fg bg fg command foreground background command bg The fg command switches a job running in the background into the foreground. The bg command restarts a suspended job, and runs it in the background. If no job number is specified, then the fg or bg command acts upon the currently running job. wait wait command wait Suspend script execution until all jobs running in background have terminated, or until the job number or process ID specified as an option terminates. Returns the exit status of waited-for command. You may use the wait command to prevent a script from exiting before a background job finishes executing (this would create a dreaded orphan process). Waiting for a process to finish before proceeding &ex39; Optionally, wait can take a job identifier as an argument, for example, wait%1 or wait $PPID. See the job id table. Within a script, running a command in the background with an ampersand (&) may cause the script to hang until ENTER is hit. This seems to occur with commands that write to stdout. It can be a major annoyance. #!/bin/bash # test.sh ls -l & echo "Done." bash$ ./test.sh Done. [bozo@localhost test-scripts]$ total 1 -rwxr-xr-x 1 bozo bozo 34 Oct 11 15:09 test.sh _

As Walter Brameld IV explains it: As far as I can tell, such scripts don't actually hang. It just seems that they do because the background command writes text to the console after the prompt. The user gets the impression that the prompt was never displayed. Here's the sequence of events: 1. Script launches background command. 2. Script exits. 3. Shell displays the prompt. 4. Background command continues running and writing text to the console. 5. Background command finishes. 6. User doesn't see a prompt at the bottom of the output, thinks script is hanging.

Placing a wait after the background command seems to remedy this. #!/bin/bash # test.sh ls -l & echo "Done." wait bash$ ./test.sh Done. [bozo@localhost test-scripts]$ total 1 -rwxr-xr-x 1 bozo bozo 34 Oct 11 15:09 test.sh Redirecting the output of the command to a file or even to /dev/null also takes care of this problem. suspend suspend command suspend This has a similar effect to ControlZ, but it suspends the shell (the shell's parent process should resume it at an appropriate time). logout logout command log out Exit a login shell, optionally specifying an exit status. times times command times Gives statistics on the system time elapsed when executing commands, in the following form: 0m0.020s 0m0.020s This capability is of relatively limited value, since it is not common to profile and benchmark shell scripts. kill kill command kill Forcibly terminate a process by sending it an appropriate terminate signal (see ). A script that kills itself &selfdestruct; kill -l lists all the signals (as does the file /usr/include/asm/signal.h). A kill -9 is a sure kill, which will usually terminate a process that stubbornly refuses to die with a plain kill. Sometimes, a kill -15 works. A zombie process, that is, a child process that has terminated, but that the parent process has not (yet) killed, cannot be killed by a logged-on user -- you can't kill something that is already dead -- but init will generally clean it up sooner or later. killall killall command kill The killall command kills a running process by name, rather than by process ID. If there are multiple instances of a particular command running, then doing a killall on that command will terminate them all. This refers to the killall command in /usr/bin, not the killall script in /etc/rc.d/init.d. command command command command The command directive disables aliases and functions for the command immediately following it. bash$ command ls This is one of three shell directives that effect script command processing. The others are builtin and enable. builtin builtin command builtin Invoking builtin BUILTIN_COMMAND runs the command BUILTIN_COMMAND as a shell builtin, temporarily disabling both functions and external system commands with the same name. enable enable command enable This either enables or disables a shell builtin command. As an example, enable -n kill disables the shell builtin kill, so that when Bash subsequently encounters kill, it invokes the external command /bin/kill. The option to enable lists all the shell builtins, indicating whether or not they are enabled. The option lets enable load a builtin as a shared library (DLL) module from a properly compiled object file. The C source for a number of loadable builtins is typically found in the /usr/share/doc/bash-?.??/functions directory. Note that the option to enable is not portable to all systems. . autoload autoload command autoloader This is a port to Bash of the ksh autoloader. With autoload in place, a function with an autoload declaration will load from an external file at its first invocation. The same effect as autoload can be achieved with typeset -fu. This saves system resources. Note that autoload is not a part of the core Bash installation. It needs to be loaded in with enable -f (see above). Job identifiers Notation Meaning Job number [N] Invocation (command line) of job begins with string S Invocation (command line) of job contains within it string S current job (last job stopped in foreground or started in background) current job (last job stopped in foreground or started in background) Last job Last background process

External Filters, Programs and Commands Standard UNIX commands make shell scripts more versatile. The power of scripts comes from coupling system commands and shell directives with simple programming constructs. Basic Commands <anchor id="basiccommands1">The first commands a novice learns ls ls command ls The basic file list command. It is all too easy to underestimate the power of this humble command. For example, using the , recursive option, ls provides a tree-like listing of a directory structure. Other useful options are , sort listing by file size, , sort by file modification time, , show escape characters, and , show file inodes (see ). The ls command returns a non-zero exit status when attempting to list a non-existent file. bash$ ls abc ls: abc: No such file or directory bash$ echo $? 2 Using <firstterm>ls</firstterm> to create a table of contents for burning a <abbrev>CDR</abbrev> disk &ex40; cat tac cat command cat tac command tac cat, an acronym for concatenate, lists a file to stdout. When combined with redirection (> or >>), it is commonly used to concatenate files. # Uses of 'cat' cat filename # Lists the file. cat file.1 file.2 file.3 > file.123 # Combines three files into one. The option to cat inserts consecutive numbers before all lines of the target file(s). The option numbers only the non-blank lines. The option echoes nonprintable characters, using ^ notation. The option squeezes multiple consecutive blank lines into a single blank line. See also and . In a pipe, it may be more efficient to redirect the stdin to a file, rather than to cat the file. cat filename | tr a-z A-Z tr a-z A-Z < filename # Same effect, but starts one less process, #+ and also dispenses with the pipe. tac, is the inverse of cat, listing a file backwards from its end. rev rev command rev reverses each line of a file, and outputs to stdout. This does not have the same effect as tac, as it preserves the order of the lines, but flips each one around (mirror image). bash$ cat file1.txt This is line 1. This is line 2. bash$ tac file1.txt This is line 2. This is line 1. bash$ rev file1.txt .1 enil si sihT .2 enil si sihT cp cp command cp This is the file copy command. cp file1 file2 copies file1 to file2, overwriting file2 if it already exists (see ). Particularly useful are the archive flag (for copying an entire directory tree), the update flag (which prevents overwriting identically-named newer files), and the and recursive flags. cp -u source_dir/* dest_dir # "Synchronize" dest_dir to source_dir #+ by copying over all newer and not previously existing files. mv This is the file move command. It is equivalent to a combination of cp and rm. It may be used to move multiple files to a directory, or even to rename a directory. For some examples of using mv in a script, see and . When used in a non-interactive script, mv takes the (force) option to bypass user input. When a directory is moved to a preexisting directory, it becomes a subdirectory of the destination directory. bash$ mv source_directory target_directory bash$ ls -lF target_directory total 1 drwxrwxr-x 2 bozo bozo 1024 May 28 19:20 source_directory/ rm rm command rm Delete (remove) a file or files. The option forces removal of even readonly files, and is useful for bypassing user input in a script. The rm command will, by itself, fail to remove filenames beginning with a dash. Why? Because rm sees a dash-prefixed filename as an option. bash$ rm -badname rm: invalid option -- b Try `rm --help' for more information. One clever workaround is to precede the filename with a -- (the end-of-options flag). bash$ rm -- -badname Another method to is to preface the filename to be removed with a dot-slash . bash$ rm ./-badname When used with the recursive flag , this command removes files all the way down the directory tree from the current directory. A careless rm -rf * can wipe out a big chunk of a directory structure. rmdir rmdir command rmdir Remove directory. The directory must be empty of all files -- including invisible dotfiles Dotfiles are files whose names begin with a dot, such as ~/.Xdefaults. Such filenames do not appear in a normal ls listing (although an ls -a will show them), and they cannot be deleted by an accidental rm -rf *. Dotfiles are generally used as setup and configuration files in a user's home directory. -- for this command to succeed. mkdir mkdir command mkdir Make directory, creates a new directory. For example, mkdir -p project/programs/December creates the named directory. The -p option automatically creates any necessary parent directories. chmod chmod command chmod Changes the attributes of an existing file or directory (see ). chmod +x filename # Makes "filename" executable for all users. chmod u+s filename # Sets "suid" bit on "filename" permissions. # An ordinary user may execute "filename" with same privileges as the file's owner. # (This does not apply to shell scripts.) chmod 644 filename # Makes "filename" readable/writable to owner, readable to others # (octal mode). chmod 444 filename # Makes "filename" read-only for all. # Modifying the file (for example, with a text editor) #+ not allowed for a user who does not own the file (except for root), #+ and even the file owner must force a file-save #+ if she modifies the file. # Same restrictions apply for deleting the file. chmod 1777 directory-name # Gives everyone read, write, and execute permission in directory, #+ however also sets the "sticky bit". # This means that only the owner of the directory, #+ owner of the file, and, of course, root #+ can delete any particular file in that directory. chmod 111 directory-name # Gives everyone execute-only permission in a directory. # This means that you can execute and READ the files in that directory #+ (execute permission necessarily includes read permission #+ because you can't execute a file without being able to read it). # But you can't list the files or search for them with the "find" command. # These restrictions do not apply to root. chmod 000 directory-name # No permissions at all for that directory. # Can't read, write, or execute files in it. # Can't even list files in it or "cd" to it. # But, you can rename (mv) the directory #+ or delete it (rmdir) if it is empty. # You can even symlink to files in the directory, #+ but you can't read, write, or execute the symlinks. # These restrictions do not apply to root. chattr chattr command chattr Change file attributes. This is analogous to chmod above, but with different options and a different invocation syntax, and it works only on ext2/ext3 filesystems. One particularly interesting chattr option is . A chattr +i filename marks the file as immutable. The file cannot be modified, linked to, or deleted, not even by root. This file attribute can be set or removed only by root. In a similar fashion, the option marks the file as append only. root# chattr +i file1.txt root# rm file1.txt rm: remove write-protected regular file `file1.txt'? y rm: cannot remove `file1.txt': Operation not permitted If a file has the (secure) attribute set, then when it is deleted its block is zeroed out on the disk. If a file has the (undelete) attribute set, then when it is deleted, its contents can still be retrieved (undeleted). If a file has the (compress) attribute set, then it will automatically be compressed on writes to disk, and uncompressed on reads. The file attributes set with chattr do not show in a file listing (ls -l). ln Creates links to pre-existings files. A link is a reference to a file, an alternate name for it. The ln command permits referencing the linked file by more than one name and is a superior alternative to aliasing (see ). The ln creates only a reference, a pointer to the file only a few bytes in size. The ln command is most often used with the , symbolic or soft link flag. Advantages of using the flag are that it permits linking across file systems or to directories. The syntax of the command is a bit tricky. For example: ln -s oldfile newfile links the previously existing oldfile to the newly created link, newfile. If a file named newfile has previously existed, an error message will result. Which type of link to use? As John Macdonald explains it: Both of these [types of links] provide a certain measure of dual reference -- if you edit the contents of the file using any name, your changes will affect both the original name and either a hard or soft new name. The differences between them occurs when you work at a higher level. The advantage of a hard link is that the new name is totally independent of the old name -- if you remove or rename the old name, that does not affect the hard link, which continues to point to the data while it would leave a soft link hanging pointing to the old name which is no longer there. The advantage of a soft link is that it can refer to a different file system (since it is just a reference to a file name, not to actual data). And, unlike a hard link, a symbolic link can refer to a directory. Links give the ability to invoke a script (or any other type of executable) with multiple names, and having that script behave according to how it was invoked. Hello or Good-bye &hellol; man info man command man info command info These commands access the manual and information pages on system commands and installed utilities. When available, the info pages usually contain more detailed descriptions than do the man pages. There have been various attempts at automating the writing of man pages. For a script that makes a tentative first step in that direction, see . Complex Commands <anchor id="cclisting1">Commands for more advanced users find find command find {} special character {} \; escaped character \; -exec COMMAND \; Carries out COMMAND on each file that find matches. The command sequence terminates with ; (the ; is escaped to make certain the shell passes it to find literally, without interpreting it as a special character). bash$ find ~/ -name '*.txt' /home/bozo/.kde/share/apps/karm/karmdata.txt /home/bozo/misc/irmeyc.txt /home/bozo/test-scripts/1.txt If COMMAND contains {}, then find substitutes the full path name of the selected file for {}. find ~/ -name 'core*' -exec rm {} \; # Removes all core dump files from user's home directory. find /home/bozo/projects -mtime 1 # Lists all files in /home/bozo/projects directory tree #+ that were modified within the last day. # # mtime = last modification time of the target file # ctime = last status change time (via 'chmod' or otherwise) # atime = last access time DIR=/home/bozo/junk_files find "$DIR" -type f -atime +5 -exec rm {} \; # ^^ # Curly brackets are placeholder for the path name output by "find." # # Deletes all files in "/home/bozo/junk_files" #+ that have not been accessed in at least 5 days. # # "-type filetype", where # f = regular file # d = directory # l = symbolic link, etc. # (The 'find' manpage and info page have complete listings.) find /etc -exec grep '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' {} \; # Finds all IP addresses (xxx.xxx.xxx.xxx) in /etc directory files. # There a few extraneous hits. Can they be filtered out? # Possibly by: find /etc -type f -exec cat '{}' \; | tr -c '.[:digit:]' '\n' \ | grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$' # # [:digit:] is one of the character classes #+ introduced with the POSIX 1003.2 standard. # Thanks, Stéphane Chazelas. The option to find should not be confused with the exec shell builtin. <firstterm>Badname</firstterm>, eliminate file names in current directory containing bad characters and <link linkend="whitespaceref">whitespace</link>. &ex57; Deleting a file by its <firstterm>inode</firstterm> number &idelete; The find command also works without the option. #!/bin/bash # Find suid root files. # A strange suid file might indicate a security hole, #+ or even a system intrusion. directory="/usr/sbin" # Might also try /sbin, /bin, /usr/bin, /usr/local/bin, etc. permissions="+4000" # suid root (dangerous!) for file in $( find "$directory" -perm "$permissions" ) do ls -ltF --author "$file" done See , , and for scripts using find. Its manpage provides more detail on this complex and powerful command. xargs xargs command xargs A filter for feeding arguments to a command, and also a tool for assembling the commands themselves. It breaks a data stream into small enough chunks for filters and commands to process. Consider it as a powerful replacement for backquotes. In situations where command substitution fails with a too many arguments error, substituting xargs often works. And even when xargs is not strictly necessary, it can speed up execution of a command involving batch-processing of multiple files. Normally, xargs reads from stdin or from a pipe, but it can also be given the output of a file. The default command for xargs is echo. This means that input piped to xargs may have linefeeds and other whitespace characters stripped out. bash$ ls -l total 0 -rw-rw-r-- 1 bozo bozo 0 Jan 29 23:58 file1 -rw-rw-r-- 1 bozo bozo 0 Jan 29 23:58 file2 bash$ ls -l | xargs total 0 -rw-rw-r-- 1 bozo bozo 0 Jan 29 23:58 file1 -rw-rw-r-- 1 bozo bozo 0 Jan... bash$ find ~/mail -type f | xargs grep "Linux" ./misc:User-Agent: slrn/0.9.8.1 (Linux) ./sent-mail-jul-2005: hosted by the Linux Documentation Project. ./sent-mail-jul-2005: (Linux Documentation Project Site, rtf version) ./sent-mail-jul-2005: Subject: Criticism of Bozo's Windows/Linux article ./sent-mail-jul-2005: while mentioning that the Linux ext2/ext3 filesystem . . . ls | xargs -p -l gzip gzips every file in current directory, one at a time, prompting before each operation. Note that xargs processes the arguments passed to it sequentially, one at a time. bash$ find /usr/bin | xargs file /usr/bin: directory /usr/bin/foomatic-ppd-options: perl script text executable . . . An interesting xargs option is , which limits to NN the number of arguments passed. ls | xargs -n 8 echo lists the files in the current directory in 8 columns. Another useful option is , in combination with find -print0 or grep -lZ. This allows handling arguments containing whitespace or quotes. find / -type f -print0 | xargs -0 grep -liwZ GUI | xargs -0 rm -f grep -rliwZ GUI / | xargs -0 rm -f Either of the above will remove any file containing GUI. (Thanks, S.C.) Or: cat /proc/"$pid"/"$OPTION" | xargs -0 echo # Formats output: ^^^^^^^^^^^^^^^ # From Han Holl's fixup of "get-commandline.sh" #+ script in "/dev and /proc" chapter. Logfile: Using <firstterm>xargs</firstterm> to monitor system log &ex41; As in find, a curly bracket pair serves as a placeholder for replacement text. Copying files in current directory to another &ex42; Killing processes by name &killbyname; Word frequency analysis using <firstterm>xargs</firstterm> &wf2; expr expr command expr All-purpose expression evaluator: Concatenates and evaluates the arguments according to the operation given (arguments must be separated by spaces). Operations may be arithmetic, comparison, string, or logical. expr 3 + 5 returns 8 expr 5 % 3 returns 2 expr 1 / 0 returns the error message, expr: division by zero Illegal arithmetic operations not allowed. expr 5 \* 3 returns 15 The multiplication operator must be escaped when used in an arithmetic expression with expr. y=`expr $y + 1` Increment a variable, with the same effect as let y=y+1 and y=$(($y+1)). This is an example of arithmetic expansion. z=`expr substr $string $position $length` Extract substring of $length characters, starting at $position. Using <firstterm>expr</firstterm> &ex45; The : (null) operator can substitute for match. For example, b=`expr $a : [0-9]*` is the exact equivalent of b=`expr match $a [0-9]*` in the above listing. &ex45a; The above script illustrates how expr uses the escaped parentheses -- $ ... $ -- grouping operator in tandem with regular expression parsing to match a substring. Here is a another example, this time from real life. # Strip the whitespace from the beginning and end. LRFDATE=`expr "$LRFDATE" : '[[:space:]]*$.*$[[:space:]]*$'` # From Peter Knowles' "booklistgen.sh" script #+ for converting files to Sony Librie/PRS-50X format. # (http://booklistgensh.peterknowles.com) Perl, sed, and awk have far superior string parsing facilities. A short sed or awk subroutine within a script (see ) is an attractive alternative to expr. See for more on using expr in string operations. Time / Date Commands <anchor id="tdlisting1">Time/date and timing date date command date Simply invoked, date prints the date and time to stdout. Where this command gets interesting is in its formatting and parsing options. Using <firstterm>date</firstterm> &ex51; The option gives the UTC (Universal Coordinated Time). bash$ date Fri Mar 29 21:07:39 MST 2002 bash$ date -u Sat Mar 30 04:07:42 UTC 2002 This option facilitates calculating the time between different dates. <firstterm>Date</firstterm> calculations &datecalc; The date command has quite a number of output options. For example gives the nanosecond portion of the current time. One interesting use for this is to generate random integers. date +%N | sed -e 's/000$//' -e 's/^0//' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # Strip off leading and trailing zeroes, if present. # Length of generated integer depends on #+ how many zeroes stripped off. # 115281032 # 63408725 # 394504284 There are many more options (try man date). date +%j # Echoes day of the year (days elapsed since January 1). date +%k%M # Echoes hour and minute in 24-hour format, as a single digit string. # The 'TZ' parameter permits overriding the default time zone. date # Mon Mar 28 21:42:16 MST 2005 TZ=EST date # Mon Mar 28 23:42:16 EST 2005 # Thanks, Frank Kannemann and Pete Sjoberg, for the tip. SixDaysAgo=$(date --date='6 days ago') OneMonthAgo=$(date --date='1 month ago') # Four weeks back (not a month!) OneYearAgo=$(date --date='1 year ago') See also and . zdump zdump command time zone dump Time zone dump: echoes the time in a specified time zone. bash$ zdump EST EST Tue Sep 18 22:09:22 2001 EST time time command time Outputs verbose timing statistics for executing a command. time ls -l / gives something like this: real 0m0.067s user 0m0.004s sys 0m0.005s See also the very similar times command in the previous section. As of version 2.0 of Bash, time became a shell reserved word, with slightly altered behavior in a pipeline. touch touch command touch Utility for updating access/modification times of a file to current system time or other specified time, but also useful for creating a new file. The command touch zzz will create a new file of zero length, named zzz, assuming that zzz did not previously exist. Time-stamping empty files in this way is useful for storing date information, for example in keeping track of modification times on a project. The touch command is equivalent to : >> newfile or >> newfile (for ordinary files). Before doing a cp -u (copy/update), use touch to update the time stamp of files you don't wish overwritten. As an example, if the directory /home/bozo/tax_audit contains the files spreadsheet-051606.data, spreadsheet-051706.data, and spreadsheet-051806.data, then doing a touch spreadsheet*.data will protect these files from being overwritten by files with the same names during a cp -u /home/bozo/financial_info/spreadsheet*data /home/bozo/tax_audit. at at command at cron command cron The at job control command executes a given set of commands at a specified time. Superficially, it resembles cron, however, at is chiefly useful for one-time execution of a command set. at 2pm January 15 prompts for a set of commands to execute at that time. These commands should be shell-script compatible, since, for all practical purposes, the user is typing in an executable shell script a line at a time. Input terminates with a Ctl-D. Using either the option or input redirection (<), at reads a command list from a file. This file is an executable shell script, though it should, of course, be non-interactive. Particularly clever is including the run-parts command in the file to execute a different set of scripts. bash$ at 2:30 am Friday < at-jobs.list job 2 at 2000-10-27 02:30 batch batch command batch at command at The batch job control command is similar to at, but it runs a command list when the system load drops below .8. Like at, it can read commands from a file with the option. The concept of batch processing dates back to the era of mainframe computers. It means running a set of commands without user intervention. cal cal command cal Prints a neatly formatted monthly calendar to stdout. Will do current year or a large range of past and future years. sleep sleep command sleep This is the shell equivalent of a wait loop. It pauses for a specified number of seconds, doing nothing. It can be useful for timing or in processes running in the background, checking for a specific event every so often (polling), as in . sleep 3 # Pauses 3 seconds. The sleep command defaults to seconds, but minute, hours, or days may also be specified. sleep 3 h # Pauses 3 hours! The watch command may be a better choice than sleep for running commands at timed intervals. usleep usleep command usleep Microsleep (the u may be read as the Greek mu, or micro- prefix). This is the same as sleep, above, but sleeps in microsecond intervals. It can be used for fine-grained timing, or for polling an ongoing process at very frequent intervals. usleep 30 # Pauses 30 microseconds. This command is part of the Red Hat initscripts / rc-scripts package. The usleep command does not provide particularly accurate timing, and is therefore unsuitable for critical timing loops. hwclock clock hwclock command hwclock clock command clock The hwclock command accesses or adjusts the machine's hardware clock. Some options require root privileges. The /etc/rc.d/rc.sysinit startup file uses hwclock to set the system time from the hardware clock at bootup. The clock command is a synonym for hwclock. Text Processing Commands <anchor id="tpcommandlisting1">Commands affecting text and text files sort sort command sort File sort utility, often used as a filter in a pipe. This command sorts a text stream or file forwards or backwards, or according to various keys or character positions. Using the option, it merges presorted input files. The info page lists its many capabilities and options. See , , and . tsort tsort command topological sort Topological sort, reading in pairs of whitespace-separated strings and sorting according to input patterns. The original purpose of tsort was to sort a list of dependencies for an obsolete version of the ld linker in an ancient version of UNIX. The results of a tsort will usually differ markedly from those of the standard sort command, above. uniq uniq command uniq This filter removes duplicate lines from a sorted file. It is often seen in a pipe coupled with sort. cat list-1 list-2 list-3 | sort | uniq > final.list # Concatenates the list files, # sorts them, # removes duplicate lines, # and finally writes the result to an output file. The useful option prefixes each line of the input file with its number of occurrences. bash$ cat testfile This line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times. bash$ uniq -c testfile 1 This line occurs only once. 2 This line occurs twice. 3 This line occurs three times. bash$ sort testfile | uniq -c | sort -nr 3 This line occurs three times. 2 This line occurs twice. 1 This line occurs only once. The sort INPUTFILE | uniq -c | sort -nr command string produces a frequency of occurrence listing on the INPUTFILE file (the options to sort cause a reverse numerical sort). This template finds use in analysis of log files and dictionary lists, and wherever the lexical structure of a document needs to be examined. Word Frequency Analysis &wf; bash$ cat testfile This line occurs only once. This line occurs twice. This line occurs twice. This line occurs three times. This line occurs three times. This line occurs three times. bash$ ./wf.sh testfile 6 this 6 occurs 6 line 3 times 3 three 2 twice 1 only 1 once expand unexpand expand command expand unexpand command unexpand The expand filter converts tabs to spaces. It is often used in a pipe. The unexpand filter converts spaces to tabs. This reverses the effect of expand. cut cut command cut awk command awk A tool for extracting fields from files. It is similar to the print $N command set in awk, but more limited. It may be simpler to use cut in a script than awk. Particularly important are the (delimiter) and (field specifier) options. Using cut to obtain a listing of the mounted filesystems: cut -d ' ' -f1,2 /etc/mtab Using cut to list the OS and kernel version: uname -a | cut -d" " -f1,3,11,12 Using cut to extract message headers from an e-mail folder: bash$ grep '^Subject:' read-messages | cut -c10-80 Re: Linux suitable for mission-critical apps? MAKE MILLIONS WORKING AT HOME!!! Spam complaint Re: Spam complaint Using cut to parse a file: # List all the users in /etc/passwd. FILENAME=/etc/passwd for user in $(cut -d: -f1 $FILENAME) do echo $user done # Thanks, Oleg Philon for suggesting this. cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{ print $2, $3 }' filename It is even possible to specify a linefeed as a delimiter. The trick is to actually embed a linefeed (RETURN) in the command sequence. bash$ cut -d' ' -f3,7,19 testfile This is line 3 of testfile. This is line 7 of testfile. This is line 19 of testfile. Thank you, Jaka Kranjc, for pointing this out. See also . paste paste command paste cut command cut Tool for merging together different files into a single, multi-column file. In combination with cut, useful for creating system log files. join join command join Consider this a special-purpose cousin of paste. This powerful utility allows merging two files in a meaningful fashion, which essentially creates a simple version of a relational database. The join command operates on exactly two files, but pastes together only those lines with a common tagged field (usually a numerical label), and writes the result to stdout. The files to be joined should be sorted according to the tagged field for the matchups to work properly. File: 1.data 100 Shoes 200 Laces 300 Socks File: 2.data 100 $40.00 200 $1.00 300 $2.00 bash$ join 1.data 2.data File: 1.data 2.data 100 Shoes $40.00 200 Laces $1.00 300 Socks $2.00 The tagged field appears only once in the output. head head command head lists the beginning of a file to stdout. The default is 10 lines, but a different number can be specified. The command has a number of interesting options. Which files are scripts? &scriptdetector; Generating 10-digit random numbers &rnd; See also . tail tail command tail lists the (tail) end of a file to stdout. The default is 10 lines, but this can be changed with the option. Commonly used to keep track of changes to a system logfile, using the option, which outputs lines appended to the file. Using <firstterm>tail</firstterm> to monitor the system log &ex12; To list a specific line of a text file, pipe the output of head to tail -n 1. For example head -n 8 database.txt | tail -n 1 lists the 8th line of the file database.txt. To set a variable to a given block of a text file: var=$(head -n $m $filename | tail -n $n) # filename = name of file # m = from beginning of file, number of lines to end of block # n = number of lines to set variable to (trim from end of block) Newer implementations of tail deprecate the older tail -$LINES filename usage. The standard tail -n $LINES filename is correct. See also , and . grep grep command grep A multi-purpose file search tool that uses Regular Expressions. It was originally a command/filter in the venerable ed line editor: g/re/p -- global - regular expression - print. grep pattern file Search the target file(s) for occurrences of pattern, where pattern may be literal text or a Regular Expression. bash$ grep '[rst]ystem.$' osinfo.txt The GPL governs the distribution of the Linux operating system. If no target file(s) specified, grep works as a filter on stdout, as in a pipe. bash$ ps ax | grep clock 765 tty1 S 0:00 xclock 901 pts/1 S 0:00 grep clock The option causes a case-insensitive search. The option matches only whole words. The option lists only the files in which matches were found, but not the matching lines. The (recursive) option searches files in the current working directory and all subdirectories below it. The option lists the matching lines, together with line numbers. bash$ grep -n Linux osinfo.txt 2:This is a file containing information about Linux. 6:The GPL governs the distribution of the Linux operating system. The (or ) option filters out matches. grep pattern1 *.txt | grep -v pattern2 # Matches all lines in "*.txt" files containing "pattern1", # but ***not*** "pattern2". The () option gives a numerical count of matches, rather than actually listing the matches. grep -c txt *.sgml # (number of occurrences of "txt" in "*.sgml" files) # grep -cz . # ^ dot # means count (-c) zero-separated (-z) items matching "." # that is, non-empty ones (containing at least 1 character). # printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz . # 3 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '$' # 5 printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '^' # 5 # printf 'a b\nc d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -c '$' # 9 # By default, newline chars (\n) separate items to match. # Note that the -z option is GNU "grep" specific. # Thanks, S.C. The (or ) option marks the matching string in color (on the console or in an xterm window). Since grep prints out each entire line containing the matching pattern, this lets you see exactly what is being matched. See also the option, which shows only the matching portion of the line(s). Printing out the <firstterm>From</firstterm> lines in stored e-mail messages &fromsh; When invoked with more than one target file given, grep specifies which file contains matches. bash$ grep Linux osinfo.txt misc.txt osinfo.txt:This is a file containing information about Linux. osinfo.txt:The GPL governs the distribution of the Linux operating system. misc.txt:The Linux operating system is steadily gaining in popularity. To force grep to show the filename when searching only one target file, simply give /dev/null as the second file. bash$ grep Linux osinfo.txt /dev/null osinfo.txt:This is a file containing information about Linux. osinfo.txt:The GPL governs the distribution of the Linux operating system. If there is a successful match, grep returns an exit status of 0, which makes it useful in a condition test in a script, especially in combination with the option to suppress output. SUCCESS=0 # if grep lookup succeeds word=Linux filename=data.file grep -q "$word" "$filename" # The "-q" option #+ causes nothing to echo to stdout. if [ $? -eq $SUCCESS ] # if grep -q "$word" "$filename" can replace lines 5 - 7. then echo "$word found in $filename" else echo "$word not found in $filename" fi demonstrates how to use grep to search for a word pattern in a system logfile. Emulating <firstterm>grep</firstterm> in a script &grp; How can grep search for two (or more) separate patterns? What if you want grep to display all lines in a file or files that contain both pattern1 and pattern2? One method is to pipe the result of grep pattern1 to grep pattern2. For example, given the following file: # Filename: tstfile This is a sample file. This is an ordinary text file. This file does not contain any unusual text. This file is not unusual. Here is some text. Now, let's search this file for lines containing both file and text . . . bash$ grep file tstfile # Filename: tstfile This is a sample file. This is an ordinary text file. This file does not contain any unusual text. This file is not unusual. bash$ grep file tstfile | grep text This is an ordinary text file. This file does not contain any unusual text. Now, for an interesting recreational use of grep . . . Crossword puzzle solver &cwsolver; egrep -- extended grep -- is the same as grep -E. This uses a somewhat different, extended set of Regular Expressions, which can make the search a bit more flexible. It also allows the boolean | (or) operator. bash $ egrep 'matches|Matches' file.txt Line 1 matches. Line 3 Matches. Line 4 contains matches, but also Matches fgrep -- fast grep -- is the same as grep -F. It does a literal string search (no Regular Expressions), which generally speeds things up a bit. On some Linux distros, egrep and fgrep are symbolic links to, or aliases for grep, but invoked with the and options, respectively. Looking up definitions in <citetitle pubwork="book">Webster's 1913 Dictionary</citetitle> &dictlookup; See also for an example of speedy fgrep lookup on a large text file. agrep (approximate grep) extends the capabilities of grep to approximate matching. The search string may differ by a specified number of characters from the resulting matches. This utility is not part of the core Linux distribution. To search compressed files, use zgrep, zegrep, or zfgrep. These also work on non-compressed files, though slower than plain grep, egrep, fgrep. They are handy for searching through a mixed set of files, some compressed, some not. To search bzipped files, use bzgrep. look look command look The command look works like grep, but does a lookup on a dictionary, a sorted word list. By default, look searches for a match in /usr/dict/words, but a different dictionary file may be specified. Checking words in a list for validity &lookup; sed awk sed command sed awk command awk Scripting languages especially suited for parsing text files and command output. May be embedded singly or in combination in pipes and shell scripts. sed Non-interactive stream editor, permits using many ex commands in batch mode. It finds many uses in shell scripts. awk Programmable file extractor and formatter, good for manipulating and/or extracting fields (columns) in structured text files. Its syntax is similar to C. wc wc command wc wc gives a word count on a file or I/O stream: bash $ wc /usr/share/doc/sed-4.1.2/README 13 70 447 README [13 lines 70 words 447 characters] wc -w gives only the word count. wc -l gives only the line count. wc -c gives only the byte count. wc -m gives only the character count. wc -L gives only the length of the longest line. Using wc to count how many .txt files are in current working directory: $ ls *.txt | wc -l # Will work as long as none of the "*.txt" files #+ have a linefeed embedded in their name. # Alternative ways of doing this are: # find . -maxdepth 1 -name \*.txt -print0 | grep -cz . # (shopt -s nullglob; set -- *.txt; echo $#) # Thanks, S.C. Using wc to total up the size of all the files whose names begin with letters in the range d - h bash$ wc [d-h]* | grep total | awk '{print $3}' 71832 Using wc to count the instances of the word Linux in the main source file for this book. bash$ grep Linux abs-book.sgml | wc -l 50 See also and . Certain commands include some of the functionality of wc as options. ... | grep foo | wc -l # This frequently used construct can be more concisely rendered. ... | grep -c foo # Just use the "-c" (or "--count") option of grep. # Thanks, S.C. tr tr command tr character translation filter. Must use quoting and/or brackets, as appropriate. Quotes prevent the shell from reinterpreting the special characters in tr command sequences. Brackets should be quoted to prevent expansion by the shell. Either tr "A-Z" "*" <filename or tr A-Z \* <filename changes all the uppercase letters in filename to asterisks (writes to stdout). On some systems this may not work, but tr A-Z '[**]' will. The option deletes a range of characters. echo "abcdef" # abcdef echo "abcdef" | tr -d b-d # aef tr -d 0-9 <filename # Deletes all digits from the file "filename". The (or ) option deletes all but the first instance of a string of consecutive characters. This option is useful for removing excess whitespace. bash$ echo "XXXXX" | tr --squeeze-repeats 'X' X The complement option inverts the character set to match. With this option, tr acts only upon those characters not matching the specified set. bash$ echo "acfdeb123" | tr -c b-d + +c+d+b++++ Note that tr recognizes POSIX character classes. This is only true of the GNU version of tr, not the generic version often fo