3.3.3. Inline Filtering

3.3.3.1. sed

Linux for Programmers and Users, Section 4.12.2.

sed

A programmable stream editor for filtering and transforming text.

SYNOPSIS

sed [-e script] [-f scriptfile] [FILE …]
-e script, --expression=script

add the script with the commands to be executed. Note: When only one sed script is used, the -e option is not needed; however, when more than one sed script is needed, use the -e before each script.

-f script-file, --file=script-file

add the contents of script-file with the commands to be executed

-i[SUFFIX], --in-place[=SUFFIX]

edit files in place (makes backup if extension supplied)

The sed command is often used as a one line script, but may also be invoked with a file containing a multi-line script – much like a programming language.

3.3.3.2. Sed Script Commands

The general format of sed commands is:

[address-spec]  command

Commands that operate on whole lines (insert or delete lines) require an address (line) specification. The ‘s’ command, which substitutes text with a line, may either operate on all of the lines processed or use an optional address (line) specification.

/regex/
Apply the command to any line matching the regular expression.
n
Apply the command to line (number) n in the data.
start, stop
Apply the command between lines start and stop in the data.

The text editing commands are as follows.

address a\text
Append line(s) of text after address
address c\text
Replace line(s) of text
address d
Delete line(s)
address i\text
Insert line(s) of text before address
address r filename
Append a file after the address. Note: use cat to insert a file before the first line.
[address] s/regex/string/
Replace the first instance of text matching the regular expression on the applied lines with the string.
[address] s/regex/string/g
Replace all instances of text matching the regular expression on the applied lines with the string.

Note

One common usage of sed is to generate edited string variable names in a shell script program. For example:

newname=$(echo $oldname | sed 's/regex/str/')

See the text book for more examples.

3.3.3.3. tr

Linux for Programmers and Users, Section 4.12.3.

tr

Replace characters from one sequence of characters with characters from another sequence. The most common usage of tr is to convert text data to either all capital letters or all lower case letters.

SYNOPSIS

tr [-cds] sequence1 sequence2
-c, -C, --complement

first complement SET1

-d, --delete

delete characters in SET1, do not translate

-s, --squeeze-repeats

replace each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character

Examples

Convert all letters in a string variable to lower case letters:

input=$(echo $input | tr A-Z a-z)

tr has some predefined characters sets that can be useful. See the man page for the full list. One of them is [:space:], which we can use to remove any white space or line feeds from a string that was read from a file. We also need some features of echo to make this work. This accomplishes the same thing as the strip() function in Python and chomp() function in Perl.

line=$(echo -ne "${line}" | tr -d '[[:space:]]')