Introduction to Unix

3.3.1. Filtering Horizontally

Linux for Programmers and Users, Section 4.1

Commands that filter data horizontally pass a subset of the lines processed to standard output.

3.3.1.1. grep

grep

Print selected parts of lines from each FILE to standard output. The selection is based on matching of PATTERN in the line of input data. grep can also read data from stdin (usually via a pipe).

SYNOPSIS

grep [OPTIONS] PATTERN [FILE…]

-i, --ignore-case

Ignore case distinctions in both the PATTERN and the input files.

-v, --invert-match

Invert the sense of matching, to select non-matching lines.

3.3.1.2. egrep, fgrep

fgrep

Fixed string only

grep

Regular expressions

egrep

Extended regular expressions

3.3.1.3. uniq

uniq

Remove repeating lines. The uniq command is often used in combination with sort to remove duplicate items.

SYNOPSIS

uniq [OPTIONS] [INPUT [OUTPUT]]

-f, --skip-fields=N

avoid comparing the first N fields. A field is a run of whitespace, then non-whitespace characters.

$ cat sample
This is a test file for the uniq command.
It contains some repeated and some nonrepeated lines.
Some of the repeated lines are consecutive, like this.
Some of the repeated lines are consecutive, like this.
Some of the repeated lines are consecutive, like this.
And, some are not consecutive, like the following.
Some of the repeated lines are consecutive, like this.
The above line, therefore, will not be considered a repeated
line by the uniq command, but this will be considered repeated!
line by the uniq command, but this will be considered repeated!
$ uniq sample
This is a test file for the uniq command.
It contains some repeated and some nonrepeated lines.
Some of the repeated lines are consecutive, like this.
And, some are not consecutive, like the following.
Some of the repeated lines are consecutive, like this.
The above line, therefore, will not be considered a repeated
line by the uniq command, but this will be considered repeated!
$

3.3.1.4. sort

sort

sort lines of textual data

SYNOPSIS

sort [OPTION] [FILE]

-f, --ignore-case

fold lower case to upper case characters

-g, --general-numeric-sort

compare according to general numerical value

-k, --key=POS1[,POS2]

start a key at POS1, end it at POS 2 (origin 1)

-t, --field-separator=SEP

use SEP instead of non-blank to blank transition

The following example sorts data from the system’s database of users according to their UID (third field) of colon (:) separated fields. Note the data is sort numerically rather than alphabetically by using the -g option.

$ getent passwd | sort -gt: -k 3,4

3.3.1.5. awk

See Horizontal Filtering With Awk

Note

If awk is to be considered for filtering data, so should perl and python.