Introduction to Unix

3.3. Filtering Data

Linux for Programmers and Users, Section 4.2 - 4.3.

We use commands that filter data to select only the portion of data that we wish to view or operate on. Filtering commands are usually run with either a filename as a command line argument or they read data from stdin, usually from a pipe. Some programs, such as awk and sed, which are really programming languages, can filter data horizontally, vertically or inline, while other tools filter data either horizontally or vertically.

---

The Pipe (|)

A fundamental concept that will be used in filtering data and for the rest of the semester is that of the pipe. A pipe is a facility of the shell that makes it very to chain together multiple commands. A is called upon with the vertical bar character (|). When used between two Unix commands, it means that output from the first command should become the input to the second command. For example, to count how many files underneath a directory have been modified in the last day, the find and wc commands may be used along with a pipe (Finding Files, Counting with wc). The find command will list the modified files and wc can count them:

find . -type f -mtime -1 | wc -l

---

Contents

• 3.3.1. Filtering Horizontally
  • 3.3.1.1. grep
  • 3.3.1.2. egrep, fgrep
  • 3.3.1.3. uniq
  • 3.3.1.4. sort
  • 3.3.1.5. awk
• 3.3.2. Filtering Vertically
  • 3.3.2.1. cut
  • 3.3.2.2. awk
  • 3.3.2.3. paste
• 3.3.3. Inline Filtering
  • 3.3.3.1. sed
  • 3.3.3.2. Sed Script Commands
  • 3.3.3.3. tr
• 3.3.4. awk
  • 3.3.4.1. Vertical Filtering With Awk
  • 3.3.4.2. Horizontal Filtering With Awk