Discuss for Deep Insight: Matching a pattern in a file and Deleting a set of lines before that pattern using SED and AWK

Consider a scenario where we are having a huge number of lines in a text file. We want to do some pattern matching and need to delete that pattern matched line as well as some lines immediately preceding that line. In this case we can follow the below steps to achieve that using SED & AWK shell commands.

Step 1: Reverse the order of the lines of the file.

This reversal is required as we are going to use the “sed” to do pattern matching and deletion of lines. As “sed” doesn’t back track, once a line is processed it is done. We cannot delete the lines preceding the pattern matched line. By reversing the order of the lines, we can easily delete the lines following the pattern matched line instead of preceding lines.

Step 2: Use the “sed” to do pattern matching and to delete the required lines.

In this operation, for each line deleted a blank line will get generated.

Step 3: Using “awk” we can remove the empty lines introduced due to the operation performed in step 2.

Step 4: Again reverse the order of the lines of the file so that we can get the original file with unwanted lines removed.

Steps	Command	Purpose of command
Step 1	tac input_file > output_file1	Reversing the order of line in input file and redirecting the output to another file “output_file1”
Step 2	sed –e ‘/<pattern_to_match>/,+nd’ output_file1 > output_file2 Where n = number of lines to delete following the pattern matched line.	This command will delete the pattern matched line and ‘n’ number of lines following that in the file “output_file1”. The output is redirected to the file “output_file2”
Step 3	awk NF output_file2 > output_file3	This command will remove the blank lines from “output_file2” and output is getting redirected to “output_file3”
Step 4	tac output_file3 > output_file4	Reversing the order of lines in the file “output_file3” so that we can get the original file “input_file” with unwanted lines removed.

Illustrative Example:

Input File:

Objective: In this file, we need to remove the commands which are getting failed along with its failure message.

line 1

mac-address-table static unicast 00:00:00:00:08:01 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:02 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:03 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:04 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:05 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:06 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:07 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:00:08:08 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:01:01:01 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:02 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:03 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:04 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:05 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:06 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:01:07 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:02:04 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:02:05 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:02:06 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:02:07 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:02:08 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:03:08 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:04:00 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:01:04:01 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:01:04:02 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:01:04:03 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:01:06:00 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:06:01 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:06:02 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:01:06:03 vlan 10 interface extreme-ethernet 0/10

% Failed to add the MAC entry in the hardware

mac-address-table static unicast 00:00:00:02:00:00 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:02:00:01 vlan 10 interface extreme-ethernet 0/10

mac-address-table static unicast 00:00:00:02:00:02 vlan 10 interface extreme-ethernet 0/10

line 101

Line Order Reversed File:

Command: tac input_file > output_file1