Text Substitution with the sed command in Linux

Introduction

sed is a powerful tool often used in the Linux terminal to edit text documents without opening them. The word "sed" is a contraction of the phrase "Stream Editor". It is essentially a tool for editing streams of text. sed takes in text from a file or text received from standard input (stdin) and lets you perform some operations on it. In this tutorial, I will demonstrate how sed can be used to make string substitutions in text files in the Linux command-line environment.

Creating a File

First, open your Linux terminal and navigate to your home directory if you aren't there already. Create a new file with the cat command by following the succeeding code block:

cat << EOF > myfile.txt
This is my box.
It is mine.
My box is big, it is a big box.
My box is black, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbox to play with my box.
EOF

This is the sample file which we will perform most of our sed operations on. In the next section, you will begin to see how the sed command is fundamentally used to alter and substitute text.

The Basic Syntax of sed

The most common way in which you will use sed is this:

sed 's/OLD-WORD/NEW-WORD/' FILENAME

You start with typing the sed keyword which lets the computer know you want to access the stream editor tool.

s is the search operator that looks through the file.

Next, you enter the word you want to substitute out or replace which is the OLD-WORD, and the word you want to substitute in which is the NEW-WORD.

Then you enter the name of the target file FILENAME you wish to perform this sed operation on.

The forward slash / which separates each keyword is called a delimiter, more on delimiters will be explained later.

Now, using the sed command in the file you created earlier, myfile.txt, you will attempt to replace instances of the old word box with a new word bag.

Enter the following command and press ENTER:

sed 's/box/bag/' myfile.txt

Immediately, the following output is displayed:

This is my bag.
It is mine.
My bag is big, it is a big box.
My bag is black, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbag to play with my box.

You may notice that the word sandbox has been altered to sandbag. This occurs because the sed command acts on text patterns it encounters, not simply the standalone word.

You may also observe that on each line, only the first instance of box is changed to bag. The rest are left unchanged. You can look at the highlighted words in the generated output to observe this.

Note: The file myfile.txt itself remains unchanged. By default the sed command prints the result of its operation to standard output (stdout). More on how to permanently save your edits will be covered later.

Substituting Specific Instances

By default, only the first instance of the specified text is altered. If you wish to change the order of the affected instance, you type in the following:

sed 's/box/bag/2' myfile.txt

The following output is displayed in your terminal upon execution:

This is my box.
It is mine.
My box is big, it is a big bag.
My box is black, my bag is strong, my box is the best.
My mother bough it for me.
Let's go to the sandbox to play with my bag.

In the command entered, you'll notice that we've added a number 2. What this does is, it makes it so that on each line where box appears, it isn't the first instance that is altered, but rather, the second instance. Looking at the generated output, you will be able to observe this.

You may replace 2 with any number to denote which instance of the old word you wish to substitute out for your new word.

Substituting All instances

If you wish for every instance of the old word box in the file to be replaced with the new word bag, enter the following command:

sed 's/box/bag/g' myfile.txt

You should see this output:

This is my bag.
It is mine.
My bag is big, it is a big bag.
My bag is black, my bag is strong, my bag is the best.
My mother bought it for me.
Let's go to the sandbag to play with my bag.

The g flag added indictates you want to make the substitution globally, i.e everywhere the old word occurs in the file.

Navigating Case-Insensitive Substitutions

The sed command is by default case-sensitive. Using the i flag in your sed operation helps you make case-insensitive substitutions. This is for situations where you wish to find and replace instances of the old word regardless of the letter case they may be in. Below is an example:

sed 's/BOX/bag/i' myfile.txt

The output:

This is my bag.
It is mine.
My bag is big, it is a big box.
My bag is black, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbag to play with my box.

You'll observe that even though BOX is used in the command, instances of its lower-case, box, in the file are altered. The substitution wouldn't work without the i flag, you can try this out in your terminal environment to see.

In the next section, we will see how to make changes on designated lines.

Substituting Text on Specific Lines

To find and replace a text pattern on a specific line, instead of all the lines in the file, use the following command:

sed '3 s/box/bag' myfile.txt

Generated output:

This is my box.
It is mine.
My bag is big, it is a big box.
My box is black, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbox to play with my box.

In the preceding generated output, you will observe that only the first instance of box on the third line has been substituted out. Placing the number 3 before s tells the computer that in myfile.txt you want to find and replace the word box for bag in the third line only.

Also, you can change that number to whatever line number you want your substitution made instead.

Note: If you have a large file, to display the line numbers, enter this command:
sed = myfile.txt

Substituting Text in a Range of Lines

To find and replace text in a range of lines:

sed '1,3 s/box/bag/' myfile.txt

The displayed output

This is my bag.
It is mine.
My bag is big, it is a big box.
My box is black, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbox to play with my box

In the sed command entered, the first number is followed by the second number and a comma separates them. This command lets the computer know that you want your sed substitutions to be made within the range of line 1 to line 3.

In the next section, we will experiment with different ways of how to execute multiple sed substitutions at a go.

Making Multiple sed Substitutions

At times, you may wish to use the sed command multiple times on the same text file, there are various ways for you to execute this in one command entry in the Linux terminal.

To demonstrate this, you will substitute the words box with bag and black with red in the text file we created, myfile.txt. This can be done using any of the following methods:

1. Using Pipe |

Enter in the following command:

sed 's/box/bag/' myfile.txt | sed 's/black/red/'

The output:

This is my bag.
It is mine.
My bag is big, it is a big box.
My bag is red, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbag to play with my box

Remember, sed is a Stream Editor. It prints out to standard output(stdout) and can also take in text piped in from standard input(stdin). In the above scenario, the output from the first sed substitution is piped in to the next sed command to make its own substitutions too. This lets us make multiple sed substitutions at once.

2. Using the -e Option

You may use the -e option provided by sed to conveniently execute multiple sed substitutions at a go. See below:

sed -e 's/box/bag/' -e 's/black/red/' myfile.txt

Output:

This is my bag.
It is mine.
My bag is big, it is a big box.
My bag is red, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbag to play with my box

3. Using the Semicolon ;

The semicolon ; allows you to quickly conjoin your multiple sed commands and execute them at once on the same file. See below:

sed 's/box/bag/;s/black/red/' myfile.txt

Printed output:

This is my bag.
It is mine.
My bag is big, it is a big box.
My bag is red, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbag to play with my box

We've seen the different ways available that help facilitate executing multiple sed substitutions at a go. In the next section we will go over what happens when you only want the lines you've altered returned to output.

Printing Edited Lines Only

If you are working on a huge file, and you don't want the output of your sed commands on the file filling up your terminal environment each time you make an execution. There is a method you can use to isolate and output only the lines of the file your command has altered. See below:

sed -n 's/box/bag/p' myfile.txt

This is the generated output:

This is my bag.
My bag is big, it is a big box.
My bag is black, my box is strong, my box is the best.
Let's go to the sandbag to play with my box

In the command executed we use the -n option and the p flag. The -n option suppresses automatic printing of the output. Meanwhile, the p flag causes double printing of the edited lines. The combination of these two will cause the command to print out the edited lines only.

Note: sed provides a whole lot of options and flags you can use to make all kinds of tweaks to your substitutions. To access them, you simply enter sed --help into your terminal or man sed for the full user manual.

Delimiters

So far, we've been using the forward-slash / as a delimiter to separate the relevant categories of our substitution commands in the terminal. Although, there will be situations where this will not suffice.

For example if you're trying to use sed on a URL, a URL already has forward slashes / so an error will occur if you try to use a forward-slash(/) as a delimiter. Fortunately, other special characters can be used as delimiters too.

Let us try to make a sed substitution on the URL https://www.wikipedia.org/homepage.html. We will take the text pattern org/homepage and replace it with inc/search. See below:

echo "https://www.wikipedia.org/homepage.html" | sed 's$org/homepage$inc/search$'

The resulting output:

https://www.wikipedia.inc/search.html

In the command entered, notice that we now use a dollar-sign $ as a delimiter to separate the text we want to substitute out org/homepage and the text to be substituted in inc/search. If we had tried using the forward-slash / as a delimiter, the operation wouldn't have been successful.

Saving Changes

It is not enough to only make our edits in stream, you may wish to have the files altered permanently. To make the changes permanent, you must pass the -i option. See below for an example acting on the earlier created myfile.txt file:

sed -i 's/black/red/' myfile.txt

To confirm that the file has been altered, you will open it with the cat command like this:

cat myfile.txt

The result:

This is my box.
It is mine.
My box is big, it is a big box.
My box is red, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbox to play with my box.

Owing to the -i option, we have now confirmed that the old word black has been replaced with a new word red permanently.

Note: It is not advised to use the -i option casually because doing so could cause you to permanently lose your data. Use it only when you are certain you want to make irreversible changes.

Saving Changes to a New File

Suppose instead you wanted to save your changes to a new file leaving the first one unaltered. Enter the following command:

sed 's/box/bag/g' myfile.txt > secondfile.txt

If the file secondfile.txt did not exist, it is created automatically in your present working directory with the output generated from the sed command executed.

If you open the file with the command cat secondfile.txt, you should see the following:

This is my bag.
It is mine.
My bag is big, it is a big bag.
My bag is red, my bag is strong, my bag is the best.
My mother bought it for me.
Let's go to the sandbag to play with my bag.

Making a Backup

You can also pass an option that tells the computer to permanently edit the file but create a backup of the original document. To do this, you type out a variation of the -i option, which is the -i.bak option. It is used like this:

sed -i.bak 's/box/bag/g' myfile.txt

The above command will save your changes to the myfile.txt file, but also creates a backup of the original file with a .bak extension in the same directory. So the backup created from the command above will be myfile.txt.bak.

Running cat myfile.txt gets you:

This is my bag.
It is mine.
My bag is big, it is a big bag.
My bag is red, my bag is strong, my bag is the best.
My mother bought it for me.
Let's go to the sandbag to play with my bag.

Running cat myfile.txt.bak gets you:

This is my box.
It is mine.
My box is big, it is a big box.
My box is red, my box is strong, my box is the best.
My mother bought it for me.
Let's go to the sandbox to play with my box.

You can see that myfile.txt is permanently altered while myfile.txt.bak contains its original content.

Conclusion

Now you know the different ways of using sed to make basic substitutions in text files in the linux command line, how to make multiple sed changes in one entry, and how to save your changes permanently.