Running Programs in Parallel Using xargs

The xargs is a concept of Linux/Unix piping, piping in Linux is a way to pass one command-processed output to another process for further processing. Using the xargs command to run many processes in parallel is a sort of command line argument known as “running programs in parallel.” Similar to this, using xargs allows for the transmission of one command’s output to another in parallel due to its one-by-one input passing method. The nature of “one-by-one” argument passing makes xargs an effective choice for parallel process handling.

In other words, think like we run a command “xyz” that gives an output “a” and we want to process “a” furthermore so we use piping in Linux, and the xargs is a way to perform piping.

For a better understanding of piping refer to this article about Linux Piping concepts.

Piping in Unix or Linux

Two commands are separated by the ‘|’ symbol. One command’s output is used as the second command’s input.

$ command1 |  command2

Using XARGS:

As we already discussed it is a Piping command, but it is not an ordinary piping command it converts the ordinary standard input(std input) into command line arguments. Command line arguments are a way to pass an argument for a process to act. Like parameters in a function. It is useful because most of the Linux commands only handle command line arguments so we need mechanisms like xargs.

When a program is performed, parameters known as command line arguments are supplied to it. These arguments are usually passed in the command line after the program name. The stream of data that a program reads from is known as standard input, or “stdin.” This data can be redirected from a file or entered interactively by the user while the program is running.

Examples 1:

The ‘ls’ command is used to print the list of directories and files, the ‘|’ pipe symbol is used to pipe the two commands, and the ‘echo’ command is used for printing something in the display so if we want to merge those commands we use the symbol ‘|’. But the problem with the ‘echo’ command is it only accepts the command line arguments so we need xargs to provide all files and directory names as a command line input to the ‘echo’ command.

Step 1: If you are currently working in a directory, you should have a look at the list of directories and files that are contained within that directory currently. With this, you will be able to comprehend how files and directories change in response to certain commands that are executed further.

Command:

ls

Output:

List of files and directories in the current working directory.

ls: It is a command that displays a list of files, directories, and other information in a directory structure.

Step 2: Try to pipe the ls command with the echo command.

Command:

ls | echo

Output:

ls | echo

• The symbol represented by the symbol | is used to pipe multiple commands together, with the output of the first command being sent on to the second command as a standard and one-by-one input.
• echo: The echo command is a command-line utility found in various operating systems such as Unix, Linux, and Windows. The main objective of the terminal or command prompt is to exhibit messages or information.
• The reason it doesn’t work is that the ls command provides standard output to echo, which the echo command can’t handle, and echo requires a by-one argument to pass parallel.

Step 3: Now, use xargs to try echo pipe is. We use xargs because we know that the previous pipe couldn’t work with ls and echo. This is because the echo command needs to take standard input(stdin), and the ls command offers raw output, which means that the echo command couldn’t work in the previous piped way.

Command:

ls | xargs echo

Output:

ls | echo with xargs

xargs: xargs main job is to build and run command lines by reading from standard input or a file and turning it into string arguments for a given command. It lets you process raw data and pass it as arguments to a command, which gets around the length limits of command-line arguments.

The reason it works is that the xargs pass one-by-one command line arguments to the parallel echo command.

Combined Example – 1 Output:

Combined Example – 1 Output

Examples – 2 of xargs (Parallel argument passing):

This example will help us gain a thorough knowledge of xargs. We need to read the content from a file containing several names written down in a line-by-line format, then and need to produce another file with the names in ascending lexical order.

Step 1: Check the list of files and directories in the current directory.

List of files and directories in the current working directory.

Step 2: Read out what’s in the file myfile.txt. We can read the contents of a file with the cat command to figure out what happens after a certain action in the further commands.

Command:

cat <file_name>

Output:

content of myfile.txt file

• cat: The “cat” command, which stands for “concatenate,” is a Unix and Linux tool that can show the contents of files, join them together, or make new ones.
• <filename>: filename whose content you want to view.

Step 3: We can use xargs piping to transform the standard output of the cat command into command line arguments, which we can then pass to tr and sort command to execute in parallel with the help of xargs.

Command:

cat <file_name_1> | xargs echo | tr ' '  '\n'  | sort > <file_name_2>

Output:

apply xargs with echo, tr, and sort

• xargs: It is used to run the command in parallel as providing command line arguments.
• <file_name_1>, <file_name_2>: input and output file respectively, [myfile.txt], [myOutput.txt].
• tr: It is used to modify the behavior of the command for translating and deleting text in files.
• sort: It is used to sort the content, in ascending or descending order, default order to be ascending.

Step 4: We obtain the output file similar to the text below after applying this step.

Command:

cat <file_name_2>

Output:

myOutput.txt file content

<file_name_2>: display output file content.
So now as we know the content is arranged in ascending order in one by one characters present in file.

Combined Example – 2 Output:

Combined Example – 2 Output

Conclusion:

xargs is a very important tool because it turns standard input into command-line arguments, which makes it easier to run commands in parallel. This feature is especially useful when working with orders that need to be processed in parallel. The given example shows how well xargs handles merging commands like “ls” and “echo” in a parallel setting, showing how useful it is for making Linux command-line actions run more smoothly. Learning how to use and understand xargs makes it easier to run tasks in parallel and more quickly.