Open In App

The “Argument List Too Long” Error in Linux Commands

Improve
Improve
Like Article
Like
Save
Share
Report

There are times when we encounter directories that contain a large number of files, if we try to perform certain operations on those particular directories we may get the error Argument List Too Long. In this article, we will see the cause of this error and how to resolve this error.

We have created a directory(temp) with a large number of files (>200K). All the operations will be performed on that directory. The OS used will be Ubuntu for the demonstration.

 What Causes the Error?

 First, let us see the examples where we encounter this error and then see the cause of its occurrence. 

Case 1: When we try to list a large number of files

Let us see the number of files in the temp directory, use the below command:

ls -lrt | wc -l

Number of files

The number of files is 220001, this is a huge number. Let us try to get all the file names with the word file.

ls -lrt file* | wc  -l

File names

The error can be seen in the image. 

Case 2: When we try to remove a large number of files

This error can also be seen when we try to remove all the files from this directory.

rm  *

removing files

The reason behind this Error

The main cause of this error lies in the argument buffer space, when we try to search for a file or try to remove a file the asterisk(*) expands the number of arguments in the shell. If the number of those arguments is greater than the argument buffer space then this error occurs, as the bash is not able to handle it.

The file command is expanded as follows:

 ls -lrt file1 file2 file3 file4 .....................  |  wc  -l

Resolving the Error

Let us explore the various ways that we can use to handle this error.

  • Method 1: Using the find command to iterate over the files
  • Method 2: Splitting the file manually
  • Method 3: Using For Loops
  • Method 4: Completely delete the directory

Let’s go through all these methods:

Method 1: Using the find command to iterate over the files

find. -iname "file*" | xargs ls -lrt | wc  -l

First, with the help of the find command, we find all the files starting with the word file and then with the help of xargs command the wc and -l flags are executed.

Find Command

The error is not showing this time.

Method 2: Splitting the file manually

By using this way we make the length of the argument provided shorter so that it fits in buffer space.

ls -lrt file12* | wc -l

Shorten the argument

ls -lrt file22* | wc -l

Splitting

Since the values are much smaller than earlier the error is resolved.

Method 3: Using For Loops

This approach takes the use of loops for iterating over the required files, skipping away the unwanted results.

for k in file*; do echo “$k”; done | wc -l

Skipping unwanted results

Note: This method is comparatively slower than the other methods.

Method 4: Completely delete the directory

If we want to get rid of all the files in the directory, but we are unable to do so because of the large number of files then we can use this way. Simply delete the complete directory and recreate it again, the error will not occur.

rm -rf /home/osboxes/temp
cd home/osboxes
mkdir temp

Delete directory

create directory

Note: After the creation of the new directory all the previous permissions and rights of users and groups will be lost and will have to be created again.

So this was the explanation of the error Argument List Too Long in Linux along with the cause and ways to resolve it. 

Conclusion

The “Argument List Too Long” error is a common issue in Linux when dealing with directories containing a large number of files, exceeding the shell’s argument buffer capacity. This error can occur during operations like listing files (ls) or removing them (rm) when wildcard patterns expand into a lengthy list of arguments.

To resolve this error, several methods can be used:

1.Using the find command with xargs: This approach allows you to search for files using the find command and then process them using xargs. It effectively avoids expanding all file names into arguments at once.

2.Manually splitting files: You can manually specify shorter patterns when listing or removing files, reducing the argument length to fit within the buffer space.

3.Using For Loops: Employing loops, like ‘for’, can help iterate over files incrementally, processing them in manageable batches.

4.Completely deleting and recreating the directory: In extreme cases, if you want to remove all files from a directory, deleting the directory itself and then recreating it can be a last resort. However, be cautious about permissions and ownership issues when taking this approach.


Last Updated : 06 Oct, 2023
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads