Size of argument and environment lists exceeds limit

Problem

If you try to cp or mv a large number of files or directories, you will encounter this error:

$ cd big_dir
$ cp * ../../some_other_dir
Failed to execute process '/bin/cp'. Reason:
The total size of the argument and environment lists 3.2MB exceeds the operating system limit of 2MB.
Try running the command again with fewer arguments.

Note that you can copy or move one directory that contains a million files or any number of files for that matter. This error happens only if the shell has to pass a large number of input arguments to the cp or mv programs. So, if you run the above command from inside a directory containing 100K files, you will surely get this error.

This is because there is a limit to the size of the arguments and environment strings that can be passed to a program. That limit can be queried:

$ getconf ARG_MAX
2097152

The result will vary on different computers. But they always have a limit and it is encoded in the Linux kernel. There does not seem to be an userspace method to increase this size.

Solution

Instead of looking to increase the ARG_MAX size, examine the real problem. Why make the shell expand all the filenames and pass them as one gigantic list of strings to the programs? Instead these alternate solutions can be tried:

  • See if you can instead cp or mv a parent directory, instead of a million files.

  • Move the files one by one by writing a loop in shell script or Python.

  • Use other programs like rsync to copy a directory to the destination.

Reference: Search for ARG_MAX in the execve(2) manpage

Tried with: Ubuntu 16.04

Advertisements

How to copy or move from shell with progressbar

When I am copying or moving a big file or a large number of small files, I wish I could see some progress of the operation and the speed at which the operation is being performed. This can be done by using the Advanced Copy mod of the cp and mv commands, which have been patched to display a progressbar with all this information during the operation. You can either get the source code or pre-built binaries from here.

I like to use these commands by default, so I move them to /usr/local/bin:

$ sudo cp ./cp /usr/local/bin
$ sudo cp ./mv /usr/local/bin

Typically, your PATH puts /usr/local/bin ahead of /bin, so the patched versions of the commands will be executed now. Else, change your PATH to accomplish this.

To view the progressbar for a operation, use the -g option:

$ cp -g -R foo-big-file.bin /media/joe/backup/

I use the fish shell, where I have added aliases for both of these commands with progressbar display:

function cpg
    cp -g $argv
end

function mvg
    mv -g $argv
end

Tried with: Ubuntu 14.04