appending a list of files to one file using xargs command

Contributed by seamonkey420 September 2, 2004  
Tagged: Korn shell

This recipe explains how to use the xargs command in ksh to work with multiple files. This recipe will show how to combine 10 files together into one file without manually doing a: cat file1 > bigfile, cat file2 >> bigfile, etc. The uses of this command are wide and very helpful; a great way to work with large amounts of files (also avoids the parameter list is too long message when trying to grep more than 1024 files)

This is a very basic ksh command recipe.

If you work with AIX/Unix/ksh; most likely there are times when you need to do multiple things with files. This recipe will show how to append a list of files to one big file using two commands versus manually cat file1 > bigfile, then cat file2 >> bigfile, etc.

Files/Directory
ok, in my example; i am going to take the contents of 5 files and combine them into one big file.

so my files are: file1, file2, file3, file4, file5
the combined file will be called: bigfile.dat

also, i have all the files in the same directory (/usr/acct/test/files/ )

First, we will create a list of the files (this is extremely helpful when working with lots of files, like 1500 files!)
for my example i would type this command:

ls | grep file > filelist

the ls command will list the contents of the directory, then i am searching for any file that has file in its name and then appending the results to a new file called filelist

Next, now we have a list of the files we want to combine into one bigfile.dat
now to combine these files, i will type this:

cat filelist | xargs cat >> bigfile.dat

this command tells me to take the list of files (filelist) and then for each file listed, append the contents to the file bigfile.dat.

the xargs command is great for using with pipe ( | ); xargs will let you work with large numbers of files in a list or grep’d.

____________________
NOTES:

here is a definition of what xargs command does courtesy of our AIX/Kornshell Reference Manual.

Description

The generated command line length is the sum of the size, in bytes, of the Command and each Argument treated as strings, including a null byte terminator for each of these strings. The xargs command limits the command line length. When the constructed command line runs, the combined Argument and environment lists can not exceed ARG_MAX bytes. Within this constraint, if you do not specify the -n or the -s flags, the default command line length is at least the value specified by LINE_MAX.

a few examples of using the command, again from our manual.

To insert file names into the middle of command lines, enter:

ls | xargs -t -I {} mv {} {}.old

This command sequence renames all files in the current directory by adding .old to the end of each name. The -I flag tells the xargs command to insert each line of the ls directory listing where {} (braces) appear. If the current directory contains the files chap1, chap2, and chap3, this constructs the following commands:
mv chap1 chap1.old
mv chap2 chap2.old
mv chap3 chap3.old

Another example:
To use a command on files whose names are listed in a file, enter:
xargs lint -a If the cfiles file contains the following text:
main.c readit.c
gettoken.c
putobj.c
the xargs command constructs and runs the following command:
lint -a main.c readit.c gettoken.c putobj.c
If the cfiles file contains more file names than fit on a single shell command line (up to LINE_MAX), the xargs command runs the lint command with the file names that fit. It then constructs and runs another lint command using the remaining file names. Depending on the names listed in the cfiles file, the commands might look like the following:
lint -a main.c readit.c gettoken.c . . .
lint -a getisx.c getprp.c getpid.c . . .
lint -a fltadd.c fltmult.c fltdiv.c . . .
This command sequence is not quite the same as running the lint command once with all the file names. The lint command checks cross-references between files. However, in this example, it cannot check between the main.c and the fltadd.c files, or between any two files listed on separate command lines.
For this reason you may want to run the command only if all the file names fit on one line. To specify this to the xargs command use the -x flag by entering:

xargs -x lint -a If all the file names in the cfiles file do not fit on one command line, the xargs command displays an error message.

Definition/Examples taken from:
AIX Version 4.3 Commands Reference, Volume 6 applies to the AIX Version 4.3, 3270 Host Connection Program 2.1 and 1.3.3 for AIX, and Distributed SMIT 2.2 for AIX licensed programs,