Re: copying many files: hard links, block size, ram disk

["Marcel M. Cary" <]

Tristan,

If the files are unlinked before being modified, the rsync --link-dest 
option might help to avoid copying the file data.  Whether that works for 
you depends on your intended use, of course.

Is your filesystem's block size a good match for your typical file size? 
Maybe if you have lots of 400 byte files, the filesystem block should be 
on the small side, say, 512 bytes?

Also depending on your intended use, maybe storing the files on ramdisk 
would help?

Marcel


On Sun, 20 Apr 2008, Kamaraju Kusumanchi wrote:

Hi Tristan,

My only suggestion is to use rsync instead of cp. Possible options
with rsync are avzp. See rsync's man page for more details. The
advantage of rsync over cp is that the files are transferred after
they are compressed (so it will be faster). Another advantage with
rsync is that if your transfer is interrupted sometime in between, you
can resume it from that point onwards.

hth
raju

On Sun, Apr 20, 2008 at 6:37 PM, Tristan Lefebure <> wrote:

Hi,

 These days I often have to copy several million of files from one folder to
 another on the same computer (and usually the same disk), and it takes a
 while with a regular 'cp' approach (several hours).

 The files are rather small (~400 Bites), so I think that most of the time is
 spent creating the files, not copying the data. Would you have a suggestion
 to speed up the process?

 I've already tried to create a tar archive, but it also take a while create
 and extract the archive. Should I use another file system (I use ext3 with
 ubuntu 7.10).

 Thanks for any help!
 --
 Tristan Lefebure

 Population Medicine & Diagnostic Sciences
 College of Veterinary Medicine
 Cornell University

 phone: (607) 253 4228

 http://www.people.cornell.edu/pages/tnl7/