Re: copying many files: hard links, block size, ram disk

[Hurf Sheldon <]

Is the goal to move them or have a duplicate copy?
If a duplicate copy, then is it for access or backup?
h
Alex C Koch wrote:

I'd think that journaling slows down an operation like this and ext2/3
isn't the best for small/many files.  Might want to try reiserFS.  Have
you already done things like turn off the last access time on the
filesystem(s)?   Tried any other tweaks?

~ Alex

Marcel M. Cary wrote:

Tristan,

If the files are unlinked before being modified, the rsync --link-dest
option might help to avoid copying the file data.  Whether that works for
you depends on your intended use, of course.

Is your filesystem's block size a good match for your typical file size?
Maybe if you have lots of 400 byte files, the filesystem block should be
on the small side, say, 512 bytes?

Also depending on your intended use, maybe storing the files on ramdisk
would help?

Marcel


On Sun, 20 Apr 2008, Kamaraju Kusumanchi wrote:

Hi Tristan,

My only suggestion is to use rsync instead of cp. Possible options
with rsync are avzp. See rsync's man page for more details. The
advantage of rsync over cp is that the files are transferred after
they are compressed (so it will be faster). Another advantage with
rsync is that if your transfer is interrupted sometime in between, you
can resume it from that point onwards.

hth
raju

On Sun, Apr 20, 2008 at 6:37 PM, Tristan Lefebure <>
wrote:

Hi,

 These days I often have to copy several million of files from one
folder to
 another on the same computer (and usually the same disk), and it takes
a
 while with a regular 'cp' approach (several hours).

 The files are rather small (~400 Bites), so I think that most of the
time is
 spent creating the files, not copying the data. Would you have a
suggestion
 to speed up the process?

 I've already tried to create a tar archive, but it also take a while
create
 and extract the archive. Should I use another file system (I use ext3
with
 ubuntu 7.10).

 Thanks for any help!
 --
 Tristan Lefebure

 Population Medicine & Diagnostic Sciences
 College of Veterinary Medicine
 Cornell University

 phone: (607) 253 4228

 http://www.people.cornell.edu/pages/tnl7/