Re: copying many files: hard links, block size, ram disk

["Alex C Koch" <]

I'd think that journaling slows down an operation like this and ext2/3
isn't the best for small/many files.  Might want to try reiserFS.  Have
you already done things like turn off the last access time on the
filesystem(s)?   Tried any other tweaks?

~ Alex

Marcel M. Cary wrote:
> Tristan,
>
> If the files are unlinked before being modified, the rsync --link-dest
> option might help to avoid copying the file data.  Whether that works for
> you depends on your intended use, of course.
>
> Is your filesystem's block size a good match for your typical file size?
> Maybe if you have lots of 400 byte files, the filesystem block should be
> on the small side, say, 512 bytes?
>
> Also depending on your intended use, maybe storing the files on ramdisk
> would help?
>
> Marcel
>
>
> On Sun, 20 Apr 2008, Kamaraju Kusumanchi wrote:
>
>> Hi Tristan,
>>
>> My only suggestion is to use rsync instead of cp. Possible options
>> with rsync are avzp. See rsync's man page for more details. The
>> advantage of rsync over cp is that the files are transferred after
>> they are compressed (so it will be faster). Another advantage with
>> rsync is that if your transfer is interrupted sometime in between, you
>> can resume it from that point onwards.
>>
>> hth
>> raju
>>
>> On Sun, Apr 20, 2008 at 6:37 PM, Tristan Lefebure <>
>> wrote:
>>> Hi,
>>>
>>>  These days I often have to copy several million of files from one
>>> folder to
>>>  another on the same computer (and usually the same disk), and it takes
>>> a
>>>  while with a regular 'cp' approach (several hours).
>>>
>>>  The files are rather small (~400 Bites), so I think that most of the
>>> time is
>>>  spent creating the files, not copying the data. Would you have a
>>> suggestion
>>>  to speed up the process?
>>>
>>>  I've already tried to create a tar archive, but it also take a while
>>> create
>>>  and extract the archive. Should I use another file system (I use ext3
>>> with
>>>  ubuntu 7.10).
>>>
>>>  Thanks for any help!
>>>  --
>>>  Tristan Lefebure
>>>
>>>  Population Medicine & Diagnostic Sciences
>>>  College of Veterinary Medicine
>>>  Cornell University
>>>
>>>  phone: (607) 253 4228
>>>
>>>  http://www.people.cornell.edu/pages/tnl7/
>>>
>>>
>>
>>
>>
>
>