Re: copying many files

[Nathan Lutchansky <]

Tristan,

It sounds like these files may all be part of a single data collection.
 If so, have you considering putting this data into an SQL database
instead?  It would be faster to use SQL to access multiple data blocks
in a single operation, especially if there is more than one lookup key
you use to locate the data.  MySQL and PostgreSQL would also give you
easy backups, clustering, and network-transparent data access.

If there is some software you have that requires this data to be
available as a collection of files, you could use something like
Fuse::DBI to expose the SQL data as a pseudo-filesystem so that existing
applications would still work properly.  It will probably take a little
work to set up, but you may find that it simplifies your life overall
when storing this type of data.  -Nathan

Tristan Lefebure wrote:
> Hi,
> 
> These days I often have to copy several million of files from one folder to 
> another on the same computer (and usually the same disk), and it takes a 
> while with a regular 'cp' approach (several hours).
> 
> The files are rather small (~400 Bites), so I think that most of the time is 
> spent creating the files, not copying the data. Would you have a suggestion 
> to speed up the process?
> 
> I've already tried to create a tar archive, but it also take a while create 
> and extract the archive. Should I use another file system (I use ext3 with 
> ubuntu 7.10).
> 
> Thanks for any help!