[ale] Parallel data transfers (NFS).

[JD]

On 11/05/2014 08:44 PM, Raj Wurttemberg wrote:
>> So - for 10TB .... I'd look at getting better connections and a great disk
>> subsystem.  Thought about infiniband and 10G networking for the NAS?
>>
>> Is the data all new or is it partially changed?  Can you get a live,
> replicated,
>> feed, from the source?
> 
> I was just looking for a way to do parallel file copies. :)
> 


The slowdown is likely to be the storage on the dev server or the network. You
have to look at each point in the chain to determine if the current solution is
already flooding any part therein.

A simple way to cause XX concurrent copies is to spawn them based on first
character in the filename.

I suspect you were looking for a -j10 switch? Something like GNU "parallel"
perhaps? https://www.gnu.org/software/parallel/

https://superuser.com/questions/327659/whats-the-best-way-to-perform-a-parallel-copy-on-unix
has more discussions on this.