[ale] Speed un-tar?

[Jim Kinney]

unless you can spread that read/write load out over many, many spindles,
you're stuck. Now add in the VMmust access through the virtual drive
process and you've got another performance hit.

You _could_ add extra drives to the VM that are hosted on a decent array
(fiber channel or LA network iSCSI), copy the files to the new home in a
batch and hit the 4G RAM limit.

If possible, can you add more RAM to that VM?


On Tue, Jul 29, 2014 at 5:10 PM, Robert L. Harris <robert.l.harris at gmail.com
> wrote:

> I'm working on a tool to parse through a lot of data for processing.  Right
> now it's taking longer than I wish it would so I'm trying to find ways to
> improve the performance.  Right now it appears the biggest bottleneck is
> IO.  I'm looking at about 2000 directories which contain between 1 and 200
> files in tar.gz format on a VM with 4 Gigs of RAM.  I need to load the data
> into an array to do some pre-processing cleanup so I am currently chopping
> the files in each of the directories into an array of groups of 10 files at
> a time ( seems to be the sweet spot to prevent swap ) and then a straight
> forward loop of which each iteration executes:
>
>   tar xzOf $Loop |
>
> and then pushes it into my array for processing.
>
> I have tried:
>
>  gzcat $Loop | tar xO |
>
> which is actually slower.  Yes, I'm at the point of trying to squeeze
> seconds of time out of a group.  Any thoughts of a method which might be
> quicker?
>
> Robert
>
>
>
>
>
>
>
> --
> :wq!
> ---------------------------------------------------------------------------
> Robert L. Harris
>
> DISCLAIMER:
>       These are MY OPINIONS             With Dreams To Be A King,
>        ALONE.  I speak for                      First One Should Be A Man
>        no-one else.                                     - Manowar
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.ale.org/pipermail/ale/attachments/20140729/38cb3da3/attachment.html
> >
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>



-- 
-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you gain
at one end you lose at the other. It's like feeding a dog on his own tail.
It won't fatten the dog.
- Speech 11/23/1900 Mark Twain


*http://heretothereideas.blogspot.com/
<http://heretothereideas.blogspot.com/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20140729/385b6337/attachment.html>