[ale] Speed un-tar?

[Robert L. Harris]

Unfortunately I can't touch the VM's configuration or the hardware
underneath it.  Supposedly I'm spread across a minimum of 6 "fast" disks
already.  I can't really go less than 10 files though as I am concerned
with information being spread across multiple files.  I was hoping someone
knew a tool/util which would rip through the data faster I had not found
yet.

Robert



On Tue, Jul 29, 2014 at 4:00 PM, Jim Kinney <jim.kinney at gmail.com> wrote:

> unless you can spread that read/write load out over many, many spindles,
> you're stuck. Now add in the VMmust access through the virtual drive
> process and you've got another performance hit.
>
> You _could_ add extra drives to the VM that are hosted on a decent array
> (fiber channel or LA network iSCSI), copy the files to the new home in a
> batch and hit the 4G RAM limit.
>
> If possible, can you add more RAM to that VM?
>
>
> On Tue, Jul 29, 2014 at 5:10 PM, Robert L. Harris <
> robert.l.harris at gmail.com
> > wrote:
>
> > I'm working on a tool to parse through a lot of data for processing.
>  Right
> > now it's taking longer than I wish it would so I'm trying to find ways to
> > improve the performance.  Right now it appears the biggest bottleneck is
> > IO.  I'm looking at about 2000 directories which contain between 1 and
> 200
> > files in tar.gz format on a VM with 4 Gigs of RAM.  I need to load the
> data
> > into an array to do some pre-processing cleanup so I am currently
> chopping
> > the files in each of the directories into an array of groups of 10 files
> at
> > a time ( seems to be the sweet spot to prevent swap ) and then a straight
> > forward loop of which each iteration executes:
> >
> >   tar xzOf $Loop |
> >
> > and then pushes it into my array for processing.
> >
> > I have tried:
> >
> >  gzcat $Loop | tar xO |
> >
> > which is actually slower.  Yes, I'm at the point of trying to squeeze
> > seconds of time out of a group.  Any thoughts of a method which might be
> > quicker?
> >
> > Robert
> >
> >
> >
> >
> >
> >
> >
> > --
> > :wq!
> >
> ---------------------------------------------------------------------------
> > Robert L. Harris
> >
> > DISCLAIMER:
> >       These are MY OPINIONS             With Dreams To Be A King,
> >        ALONE.  I speak for                      First One Should Be A Man
> >        no-one else.                                     - Manowar
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL: <
> >
> http://mail.ale.org/pipermail/ale/attachments/20140729/38cb3da3/attachment.html
> > >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://mail.ale.org/mailman/listinfo/ale
> > See JOBS, ANNOUNCE and SCHOOLS lists at
> > http://mail.ale.org/mailman/listinfo
> >
>
>
>
> --
> --
> James P. Kinney III
>
> Every time you stop a school, you will have to build a jail. What you gain
> at one end you lose at the other. It's like feeding a dog on his own tail.
> It won't fatten the dog.
> - Speech 11/23/1900 Mark Twain
>
>
> *http://heretothereideas.blogspot.com/
> <http://heretothereideas.blogspot.com/>*
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.ale.org/pipermail/ale/attachments/20140729/385b6337/attachment.html
> >
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>



-- 
:wq!
---------------------------------------------------------------------------
Robert L. Harris

DISCLAIMER:
      These are MY OPINIONS             With Dreams To Be A King,
       ALONE.  I speak for                      First One Should Be A Man
       no-one else.                                     - Manowar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20140729/e382a9b2/attachment.html>