[ale] Bash/Python Question

[Tim Watts]

Even the user shell should be python? :-)

To be clear on my earlier post: by "use a queue" I meant "use a queue
with a fixed maximum set of workers".

On Tue, 2010-03-16 at 17:25 -0400, Richard Bronosky wrote:
> I agree with limiting the concurrent downloads. However I disagree
> with using bash at all for this. This should be done in pure python.
> 
> On 3/16/10, Tim Watts <tim at cliftonfarm.org> wrote:
> > If the downloaded files are not interrelated, perhaps you could invoke
> > download.py and parser.py as a unit on multiple threads or processes.
> > Then from bash you only need to call gatherer.py.
> >
> > BTW, if you have a large number of files to download firing off a butt
> > load of processes all at once isn't necessarily going to give you a
> > faster result. If we're talking 10's, probably don't worry about; but
> > 100's, use a queue.
> >
> >
> > On Tue, 2010-03-16 at 02:47 -0400, Omar Chanouha wrote:
> >> Hey all,
> >>
> >>    I am creating an information gatherer for a school project. I have
> >> a python file called gatherer that executes a bunch of python
> >> downloader files. I also have a python file that parses the downloaded
> >> information and places it into a database. Every day I want to execute
> >> the following:
> >>
> >> #!/bin/bash
> >> gatherer.py
> >> parser.py
> >>
> >> Unfortunately, the gatherer only initializes a bunch of downloader
> >> scripts. Therefore it exits just after the downloaders are
> >> initialized, not after they are finished. This means that the parser
> >> begins executing when the files are being downloaded, which of course
> >> leads to the parser seeing a bunch of empty files.
> >>
> >> Does anyone have a better solution than executing the parser at a
> >> constant time after the downloader?
> >>
> >> The gatherer looks something like:
> >>
> >> for file in list:
> >>  download.py file
> >>
> >> I need the gatherer to work this way because I want the files to
> >> download in parallel in order to speed up the process.
> >>
> >> Thanks,
> >>
> >> Omar
> >> _______________________________________________
> >> Ale mailing list
> >> Ale at ale.org
> >> http://mail.ale.org/mailman/listinfo/ale
> >> See JOBS, ANNOUNCE and SCHOOLS lists at
> >> http://mail.ale.org/mailman/listinfo
> >
> >
> > ________
> > Nearly all men can stand adversity, but if you want to test a man's
> > character, give him power.
> > -- Abraham Lincoln
> >
> >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://mail.ale.org/mailman/listinfo/ale
> > See JOBS, ANNOUNCE and SCHOOLS lists at
> > http://mail.ale.org/mailman/listinfo
> >
> 


________
I am a firm believer in the people. If given the truth, they can be
depended upon to meet any national crises. The great point is to bring
them the real facts.
-- Abraham Lincoln