[mirror-admin] report_mirror taking a long time to run

[HEAnet Mirror Admin]

Hi,

I'm still noticing excessively long run times for report_mirror, it
took nearly six hours to complete this morning. Here's the output with
start/finish timestamps (in epoch seconds) for each of the directory
traversals:

gen_dirtree started at 1290575953.78 /ftp/pub/fedora/linux/
gen_dirtree finished at 1290583867.54
gen_dirtree started at 1290583867.54 /ftp/pub/fedora/epel/
gen_dirtree finished at 1290585276.7
gen_dirtree started at 1290585276.7 /ftp/pub/fedora-archive/fedora/extras/
gen_dirtree finished at 1290586649.04
gen_dirtree started at 1290586649.04 /ftp/pub/fedora-archive/fedora/core/
gen_dirtree finished at 1290588254.69
gen_dirtree started at 1290588254.69 /ftp/pub/fedora-archive/fedora/linux/
gen_dirtree finished at 1290599021.38
Category Fedora Linux directories updated: 500  added: 0  deleted 0
Category Fedora EPEL directories updated: 162  added: 0  deleted 0
Category Fedora Extras directories updated: 87  added: 0  deleted 0
Category Fedora Core directories updated: 426  added: 0  deleted 0
Category Fedora Archive directories updated: 1  added: 0  deleted 652
checked in successful

Again, most of the delay seems to be happening when reading the fedora
archive directory. 

We're using the latest version of report_mirror from git on Ubuntu
8.04 with Python 2.5.2, I've also attached our report_mirror.conf.

rg

On Fri, 2 Jul 2010 17:09:24 +0100
HEAnet Mirror Admin <rh-mirror-list at heanet.ie> wrote:

> Hi Matt,
> 
> On Tue, 22 Jun 2010 17:11:09 -0500
> <Matt_Domsch at dell.com> wrote:
> 
> > Can you through a print statement in there, to see if the delay is
> > in walking the disk, or if the delay is in the upload to the
> > database?
> > 
> > I'll be out of the office and mostly offline for the next 4 weeks or
> > so, expect very long delays in my responses, but I'll try to get at
> > things when I return or as I can between events.
> 
> I printed out some timestamps (epoch) before and after each call to
> gen_dirtree, these are attached.
> 
> It certainly looks like the delay is occuring when it trawls through
> the directories, although I'm not exactly sure why. The storage
> backend is a NAS appliance mounted over NFS (Isilon IQ), we haven't
> noticed any performance problems with it.
> 
> rg
> 


-- 
Rob Gallagher | Public Key: 0x1DD13A78

HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin 1.
Registered in Ireland, no 275301
T: (+353-1) 6609040  F: (+353-1) 6603666 WWW: http://www.heanet.ie/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: report_mirror.conf
Type: application/octet-stream
Size: 1805 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20101124/4865bfab/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20101124/4865bfab/attachment.bin 
-------------- next part --------------
--