[mirror-admin] report_mirror taking a long time to run

[HEAnet Mirror Admin]

Hi Matt,

On Wed, 24 Nov 2010 12:00:27 -0600
Matt Domsch <matt at domsch.com> wrote:

> Now, that doesn't explain the slow directory listings.  If you look at
> the python code though, it's simply doing an os.walk() on the tree, so
> it's limited to the speed it can do that.  This part of the tree is
> really really big - it has hundreds of thousands of files and hundreds
> of directories, so it could easily take the nearly 3 hours you note
> here just to walk it.  For this reason, as the content changes only
> very rarely, you can set enabled=0 for your regular rsync runs, and on
> the rare occasions when you sync the archive content, run it once with
> enabled=1.

Thanks for the info, I've applied your suggested changes to our
report_mirror.conf. The content categories section now reads:

[Fedora Linux]
enabled=1
path=/ftp/pub/fedora/linux/

[Fedora EPEL]
enabled=1
path=/ftp/pub/fedora/epel/

[Fedora Archive]
enabled=0
path=/ftp/pub/fedora-archive/

It's just finished a run with the new configuration, however things are
still being held up by Fedora Archive:

Started report_mirror - Fri Nov 26 10:30:52 GMT 2010
gen_dirtree started at 1290767452.79 /ftp/pub/fedora/linux/
gen_dirtree finished at 1290772084.53
gen_dirtree started at 1290772084.53 /ftp/pub/fedora/epel/
gen_dirtree finished at 1290773244.28
gen_dirtree started at 1290773244.28 /ftp/pub/fedora-archive/
gen_dirtree finished at 1290784445.19
Category Fedora Linux directories updated: 500  added: 0  deleted 0
Category Fedora EPEL directories updated: 162  added: 0  deleted 0
Category Fedora Archive directories updated: 653  added: 0  deleted 0
checked in successful
Finished report_mirror - Fri Nov 26 15:14:41 GMT 2010

Would commenting out the Fedora Archive section make any difference?

rg

-- 
Rob Gallagher | Public Key: 0x1DD13A78

HEAnet Limited, Ireland's Education and Research Network
1st Floor, 5 George's Dock, IFSC, Dublin 1.
Registered in Ireland, no 275301
T: (+353-1) 6609040  F: (+353-1) 6603666 WWW: http://www.heanet.ie/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20101126/c9487193/attachment.bin 
-------------- next part --------------
--