[mirror-admin] How do I know when a mirror is done rsyncing?

[Jonathan Dieter]

On Fri, 2008-09-26 at 01:41 -0700, J.H. wrote:
> We aren't blocking the --delay-update - is there anything more specific
> you can give us to check with as we have the same setup on both our .US
> and .EU mirrors and no one else has reported any issues or problems...

I'm generating deltarpms from the RPMS, so I rsync in a two-step
process.  The first step is to download the new RPMS, but *not* the
repodata and not deleting the old RPMS.  Then I generate the deltarpms
from old->new RPMS, and run the second rsync which *should* get the
repodata and delete all the old RPMS without downloading any new RPMS.

First I run 
rsync -vaH --numeric-ids --delay-updates --exclude=DRPMS --exclude=debug
--exclude=repodata
mirrors.eu.kernel.org::fedora/updates/9/i386.newkey/ ./

And then, after generating the deltarpms, I run
rsync -vaH --numeric-ids --delay-updates --exclude=DRPMS --exclude=debug
--exclude="*.sh" --delete --delete-after
mirrors.eu.kernel.org::fedora/updates/9/i386.newkey/

I'm running into two different intermittent problems (though they seem
to be linked):
1. I run the first command, generate the deltarpms, and then run second
command.  When running the second command, new RPMS are sometimes
downloaded, which means they won't have deltarpms (as the old RPMS will
be deleted).

2. (This is what finally caused me to email) I run the first command,
generate deltarpms, and run the second command.  No new RPMS are
downloaded the second time and the old RPMS are deleted.  So far, so
good.  But the next day, I realize that a number of new RPMS listed in
primary.xml are missing.  So I run the first command again, and find
*more* new RPMS that should have come through the first time.

It seems that in both cases, the problem is that I'm accessing the
mirror while it's in an inconsistent state, and, in the second case,
reading repomd.xml to check for changes (under the assumption that it
will only change once the mirror is up-to-date) doesn't work either.

> That said if your a public mirror you likely want to us a Tier 1 to pull
> from as those are more likely to have the bits?  Are you using the
> private rsync for .EU or just mirroring straight off the public feed?

I'm a public mirror, but not one with lots of bandwidth.  The main
purpose of the mirror is to host deltarpms for those who have limited
bandwidth (at least until Fedora starts generating deltarpms, which may
happen some time between now and the beginning of the new millennium).

Because I have limited bandwidth, I don't want to be listed in the
Fedora mirrorlist, which means that I don't think I can mirror directly
off of a Tier 1 mirror.

I think I'm coming off the public feed (seeing as the data I'm rsyncing
is publicly available), but maybe I'm wrong.  I guess you should be able
to tell from the commands listed above.

Thanks for the quick reply,

Jonathan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20080926/61bd8b45/attachment.bin 
-------------- next part --------------
--