[mirror-admin] How do I know when a mirror is done rsyncing?

[J.H.]

Inline

- John 'Warthog9' Hawley
Chief Kernel.org Admin

Jonathan Dieter wrote:
> On Fri, 2008-09-26 at 01:41 -0700, J.H. wrote:
>   
>> We aren't blocking the --delay-update - is there anything more specific
>> you can give us to check with as we have the same setup on both our .US
>> and .EU mirrors and no one else has reported any issues or problems...
>>     
>
> I'm generating deltarpms from the RPMS, so I rsync in a two-step
> process.  The first step is to download the new RPMS, but *not* the
> repodata and not deleting the old RPMS.  Then I generate the deltarpms
> from old->new RPMS, and run the second rsync which *should* get the
> repodata and delete all the old RPMS without downloading any new RPMS.
>
> First I run 
> rsync -vaH --numeric-ids --delay-updates --exclude=DRPMS --exclude=debug
> --exclude=repodata
> mirrors.eu.kernel.org::fedora/updates/9/i386.newkey/ ./
>
> And then, after generating the deltarpms, I run
> rsync -vaH --numeric-ids --delay-updates --exclude=DRPMS --exclude=debug
> --exclude="*.sh" --delete --delete-after
> mirrors.eu.kernel.org::fedora/updates/9/i386.newkey/
>   
Well you might be running into a couple of different problems here 
already.  Mirrors.eu.kernel.org is actually a round-robin address for 
mirrors.se.kernel.org and mirrors.nl.kernel.org - which are not 
guaranteed to be exactly the same in terms of being up to date, as the 
each independently sync their own repos.  I would start by choosing the 
closest one and sticking to that exclusively as you might be getting 
flap just between the two mirrors.

> I'm running into two different intermittent problems (though they seem
> to be linked):
> 1. I run the first command, generate the deltarpms, and then run second
> command.  When running the second command, new RPMS are sometimes
> downloaded, which means they won't have deltarpms (as the old RPMS will
> be deleted).
>
> 2. (This is what finally caused me to email) I run the first command,
> generate deltarpms, and run the second command.  No new RPMS are
> downloaded the second time and the old RPMS are deleted.  So far, so
> good.  But the next day, I realize that a number of new RPMS listed in
> primary.xml are missing.  So I run the first command again, and find
> *more* new RPMS that should have come through the first time.
>
> It seems that in both cases, the problem is that I'm accessing the
> mirror while it's in an inconsistent state, and, in the second case,
> reading repomd.xml to check for changes (under the assumption that it
> will only change once the mirror is up-to-date) doesn't work either.
>   

My guess is that your getting the round-robin effect and the mirrors 
themselves are slightly out of sync with each other (which isn't a big 
deal except in cases like your hitting now)

>> That said if your a public mirror you likely want to us a Tier 1 to pull
>> from as those are more likely to have the bits?  Are you using the
>> private rsync for .EU or just mirroring straight off the public feed?
>>     
>
> I'm a public mirror, but not one with lots of bandwidth.  The main
> purpose of the mirror is to host deltarpms for those who have limited
> bandwidth (at least until Fedora starts generating deltarpms, which may
> happen some time between now and the beginning of the new millennium).
>
> Because I have limited bandwidth, I don't want to be listed in the
> Fedora mirrorlist, which means that I don't think I can mirror directly
> off of a Tier 1 mirror.
>
> I think I'm coming off the public feed (seeing as the data I'm rsyncing
> is publicly available), but maybe I'm wrong.  I guess you should be able
> to tell from the commands listed above.
>   
If your just pointing at mirrors.eu.kernel.org and using what we publish 
as the rsync module - your using the public rsync.

- John

> Thanks for the quick reply,
>
> Jonathan
>   
> ------------------------------------------------------------------------
>
> --

--