[mirror-admin] Slow syncing?

[J.H.]

Marek,

You are correct I'm measuring the complete time from start to finish.
That said the data is still valid, the time it takes to generate the
file list should be relatively uniform across runs (not perfectly I'll
admit but close enough for my purposes).  That said I don't use the
start time / duration in my graphs for anything other than when the
process started / ended.

The transfer speed is used from the values rsync gives back, or in the
case below: 28379.86 bytes/sec, so those should be accurate.  A sync
that does nothing is still going to transfer very little and it's
internal measurements are far more accurate / useful for graphing
purposes than anything else I have.

Bonus, since 'null' runs are so small in comparison to a real update
they more or less are going to get drowned out in the graphs by the
larger more obvious syncs, which given the graphs I've already sent are
very obviously faster at least for me.

- John 'Warthog9' Hawley

On 01/11/2010 01:42 AM, Marek Mahut wrote:
> Hello J.H.,
> 
> Looking at your log:
> 
> <snip>
> receiving file list ... done
> 
> Number of files: 827975
> Number of files transferred: 0
> Total file size: 1274058729239 bytes
> Total transferred file size: 0 bytes
> Literal data: 0 bytes
> Matched data: 0 bytes
> File list size: 31402197
> File list generation time: 1085.205 seconds
> File list transfer time: 0.001 seconds
> Total bytes sent: 85
> Total bytes received: 31402231
> 
> sent 85 bytes  received 31402231 bytes  28379.86 bytes/sec
> total size is 1274058729239  speedup is 40572.13
> Return Value: 0
> </snip>
> 
> If no file is transfered, you're measuring only the speed of the
> filelist generation, is that correct? This is always much slower to
> generate the rsync filelist and doesn't show real transfer speed.
> 
> J.H. wrote:
>> On 01/08/2010 01:32 PM, Stephen John Smoogen wrote:
>>> On Thu, Jan 7, 2010 at 9:44 PM, Chris Adams <cmadams at hiwaay.net> wrote:
>>>> Once upon a time, J.H. <warthog19 at eaglescrag.net> said:
>>>>> So here's the graphs for 2009, I've actually got graphs and data going
>>>>> back to 2005 (those are the _all files).  Things actually look really
>>>>> good and as far as I can tell things have actually gotten *better* not
>>>>> worse.
>>>>>
>>>>> It's *possible* the problems people are seeing are nothing but the
>>>>> change in connectivity with the move to the new datacenter, which
>>>>> Fedora
>>>>> isn't going to be able to do much about.
>>>> Well, putting on my ISP network admin hat, if a customer came to us and
>>>> said they saw a drop from 10+ megabits to 2 megabits in downloads
>>>> when a
>>>> remote site moved, I'd say the remote site needs to talk with their new
>>>> provider.  There's no good reason for that to happen between well
>>>> connected networks (barring one end or the other being congested, which
>>>> doesn't appear to be the case here).
>>>>
>>>> I'm curious to see the results of the Red Hat site move this weekend; I
>>>> don't expect to always get the 57 megabits I got on a DVD download test
>>>> this week, but if that also drops to the 2 megabits range, I'll be a
>>>> disappointed Red Hat customer.
>>>
>>> Can you show the netroute you are taking. I usually run into an issue
>>> where it was a cross point 3-4 hops up that was the problem and its
>>> only found after a week or so of handwavy 'things got slower..' , 'not
>>> for me', 'me too', 'hey I am slow now too.' And then it turns out that
>>> some router at Comcast is saying it is a quicker route and you are
>>> going through some backwater network when it should have stayed on
>>> some other one.
>>>
>>> Also it would be good to see how the data was generated for the
>>> graphics (and how it is collected) so that other sites can generate
>>> similar stuff to show whats up.
>>
>> Ask and you shall receive!  The 4th re-write of the log checking script,
>> I don't claim it's pretty since it was a one-off but it will generate
>> data (chktimes2.pl)
>>
>> 'fedora' is an example output of the rsync_fedora.pl script, which
>> happens to be the script we are using (slightly modified) to sync our
>> repositories.
>>
>> plot3.gnu is the file I used to generate the graphs.
>>
>> So if anyone else wants to re-create what I've done they are more than
>> welcome.
>>
>> - John 'Warthog9' Hawley
>>
> 
> 
> 
> 
> --

--