[mirror-admin] Slow syncing?

[Marek Mahut]

Hello J.H.,

Looking at your log:

<snip>
receiving file list ... done

Number of files: 827975
Number of files transferred: 0
Total file size: 1274058729239 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 31402197
File list generation time: 1085.205 seconds
File list transfer time: 0.001 seconds
Total bytes sent: 85
Total bytes received: 31402231

sent 85 bytes  received 31402231 bytes  28379.86 bytes/sec
total size is 1274058729239  speedup is 40572.13
Return Value: 0
</snip>

If no file is transfered, you're measuring only the speed of the
filelist generation, is that correct? This is always much slower to
generate the rsync filelist and doesn't show real transfer speed.

J.H. wrote:
> On 01/08/2010 01:32 PM, Stephen John Smoogen wrote:
>> On Thu, Jan 7, 2010 at 9:44 PM, Chris Adams <cmadams at hiwaay.net> wrote:
>>> Once upon a time, J.H. <warthog19 at eaglescrag.net> said:
>>>> So here's the graphs for 2009, I've actually got graphs and data going
>>>> back to 2005 (those are the _all files).  Things actually look really
>>>> good and as far as I can tell things have actually gotten *better* not
>>>> worse.
>>>>
>>>> It's *possible* the problems people are seeing are nothing but the
>>>> change in connectivity with the move to the new datacenter, which Fedora
>>>> isn't going to be able to do much about.
>>> Well, putting on my ISP network admin hat, if a customer came to us and
>>> said they saw a drop from 10+ megabits to 2 megabits in downloads when a
>>> remote site moved, I'd say the remote site needs to talk with their new
>>> provider.  There's no good reason for that to happen between well
>>> connected networks (barring one end or the other being congested, which
>>> doesn't appear to be the case here).
>>>
>>> I'm curious to see the results of the Red Hat site move this weekend; I
>>> don't expect to always get the 57 megabits I got on a DVD download test
>>> this week, but if that also drops to the 2 megabits range, I'll be a
>>> disappointed Red Hat customer.
>>
>> Can you show the netroute you are taking. I usually run into an issue
>> where it was a cross point 3-4 hops up that was the problem and its
>> only found after a week or so of handwavy 'things got slower..' , 'not
>> for me', 'me too', 'hey I am slow now too.' And then it turns out that
>> some router at Comcast is saying it is a quicker route and you are
>> going through some backwater network when it should have stayed on
>> some other one.
>>
>> Also it would be good to see how the data was generated for the
>> graphics (and how it is collected) so that other sites can generate
>> similar stuff to show whats up.
> 
> Ask and you shall receive!  The 4th re-write of the log checking script,
> I don't claim it's pretty since it was a one-off but it will generate
> data (chktimes2.pl)
> 
> 'fedora' is an example output of the rsync_fedora.pl script, which
> happens to be the script we are using (slightly modified) to sync our
> repositories.
> 
> plot3.gnu is the file I used to generate the graphs.
> 
> So if anyone else wants to re-create what I've done they are more than
> welcome.
> 
> - John 'Warthog9' Hawley
> 


-- 
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/

Marek Mahut                                               Red Hat IT
Red Hat Czech                                 Tel.: +420-532-294-666
Purkynova 99                                   GSM: +420-739-636-703
612 00 Brno                                 Email: mmahut at redhat.com
Czech Republic                             Web: http://cz.redhat.com
____________________________________________________________________

Registered Address: Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno,
Czech Republic
Registered in Brno under identification number CZ27690016

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20100111/6220fde8/attachment.bin 
-------------- next part --------------
--