[mirror-admin] redundant transfers testing->updates

[Chris Schanzle]

On 01/19/2012 10:04 PM, Chris Schanzle wrote:
> On 01/19/2012 08:58 PM, Dennis Gilmore wrote:
>> El Thu, 19 Jan 2012 12:57:02 -0500
>> Chris Schanzle<schanzle at nist.gov>   escribió:
>>> On 07/13/2011 09:24 AM, Chris Schanzle wrote:
>>>> A friendly suggestion...
>>>>
>>>> For mirrors that carry testing+updates, as packages move from
>>>> testing to updates, they are transferred again as new files in
>>>> updates.  It would be a significant reduction in traffic if
>>>> packages could be be hardlinked from testing to updates, then
>>>> removed from testing after a reasonable delay (24hrs?).
>>>
>>>
>>> The recent hardlinking for directory hashing refreshed my memory from
>>> the above suggestion, which had no further discussion.  Stupid idea
>>> or decided not worth implementing due to infrastructure design issues?
>>>
>>> --
>>
>> this was supposed to b happening. maybe i need to add a manual hardlink
>> or --link-dest on the scripts that push bits to the mirrors.
>>
>> Thanks for the reminder.
>>
>> Dennis
>
> Thanks!  Don't forget to watch out for hardlink failures due to just permissions being different (I think that got nipped a while back).
>
> Here's a sample package that came through the process which shows the landing in updates/testing, later promoted to updates, but was not hardlinked first and was removed from updates/testing same day it landed in updates/.
>
> $ grep pvm-gui-3.4.6-1.fc16 fedora-rsync-{201112,201201}.log
> /local/tmp/fedora-rsync-201112.log:2011/12/30 02:55:47 [9567]>f+++++++++ updates/testing/16/i386/pvm-gui-3.4.6-1.fc16.i686.rpm
> /local/tmp/fedora-rsync-201112.log:2011/12/30 03:01:15 [9567]>f+++++++++ updates/testing/16/x86_64/pvm-gui-3.4.6-1.fc16.x86_64.rpm
> /local/tmp/fedora-rsync-201201.log:2012/01/12 03:17:34 [15367]>f+++++++++ updates/16/i386/pvm-gui-3.4.6-1.fc16.i686.rpm
> /local/tmp/fedora-rsync-201201.log:2012/01/12 03:24:04 [15367]>f+++++++++ updates/16/x86_64/pvm-gui-3.4.6-1.fc16.x86_64.rpm
> /local/tmp/fedora-rsync-201201.log:2012/01/12 03:28:17 [15361] *deleting   updates/testing/16/i386/pvm-gui-3.4.6-1.fc16.i686.rpm
> /local/tmp/fedora-rsync-201201.log:2012/01/12 03:29:08 [15361] *deleting   updates/testing/16/x86_64/pvm-gui-3.4.6-1.fc16.x86_64.rpm
>
> Again, rsync may end up deleting things first depending on sort order and --delete-{before,after} options, so it's important to have an overlap of time where when a package is promoted from testing to updates, it remains in testing for a while (perhaps a day).
>
> Best regards,
> Chris


Friendly reminder I still see the same behavior as packages migrate from testing to updates, they are transferred as new files.  The testing files must stick around for a while so that rsync will find them as hard-links.

$ egrep  '16/.*/fping-3.0-1' /local/tmp/fedora-rsync-20120[23].log
/local/tmp/fedora-rsync-201202.log:2012/02/27 00:53:31 [17524] >f+++++++++ 139735 updates/testing/16/SRPMS/fping-3.0-1.fc16.src.rpm
/local/tmp/fedora-rsync-201202.log:2012/02/27 00:53:41 [17524] >f+++++++++ 32205 updates/testing/16/i386/fping-3.0-1.fc16.i686.rpm
/local/tmp/fedora-rsync-201202.log:2012/02/27 00:54:05 [17524] >f+++++++++ 34285 updates/testing/16/x86_64/fping-3.0-1.fc16.x86_64.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 04:11:01 [15918] >f+++++++++ 139735 updates/16/SRPMS/fping-3.0-1.fc16.src.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 04:17:16 [15918] >f+++++++++ 32205 updates/16/i386/fping-3.0-1.fc16.i686.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 04:59:37 [15918] >f+++++++++ 34285 updates/16/x86_64/fping-3.0-1.fc16.x86_64.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 06:38:19 [15916] *deleting   0 updates/testing/16/SRPMS/fping-3.0-1.fc16.src.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 06:38:19 [15916] *deleting   0 updates/testing/16/i386/fping-3.0-1.fc16.i686.rpm
/local/tmp/fedora-rsync-201203.log:2012/03/07 06:38:24 [15916] *deleting   0 updates/testing/16/x86_64/fping-3.0-1.fc16.x86_64.rpm

Thanks!

--