[mirror-admin] Generating fullfilelist

[Chris Schanzle]

On 03/29/2016 05:15 PM, Carlos wrote:
> Stephen John Smoogen (smooge at gmail.com) wrote on Tue, Mar 29, 2016 at 03:57:09PM BRT:
>> currently fullfilelist gets generated in several areas via a find *
>> command and has the format of:
>>
>>
>> fedora
>> fedora/linux
>>
>> if we use a rsync and put the the file format would be:
>>
>> ==> good-fullfilelist <==
>> drwxr-xr-x        4096 2015/08/20 01:15:38 .
>> -rw-r--r--     1273297 2016/03/28 22:46:34 DIRECTORY_SIZES.txt
>> drwxr-xr-x        4096 2016/02/25 01:54:43 alt
>> drwxrwxr-x        4096 2016/01/29 20:58:53 alt/anaconda
>> -rw-r--r--   486539264 2016/01/29 21:05:07
>> alt/anaconda/rawhide-docker-2-boot.iso
>> -rw-r--r--   488636416 2016/01/21 22:41:23 alt/anaconda/rawhide-docker-boot.iso
>> -rw-rw-r--          90 2016/01/21 22:48:33
>> alt/anaconda/rawhide-docker-boot.iso.sha256
>> drwxrwsr-x        4096 2015/10/02 18:25:17 alt/atomic
>> drwxrwsr-x        4096 2016/01/27 17:46:45 alt/atomic/stable
>> drwxrwsr-x        4096 2016/03/08 19:23:15 alt/atomic/stable/Cloud-Images
>>
>> I can then cut -c44- good-fullfilelist > fullfilelist to get a similar
>> output as the 'find *' however it won't exactly be the same and I want
>> to get an idea of who this will break?
> The paths will of course be the same, only the order can change.
>
> However, cut -c44- may be dangerous because rsync will break the alignment if
> the file size doesn't fit in the allocated space. If this happens fullfilelist
> will be corrupted and make a mess for mirrors.

rsync's output will fit 11 characters for the file size in bytes before shifting columns.  That is in excess of 93 GiB.

Unless Fedora starts distributing sparse files, I don't think this is a concern.


Experimentation via:

cd /dev/shm
for i in 1G 10G 50G 93G 100G; do dd if=/dev/zero of=$i bs=$i seek=1 count=0 2> /dev/null; done
rsync .
drwxrwxrwt         160 2016/03/30 10:17:27 .
-rw-r--r-- 107374182400 2016/03/30 10:17:31 100G
-rw-r--r-- 10737418240 2016/03/30 10:17:31 10G
-rw-r--r--  1073741824 2016/03/30 10:17:31 1G
-rw-r--r-- 53687091200 2016/03/30 10:17:31 50G
-rw-r--r-- 99857989632 2016/03/30 10:17:31 93G

--