[mirror-admin] Please use --delay-updates

[Carlos Carvalho]

Chuck Anderson (cra at wpi.edu) wrote on 16 April 2010 08:41:
 >Each time you run rsync against your upstream mirror, it scans the 
 >entire filesystem to build a filelist.  This could take anywhere from 
 >5 to 20 minutes or more

More... :-(

 >and has been a factor in overloading the master mirrors in the past.

I'd say nowadays too... The table below shows the time we take just to
get the file list from sync.fedoraproject, for the last days. We
mirror everything starting from release 11. It shows clearly that the
machine suffers significantly from disk scanning. The file list is
only about 22MB. Times are in UTC-3.

If fullfilelist was done properly we could completely avoid this
scanning...

Apr 16 09:11:16 | 50m30s  
Apr 16 06:03:59 | 43m27s  
Apr 16 03:07:31 | 46m58s  
Apr 16 00:11:43 | 51m07s  
Apr 15 22:18:12 | 1h00m47s
Apr 15 20:25:30 | 43m43s  
Apr 15 15:03:45 | 43m09s  
Apr 15 12:20:13 | 59m38s  
Apr 15 09:02:02 | 41m12s  
Apr 15 06:17:44 | 57m12s  
Apr 15 02:58:46 | 38m11s  
Apr 15 00:06:10 | 45m29s  
Apr 14 21:10:12 | 46m06s  
Apr 14 14:56:23 | 35m43s  
Apr 14 12:22:25 | 1h01m41s
Apr 14 08:59:46 | 39m04s  
Apr 14 05:52:39 | 31m57s  
Apr 14 03:36:52 | 43m08s  
Apr 13 23:50:23 | 29m44s  
Apr 13 22:07:14 | 36m44s  
Apr 13 20:59:43 | 47m50s  
Apr 13 14:57:43 | 36m49s  
Apr 13 11:55:59 | 35m09s  
Apr 13 09:20:39 | 59m57s  
Apr 13 06:14:18 | 53m49s  
Apr 13 03:11:20 | 50m48s  
Apr 12 23:59:35 | 38m58s  
Apr 12 21:03:20 | 39m34s  
Apr 12 19:16:15 | 28m29s  
Apr 12 14:50:29 | 29m48s  
Apr 12 12:14:13 | 53m37s  
Apr 12 09:07:19 | 46m45s  
Apr 12 06:25:27 | 1h04m58s
Apr 12 02:58:07 | 37m37s  
Apr 11 23:54:17 | 33m44s  
Apr 11 21:04:20 | 43m41s  
Apr 11 17:46:31 | 25m56s  
Apr 11 14:51:15 | 30m43s  
Apr 11 11:49:00 | 28m24s  
Apr 11 08:59:45 | 39m07s  
Apr 11 06:06:50 | 46m20s  
Apr 11 03:08:19 | 47m44s  
Apr 10 23:54:21 | 33m42s  
Apr 10 21:17:33 | 56m56s  
Apr 10 18:01:02 | 40m31s  
Apr 10 16:52:26 | 42m16s  
Apr 10 14:56:53 | 36m18s  
Apr 10 12:15:19 | 54m50s  
Apr 10 05:58:19 | 37m49s  
Apr 10 04:20:35 | 47m20s  
Apr 10 02:30:13 | 55m15s  
Apr  9 18:04:50 | 43m34s  
Apr  9 14:56:19 | 35m37s  
Apr  9 12:06:54 | 46m16s  
Apr  9 09:23:29 | 1h02m45s
Apr  9 06:32:54 | 1h12m20s
Apr  9 03:25:24 | 42m50s  
Apr  8 23:49:47 | 29m11s  
Apr  8 20:43:43 | 23m06s  
Apr  8 17:52:47 | 32m11s  
Apr  8 14:46:50 | 26m11s  
Apr  8 11:52:03 | 31m20s  
Apr  8 08:50:58 | 30m25s  
Apr  8 05:50:28 | 29m57s  
Apr  8 02:50:15 | 29m35s  
Apr  8 00:29:45 | 41m25s  
Apr  7 21:18:04 | 45m14s  
Apr  7 14:59:11 | 38m28s  
Apr  7 12:08:50 | 48m07s  
Apr  7 09:07:02 | 46m13s  
Apr  7 05:56:11 | 35m40s  
Apr  7 02:48:21 | 27m50s  
Apr  6 23:53:07 | 32m33s  
Apr  6 21:12:55 | 46m56s  
Apr  6 18:00:02 | 39m26s  
Apr  6 14:58:57 | 38m23s  
Apr  6 11:52:23 | 31m47s  
Apr  6 08:48:45 | 28m08s  
Apr  6 05:53:17 | 32m48s  
Apr  6 02:44:46 | 24m14s  
Apr  5 23:52:09 | 31m32s  
Apr  5 21:03:11 | 42m40s  
Apr  5 19:16:14 | 31m21s  
Apr  5 14:57:41 | 37m04s  
Apr  5 11:59:46 | 39m09s  
Apr  5 08:51:48 | 31m08s  
Apr  5 05:46:23 | 25m52s  
Apr  5 02:46:36 | 26m05s  
Apr  4 23:47:52 | 27m17s  
Apr  4 21:00:15 | 39m42s  
Apr  4 17:54:45 | 34m10s  
Apr  4 14:43:29 | 22m49s  
Apr  4 11:45:04 | 24m33s  
Apr  4 08:45:28 | 25m00s  
Apr  4 05:42:24 | 21m54s  
Apr  4 02:49:43 | 29m12s  
Apr  3 23:48:53 | 28m20s  
Apr  3 20:52:43 | 32m06s  
Apr  3 17:52:15 | 31m38s  

--