[mirror-admin] Having to throttle back rsync on download servers

[Chris Schanzle]

On 02/25/2014 07:14 PM, Stephen John Smoogen wrote:
>
> We are experiencing a problem with the download servers peaking out nfs traffic with our NetApp. We are trying to lower this and looking at some mirrors that are rsyncing from the server over a 100 times a day.

Thanks for investigating and trying to make it better!


I'd like to share some data, ask a few questions, offer suggestions, and consider some ideas.


I notice it often takes a long time for rsync 'receiving file list' even when no files are transferred. The norm this month is 15 to 25 minutes, with numerous occasions 40 mins, an occasional >60 mins, and a few ~120 mins.   That implies metadata lookups are the bottleneck.  E.g., I often see rsync --stats like:

     sent 1.52K bytes  received 29.40M bytes  14.32K bytes/sec

File data often transfers @ 2 to 4 MB/sec, so both our and your network connection is not dramatically overloaded.  Latency (ping) is consistently ~85ms.


I'm surprised that the download servers use NFS, as attribute caching of inode data would be so critical to rsync metadata lookup performance.  With NFS you just toss it away after a specified time passes.  Yet to keep reasonably consistent with updates, you can't tune the NFS clients (download servers) attribute caching too high.

Why aren't read-only block level devices (e.g., ISCSI) or GFS2 used on the download servers?


Locally, my mirror uses a local filesystem with vfs_cache_pressure tuned down to ensure all inodes stay in RAM. This dramatically reduces the random I/O it requires to look up inode data.  E.g., it takes ~1.5 sec to "du -sh" our fedora repo (715G, ~700k objects, some things are excluded).

Experience has taught me the *last* thing a file server should cache is files: it should cache metadata.  File data can stream off fairly efficiently with sequential I/O.

I dream of a time when it will be commonplace to store filesystem metadata on a separate, reliable, fast random I/O (today that means mirrored SSD) device.

It would be nifty to read about FedoraProject's config and the "lessons learned" (especially the mistakes).  It would be very helpful for others doing similar work!

Regards,
Chris

--