[mirror-admin] Having to throttle back rsync on download servers

[Kevin Fenzi]

On Wed, 26 Feb 2014 01:31:14 -0500
Chris Schanzle <schanzle at nist.gov> wrote:

...snip...

> I'd like to share some data, ask a few questions, offer suggestions,
> and consider some ideas.

Always welcome. ;) 

...snip...

> Why aren't read-only block level devices (e.g., ISCSI) or GFS2 used
> on the download servers?

Well, we have 5 download servers. They all mount the storage via nfs
from a netapp. If we used iscsi we would have to
use gfs2 or something like it, and I don't think it's performance
would be very nice here, and add a bunch of complexity, additionally,
we mount this volume on some other releng machines to handle syncing
updates to it, etc. NFS is really a lot easier for us in this case. 
 
> Locally, my mirror uses a local filesystem with vfs_cache_pressure
> tuned down to ensure all inodes stay in RAM. This dramatically
> reduces the random I/O it requires to look up inode data.  E.g., it
> takes ~1.5 sec to "du -sh" our fedora repo (715G, ~700k objects, some
> things are excluded).

Yeah, we tweaked that some yesterday as well. It's not clear that it
affects NFS however. ;( We didn't notice a dramatic improvement. 

> Experience has taught me the *last* thing a file server should cache
> is files: it should cache metadata.  File data can stream off fairly
> efficiently with sequential I/O.
> 
> I dream of a time when it will be commonplace to store filesystem
> metadata on a separate, reliable, fast random I/O (today that means
> mirrored SSD) device.

Absoluetely. ;) 
 
> It would be nifty to read about FedoraProject's config and the
> "lessons learned" (especially the mistakes).  It would be very
> helpful for others doing similar work!

yeah, I am sure we will write up something on this when things are more
stable. ;) 

We are now playing around with cachefilesd to see if it could help to
locally cache the nfs data. It seems like this might be a win at least
to even out spikes in common operations. 

We will keep everyone posted on progress...

kevin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.ale.org/pipermail/mirror-admin/attachments/20140226/d44798c8/attachment.sig>
-------------- next part --------------
--