[mirror-admin] download.wpi.edu filesystem corruption vs. not using rsync --checksum

[Chuck Anderson]

During a server upgrade, filesystem corruption was discovered on my
mirror server, download.wpi.edu a.k.a. sinclair.wpi.edu
a.k.a. valen.wpi.edu.  Over the last two days I've been repairing the
damage and everything is back to normal now.

I have checked the RPM checksums/signatures and the ISO/image
checksums and found several bad packages/ISOs.  These have been
replaced from the Fedora master mirror and everything checks out fine
now.  Here is a list of bad files I found:

RPM packages:

./epel/5/SRPMS/koffice-1.6.3-25.20090306svn.el5.src.rpm: (sha1) dsa sha1 MD5 GPG NOT OK
./epel/5/x86_64/debug/dx-debuginfo-4.4.4-3.el5.x86_64.rpm: (sha1) dsa sha1 MD5 GPG NOT OK
./fedora/linux/releases/20/Everything/i386/os/Packages/k/kile-2.1.3-3.fc20.i686.rpm: rsa sha1 (MD5) PGP MD5 NOT OK
./fedora/linux/releases/20/Everything/i386/debug/q/qtcurve-kde4-debuginfo-1.8.14-3.fc20.i686.rpm: rsa sha1 (MD5) PGP MD5 NOT OK
./fedora/linux/releases/20/Everything/armhfp/debug/s/seamonkey-debuginfo-2.21-1.fc20.armv7hl.rpm: rsa sha1 (MD5) PGP MD5 NOT OK
./fedora/linux/releases/18/Everything/x86_64/os/Packages/r/ruby-doc-1.9.3.327-22.fc18.x86_64.rpm: RSA SHA1 (MD5) PGP MD5 NOT OK
./fedora/linux/development/rawhide/source/SRPMS/s/speed-dreams-2.1.0-16.trunk_r4810.fc21.3.src.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/x86_64/os/Packages/n/newt-devel-0.52.17-1.fc21.i686.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/i386/os/Packages/x/xorg-x11-drv-dummy-0.3.6-17.fc21.i686.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/i386/os/Packages/x/xlockmore-gtk-5.43-4.fc21.i686.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/i386/os/Packages/n/newt-devel-0.52.17-1.fc21.i686.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/i386/os/Packages/o/OmegaT-2.6.1-0.11.Beta.fc21.i686.rpm: sha1 MD5 NOT OK
./fedora/linux/development/rawhide/i386/os/Packages/k/kile-2.1.3-3.fc20.i686.rpm: rsa sha1 (MD5) PGP MD5 NOT OK
./fedora/linux/development/rawhide/i386/debug/q/qtcurve-kde4-debuginfo-1.8.14-3.fc20.i686.rpm: rsa sha1 (MD5) PGP MD5 NOT OK
./fedora/linux/updates/19/x86_64/binutils-debuginfo-2.23.52.0.1-9.fc19.x86_64.rpm: rsa sha1 (MD5) PGP MD5 NOT OK

ISO images:

./fedora/linux/releases/18/Fedora/source/iso/Fedora-18-source-DVD.iso

There was a small window of time during which bad files may have been
downloaded before I deactivated access to the mirror.  According to my
logs, the following file transfers have taken place during this
window:

./fedora/linux/development/rawhide/i386/os/Packages/p/poco-foundation-1.4.2p1-2.fc19.4.i686.rpm
./fedora/linux/development/rawhide/i386/os/Packages/p/poco-net-1.4.2p1-2.fc19.4.i686.rpm
./fedora/linux/development/rawhide/i386/os/Packages/p/policycoreutils-gui-2.3-1.fc21.i686.rpm
./fedora/linux/releases/20/Everything/i386/os/Packages/p/poco-foundation-1.4.2p1-2.fc19.4.i686.rpm
./fedora/linux/releases/20/Everything/i386/os/Packages/p/poco-net-1.4.2p1-2.fc19.4.i686.rpm
./fedora/linux/releases/20/Everything/i386/os/Packages/p/poco-foundation-1.4.2p1-2.fc19.4.i686.rpm
./fedora/linux/releases/20/Everything/i386/os/Packages/p/poco-net-1.4.2p1-2.fc19.4.i686.rpm

I would ask that mirror operators who are syncing from
download.wpi.edu please check all the above files as an extra
precaution to be sure they are not corrupt:

rpm -K <package.rpm>
rpm -K <package.drpm>
sha256sum --check *CHECKSUM

This type of manual checking is necessary (but not sufficient--see
below) due to the widespread practice, Fedora documented
recommendations [1], and Fedora enforcement (is this still restricted
on the master mirrors?) that rsync's --checksum option not be used.
Without --checksum, rsync will not repair silent data corruption that
leaves the file the same size and with the same timestamp, since size
+ timestamp are the only things that must match by default to consider
a file "good enough" for rsync to not re-transfer it.

Unfortunately, rpm -K and sha256sum --check are not sufficient to
detect all such silent data corruption, becuase not all contents of
the mirror are covered by RPM checksums/signatures and ISO/Image
checksums.  Hopefully this is good enough in this case though.

Should we consider deploying the --checksum option to rsync?  I know
it slows things down, but by how much I'm not sure.

[1] https://fedoraproject.org/wiki/Infrastructure/Mirroring

--