[mirror-admin] Large amount of content removed - issue with master mirror or quick-fedora-mirror script?

[Jaspn Tibbitts]

HEAnet Mirror Admin <rh-mirror-list at heanet.ie> writes:

> I noticed our fedora mirror was practically emptied on Thursday
> morning, it's resynced now but I'm trying to figure out why everything
> was deleted in order to avoid any future occurances.

There's a reason it hasn't actually been announced.....  (Well, that and
I'm on vacation in Europe and so won't be able to be as responsive as I
would otherwise be.)

My current understanding of the issue is that the file lists are
currently copied into place on the master mirror.  This means that
there's a window where you can fetch an incomplete file list, which gets
processed as normal.  This ends up with a whole bunch of files on the
client which don't appear to exist on the server.

There's an issue open on this in pagure.  The first step is to at least
verify that the checksums section is actually present before processing
anything, since the server code will always add it last (and I can
implement changes in the client more easily).   But on the server side,
these files need to be renamed into place.  The example wrapper code I
provided is actually being used verbatim, so I really need to make it
less "example" and more "production-ready".  I can also extend the
format with an end marker of some type.

> We have been using quick-fedora-mirror* for a while now, and this is the
> first time this has happened. quick-fedora-mirror works really well
> and makes our syncs much faster, I'd like to keep using it so I'm
> looking to determine if this was a SNAFU with the quick-fedora-mirror
> script itself, the master site at dl.fedoraproject.org or some
> combination of both? Or mabye something in our own setup?

It's not you; it has happened to me as well.  Exactly twice, to be
exact; I thought I understood what happened the first time and fixed
that, but it hit me again just the other day.  Maybe even at the same
time it hit you.  So I've been going over everything again to make sure
I understand the cause this time.

But I also have three complete mirrors here that I use for testing (and
which aren't used for anything else) so such an incident is not a huge
deal for me.  I wasn't aware that anyone else was seriously using the
code (besides one place where Fedora is using it internally).

> Is it possible for this file to be empty for a brief period?

It can't be completely empty; it's parsed for the version section and
will fail if that's not present.  But there's no check for "did it get
the whole file", and sadly the format doesn't make it easy.

I should also consider a cap on the number of deletions permitted in one
run, but any such limit is likely to trigger when F22 goes to the archive.

> Unfortunately the temporary data that quick-fedora-mirror generates is
> gone, but I do have a detailed log file that I can send on if required.

It's not deleted if you have debugging high enough, but the file lists
are so large that you simply can't keep a lot of them around unless you
have a very large /tmp or configure those files to go elsewhere.

 - J<

--