[mirror-admin] EPEL is gone

[Mike McGrath]

On Fri, 2 Oct 2009, Mike McGrath wrote:

> On Thu, 1 Oct 2009, Mike McGrath wrote:
>
> > One of our scripts seems to have gone nuts and deleted a couple of EPEL
> > dirs.  If you wouldn't mind disabling your epel sync scripts until we have
> > corrected the issue.
> >
>
> Ok, so it wasn't all of epel, but lots of it.  We seem to have it back in
> place now and it should be safe to sync again.  Expect a post mortem
> tomorrow morning when we have all the details.
>

As promised.  Though our updates process we run a number of scripts and
jobs, some happen automatically.  One of these is a clean_repo job.  This
job runs 7 days after one of our processes has been started (it has an
internal scheduler)

Well this process A) hadn't run longer then 7 days in a very long while
and B) was not yet configured to handle EPEL which only recently came
under this systems control.  As a result, it deleted the updates.  Then
around 19:45 (give or take 30 minutes) a cron job attempted to sync this
empty directory with the mirror and thus deleted a lot of content.

The clean_repo job has been disabled until it can be updated.  Sorry for
the trouble.  I think most of the users didn't see too terrible of a
disruption.

	-Mike

--