[ale] Godaddy outage

Wed Sep 12 09:55:45 EDT 2012

--
I saw the JSC mission control primary NFS server for every workstation in the
building failover during ascent making all flight monitoring applications
static for 45 seconds. This happened while SRBs were firing just after the
shuttle was launched. The console cabling from the primary server was so tight
that when the main engineer pulled the console 3 inches closer to get a better
view, the cable disconnect caused a failover event.
I didn't cause this, but was sitting a few yards away monitoring a different
server during critical flight phases.
--
Spaces are critical. A month of work was destroyed by a single space error.
* "rm -rf directory*" vs "rm -rf directory *"
--
Elsewhere, I have seen a few $2M/hr outages due to corrupted Oracle tables after
a software vendor told a support guy to type a specific command on the running,
production, dispatching system.  That took about 7 hours to recover and months
to convince upper management that it was a fluke and not the fault of the guy
doing the typing.
--
The problems that I've causes were usually due to bad scripting.
* find+rm is dangerous.
* rsync can destroy a system - especially if this is a backup just prior to an
upgrade
--

I suspect we all have seen some pretty interesting outages over the years.

On 09/11/2012 08:19 PM, simontek at gmail.com wrote:
> Worked at a data center in LA, netzero looped their switch, took out the
> whole network. I assumed, they were an isp, and knew better. They called to
> complain, I had the fun of telling them, I took them offline til they fixed
> their issues. Not fun. Also babysat them, every time they came in to work on
> stuff. Sent via BlackBerry from T-Mobile
> 
> -----Original Message----- From: Jeff Hubbs <jhubbslist at att.net> Sender:
> ale-bounces at ale.org Date: Tue, 11 Sep 2012 20:11:24 To:
> <stephen.r.blevins at gmail.com>; Atlanta Linux Enthusiasts<ale at ale.org> 
> Reply-To: Atlanta Linux Enthusiasts <ale at ale.org> Subject: Re: [ale] Godaddy
> outage
> 
> I once worked at a place where there was a guy who, not meaning to cause 
> trouble, created subdirectory after subdirectory on a Mac until the OS 
> wouldn't function anymore.  We named it the "Copeland Worm" in his honor.
> 
> 
> On 9/11/12 7:59 PM, Stephen R. Blevins wrote:
>> Early in my IT career (early 1980's), I learned that "No malevolent 
>> cracker, no matter how malicious, can even begin to do the damage an 
>> authorized and well-meaning but incompetent user can do."
>> 
>> QED
>> 
>> 
>> Stephen R. Blevins stephen.r.blevins at gmail.com
>> 
>> On 09/11/2012 02:30 PM, Michael H. Warfield wrote:
>>> On Tue, 2012-09-11 at 13:53 -0400, Matt Hessel wrote:
>>>> Well anonymous is claiming they took it down, I don't know if anyone
>>>> at godaddy broke it. :)
>>> NO!
>>> 
>>> First and foremost...  "Anonymous" has not claimed any action.  One 
>>> individual down in Brazil using a handle that has been associated with 
>>> Anonymous has claimed to have done this but stated they were acting 
>>> independently.  The collective has not claimed this and it remains 
>>> unconfirmed.
>>> 
>>> Second...  GoDaddy itself now claims it was not hackers and not a DoS 
>>> attack but a royal screwup in their routers that resulted in corrupted 
>>> routing tables.  I'm not totally sure how much credibility I will lend to
>>> that idea but, if true, this is one of the grandest screwups since 
>>> Microsoft dicked up their DNS years and years ago with all their public 
>>> name servers on a single network segment and then cut them off from the 
>>> private master name server with a firewall update.
>>> 
>>> I'm not sure which is worse.  Being hammered by a collective of malicious
>>> individuals out to get you or displaying a level of technical 
>>> incompetence and inability to follow RFCs and BCPs that would put a 
>>> technotard to shame!  How did they manage to put all their (DNS) eggs in 
>>> one basket so that a single point of failure could have such wide spread 
>>> consequences???  Well, I guess they are on good company.  MS has done it.
>>> AT&T has done it.  Others have done it.  You would think they would know
>>> better but they obviously do not.
>>> 
>>> Regards, Mike
>>> 
>>>> On Sep 11, 2012 1:41 PM, "Scott Plante" <splante at insightsys.com>
>>>> wrote:
>>>>> Yes, we use GoDaddy for registration but not DNS nor hosting and we
>>>>> were unaffected. Our one client who was affected used them for
>>>>> registration and DNS, but not hosting and they were affected. It was
>>>>> just name resolution though, you could still access their externally
>>>>> hosted site by IP of course. I don't know anyone who was hosting with
>>>>> GoDaddy. You couldn't get to godaddy.com but I didn't know their IP
>>>>> to try that.
>>>>> 
>>>>> I imagine someone's in big trouble, if not fired, over that one.
>>>>> 
>>>>> Scott
>>>>> 
>>>>> ------------------------------ *From: *"Brian Stanaland"
>>>>> <brian at stanaland.org> *To: *"Atlanta Linux Enthusiasts"
>>>>> <ale at ale.org> *Sent: *Monday, September 10, 2012 5:51:13 PM *Subject:
>>>>> *Re: [ale] Godaddy outage
>>>>> 
>>>>> I know one group with DNS by GoDaddy but hosting elsewhere has been 
>>>>> affected. All machines are still reachable via IP address, of
>>>>> course. Speaking of which, anyone know if GoDaddy hosted sites can be
>>>>> reached by IP?
>>>>> 
>>>>> --Brian
>>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Sep 10, 2012 at 5:18 PM, Michael H. Warfield
>>>>> <mhw at wittsend.com>wrote:
>>>>> 
>>>>>> On Mon, 2012-09-10 at 15:49 -0400, Scott Plante wrote:
>>>>>>> You guys notice the Godaddy DNS outage? I have a customer' s
>>>>>>> website
>>>>>> down. 
>>>>>> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
>>>>>>
>>>>>>
>>>>>> 
Been following this...  Their DNS servers are impacted.  Hosting servers
>>>>>> indeterminate.  Claims are made that #Anonymous3 down in Brazil is 
>>>>>> behind this for one reason or another but no one else from
>>>>>> Anonymous has stepped up to the plate and claimed responsibility.
>>>>>> Looks to be a loose cannon with a wild hair at this point...
>>>>>> 
>>>>>> If you are using them as a registrar but are managing your own DNS
>>>>>> then you do not seem to be impacted at this time.
>>>>>> 
>>>>>> If you are using their DNS servers then you are probably impacted 
>>>>>> whether you are hosting with them or not.
>>>>>> 
>>>>>> If you are using their hosting services but managing your own DNS, 
>>>>>> please let us know.  I have no data points on this curve.
>>>>>> 
>>>>>>> Scott
>>>>>> Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 |
>>>>>> mhw at WittsEnd.com /\/\|=mhw=|\/\/          | (678) 463-0932 | 
>>>>>> http://www.wittsend.com/mhw/ NIC whois: MHW9          | An optimist
>>>>>> believes we live in the best of all PGP Key: 0x674627FF        |
>>>>>> possible worlds.  A pessimist is sure of it!
>>>>>> 
>>>>>> _______________________________________________ Ale mailing list 
>>>>>> Ale at ale.org http://mail.ale.org/mailman/listinfo/ale See JOBS,
>>>>>> ANNOUNCE and SCHOOLS lists at http://mail.ale.org/mailman/listinfo
>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- Government’s view of the economy could be summed up in a few short
>>>>> phrases: If it moves, tax it. If it keeps moving, regulate it. And if
>>>>> it stops moving, subsidize it. – *Ronald Reagan (1986)
>>>>> 