<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 09/12/2012 11:58 AM, Scott Plante
wrote:<br>
</div>
<blockquote
cite="mid:400103a9-e43a-48ee-9f47-f8d182447ddd@mercury.isint"
type="cite">
<style type="text/css">p { margin: 0; }</style>
<div style="font-family: Arial; font-size: 12pt; color: #000000">I
was at a then Fortune 50 company and their Sys Admin
"consultant" from CapGemini deleted their running ERP database.
Luckily he realized what he did and asked for help. I was there
and convinced them to just hard power off the machine. When it
booted, we were able to get the database segments out of
lost+found, inspect the binary header at the top of each file to
determine the order, rename the files and bring up the database
with no data loss or corruption. Poor guy was let go, which I
thought was too bad because they'd have been screwed if he'd
kept his mouth shut. A proper shutdown or even database stop and
the files would have been lost, or at least harder to recover.
We were back up and running in <20 minutes.<br>
</div>
</blockquote>
<br>
The next time somebody does something dumb they will not be so
lucky. No one will take the blame and they will not know what really
happened.<br>
<br>
<blockquote
cite="mid:400103a9-e43a-48ee-9f47-f8d182447ddd@mercury.isint"
type="cite">
<div style="font-family: Arial; font-size: 12pt; color: #000000"><br>
<hr id="zwchr">
<div
style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
</b>"Jeff Lightner" <a class="moz-txt-link-rfc2396E" href="mailto:JLightner@water.com"><JLightner@water.com></a><br>
<b>To: </b>"Atlanta Linux Enthusiasts" <a class="moz-txt-link-rfc2396E" href="mailto:ale@ale.org"><ale@ale.org></a><br>
<b>Sent: </b>Wednesday, September 12, 2012 10:29:50 AM<br>
<b>Subject: </b>Re: [ale] Godaddy outage<br>
<br>
I was at a Fortune 500 once where the main ERP DB went belly
up. That caused us to:<br>
1) Declare a disaster and invoke our DR plan which included
sending folks and tapes to Philadelphia.<br>
2) Spend time on trying to recover the original Production.<br>
3) Build in house systems to try to figure out what caused
the issue.<br>
4) Jump through dozens of other hoops.<br>
<br>
In the end we really couldn't pinpoint what had caused the
issue though I suspected fat finger somewhere. The powers
that be wouldn't accept this and they kept a team on it for
over a year. That team eventually did put out an RCA but I
always had my doubts.<br>
<br>
Later one of the DBAs did a fat finger causing another outage
and immediately did a mea culpa. He asked if they wanted him
to resign and they said no. We were all just so relieved that
he had admitted the mistake and saved us all the extra
headaches. (As if recovering a major product DB all by itself
isn't headache enough - it takes time even with good backups
and logs.)<br>
<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: <a class="moz-txt-link-abbreviated" href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a> [<a class="moz-txt-link-freetext" href="mailto:ale-bounces@ale.org">mailto:ale-bounces@ale.org</a>] On
Behalf Of JD<br>
Sent: Wednesday, September 12, 2012 9:56 AM<br>
To: Atlanta Linux Enthusiasts<br>
Subject: Re: [ale] Godaddy outage<br>
<br>
--<br>
I saw the JSC mission control primary NFS server for every
workstation in the building failover during ascent making all
flight monitoring applications static for 45 seconds. This
happened while SRBs were firing just after the shuttle was
launched. The console cabling from the primary server was so
tight that when the main engineer pulled the console 3 inches
closer to get a better view, the cable disconnect caused a
failover event.<br>
I didn't cause this, but was sitting a few yards away
monitoring a different server during critical flight phases.<br>
--<br>
Spaces are critical. A month of work was destroyed by a single
space error.<br>
* "rm -rf directory*" vs "rm -rf directory *"<br>
--<br>
Elsewhere, I have seen a few $2M/hr outages due to corrupted
Oracle tables after a software vendor told a support guy to
type a specific command on the running, production,
dispatching system. That took about 7 hours to recover and
months to convince upper management that it was a fluke and
not the fault of the guy doing the typing.<br>
--<br>
The problems that I've causes were usually due to bad
scripting.<br>
* find+rm is dangerous.<br>
* rsync can destroy a system - especially if this is a backup
just prior to an upgrade<br>
--<br>
<br>
I suspect we all have seen some pretty interesting outages
over the years.<br>
<br>
<br>
On 09/11/2012 08:19 PM, <a class="moz-txt-link-abbreviated" href="mailto:simontek@gmail.com">simontek@gmail.com</a> wrote:<br>
> Worked at a data center in LA, netzero looped their
switch, took out<br>
> the whole network. I assumed, they were an isp, and knew
better. They<br>
> called to complain, I had the fun of telling them, I took
them offline<br>
> til they fixed their issues. Not fun. Also babysat them,
every time<br>
> they came in to work on stuff. Sent via BlackBerry from
T-Mobile<br>
><br>
> -----Original Message----- From: Jeff Hubbs
<a class="moz-txt-link-rfc2396E" href="mailto:jhubbslist@att.net"><jhubbslist@att.net></a> Sender:<br>
> <a class="moz-txt-link-abbreviated" href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a> Date: Tue, 11 Sep 2012 20:11:24 To:<br>
> <a class="moz-txt-link-rfc2396E" href="mailto:stephen.r.blevins@gmail.com"><stephen.r.blevins@gmail.com></a>; Atlanta Linux
Enthusiasts<a class="moz-txt-link-rfc2396E" href="mailto:ale@ale.org"><ale@ale.org></a><br>
> Reply-To: Atlanta Linux Enthusiasts <a class="moz-txt-link-rfc2396E" href="mailto:ale@ale.org"><ale@ale.org></a>
Subject: Re: [ale]<br>
> Godaddy outage<br>
><br>
> I once worked at a place where there was a guy who, not
meaning to<br>
> cause trouble, created subdirectory after subdirectory on
a Mac until<br>
> the OS wouldn't function anymore. We named it the
"Copeland Worm" in his honor.<br>
><br>
><br>
> On 9/11/12 7:59 PM, Stephen R. Blevins wrote:<br>
>> Early in my IT career (early 1980's), I learned that
"No malevolent<br>
>> cracker, no matter how malicious, can even begin to
do the damage an<br>
>> authorized and well-meaning but incompetent user can
do."<br>
>><br>
>> QED<br>
>><br>
>><br>
>> Stephen R. Blevins <a class="moz-txt-link-abbreviated" href="mailto:stephen.r.blevins@gmail.com">stephen.r.blevins@gmail.com</a><br>
>><br>
>> On 09/11/2012 02:30 PM, Michael H. Warfield wrote:<br>
>>> On Tue, 2012-09-11 at 13:53 -0400, Matt Hessel
wrote:<br>
>>>> Well anonymous is claiming they took it down,
I don't know if<br>
>>>> anyone at godaddy broke it. :)<br>
>>> NO!<br>
>>><br>
>>> First and foremost... "Anonymous" has not
claimed any action. One<br>
>>> individual down in Brazil using a handle that has
been associated<br>
>>> with Anonymous has claimed to have done this but
stated they were<br>
>>> acting independently. The collective has not
claimed this and it<br>
>>> remains unconfirmed.<br>
>>><br>
>>> Second... GoDaddy itself now claims it was not
hackers and not a<br>
>>> DoS attack but a royal screwup in their routers
that resulted in<br>
>>> corrupted routing tables. I'm not totally sure
how much credibility<br>
>>> I will lend to that idea but, if true, this is
one of the grandest<br>
>>> screwups since Microsoft dicked up their DNS
years and years ago<br>
>>> with all their public name servers on a single
network segment and<br>
>>> then cut them off from the private master name
server with a firewall update.<br>
>>><br>
>>> I'm not sure which is worse. Being hammered by a
collective of<br>
>>> malicious individuals out to get you or
displaying a level of<br>
>>> technical incompetence and inability to follow
RFCs and BCPs that<br>
>>> would put a technotard to shame! How did they
manage to put all<br>
>>> their (DNS) eggs in one basket so that a single
point of failure<br>
>>> could have such wide spread consequences???
Well, I guess they are on good company. MS has done it.<br>
>>> AT&T has done it. Others have done it. You
would think they would<br>
>>> know better but they obviously do not.<br>
>>><br>
>>> Regards, Mike<br>
>>><br>
>>>> On Sep 11, 2012 1:41 PM, "Scott Plante"
<a class="moz-txt-link-rfc2396E" href="mailto:splante@insightsys.com"><splante@insightsys.com></a><br>
>>>> wrote:<br>
>>>>> Yes, we use GoDaddy for registration but
not DNS nor hosting and<br>
>>>>> we were unaffected. Our one client who
was affected used them for<br>
>>>>> registration and DNS, but not hosting and
they were affected. It<br>
>>>>> was just name resolution though, you
could still access their<br>
>>>>> externally hosted site by IP of course. I
don't know anyone who<br>
>>>>> was hosting with GoDaddy. You couldn't
get to godaddy.com but I<br>
>>>>> didn't know their IP to try that.<br>
>>>>><br>
>>>>> I imagine someone's in big trouble, if
not fired, over that one.<br>
>>>>><br>
>>>>> Scott<br>
>>>>><br>
>>>>> ------------------------------ *From:
*"Brian Stanaland"<br>
>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:brian@stanaland.org"><brian@stanaland.org></a> *To:
*"Atlanta Linux Enthusiasts"<br>
>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:ale@ale.org"><ale@ale.org></a> *Sent: *Monday,
September 10, 2012 5:51:13 PM *Subject:<br>
>>>>> *Re: [ale] Godaddy outage<br>
>>>>><br>
>>>>> I know one group with DNS by GoDaddy but
hosting elsewhere has<br>
>>>>> been affected. All machines are still
reachable via IP address, of<br>
>>>>> course. Speaking of which, anyone know if
GoDaddy hosted sites can<br>
>>>>> be reached by IP?<br>
>>>>><br>
>>>>> --Brian<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Mon, Sep 10, 2012 at 5:18 PM, Michael
H. Warfield<br>
>>>>> <a class="moz-txt-link-rfc2396E" href="mailto:mhw@wittsend.com"><mhw@wittsend.com></a>wrote:<br>
>>>>><br>
>>>>>> On Mon, 2012-09-10 at 15:49 -0400,
Scott Plante wrote:<br>
>>>>>>> You guys notice the Godaddy DNS
outage? I have a customer' s<br>
>>>>>>> website<br>
>>>>>> down.<br>
>>>>>>
<a class="moz-txt-link-freetext" href="http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millio">http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millio</a><br>
>>>>>> ns-of-sites/<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
Been following this... Their DNS servers are impacted.
Hosting servers<br>
>>>>>> indeterminate. Claims are made that
#Anonymous3 down in Brazil<br>
>>>>>> is behind this for one reason or
another but no one else from<br>
>>>>>> Anonymous has stepped up to the plate
and claimed responsibility.<br>
>>>>>> Looks to be a loose cannon with a
wild hair at this point...<br>
>>>>>><br>
>>>>>> If you are using them as a registrar
but are managing your own<br>
>>>>>> DNS then you do not seem to be
impacted at this time.<br>
>>>>>><br>
>>>>>> If you are using their DNS servers
then you are probably impacted<br>
>>>>>> whether you are hosting with them or
not.<br>
>>>>>><br>
>>>>>> If you are using their hosting
services but managing your own<br>
>>>>>> DNS, please let us know. I have no
data points on this curve.<br>
>>>>>><br>
>>>>>>> Scott<br>
<span>>>>>>> Regards, Mike -- Michael H.
Warfield (AI4NB) | <span id="gc-number-0"
class="gc-cs-link" title="Call with Google Voice">(770)
985-6132</span> |</span><br>
<span>>>>>>> <a class="moz-txt-link-abbreviated" href="mailto:mhw@WittsEnd.com">mhw@WittsEnd.com</a>
/\/\|=mhw=|\/\/ | <span id="gc-number-1"
class="gc-cs-link" title="Call with Google Voice">(678)
463-0932</span> |</span><br>
>>>>>> <a class="moz-txt-link-freetext" href="http://www.wittsend.com/mhw/">http://www.wittsend.com/mhw/</a> NIC
whois: MHW9 | An optimist<br>
>>>>>> believes we live in the best of all
PGP Key: 0x674627FF |<br>
>>>>>> possible worlds. A pessimist is sure
of it!<br>
>>>>>><br>
>>>>>>
_______________________________________________ Ale mailing
list<br>
>>>>>> <a class="moz-txt-link-abbreviated" href="mailto:Ale@ale.org">Ale@ale.org</a>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a> See JOBS,<br>
>>>>>> ANNOUNCE and SCHOOLS lists at<br>
>>>>>> <a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>> -- Government's view of the economy could
be summed up in a few<br>
>>>>> short<br>
>>>>> phrases: If it moves, tax it. If it keeps
moving, regulate it. And<br>
>>>>> if it stops moving, subsidize it. -
*Ronald Reagan (1986)<br>
>>>>><br>
_______________________________________________<br>
Ale mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br>
<br>
<br>
<br>
<br>
Athena(r), Created for the Cause(tm)<br>
Making a Difference in the Fight Against Breast Cancer<br>
<br>
---------------------------------<br>
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or
confidential information and is for the sole use of the
intended recipient(s). If you are not the intended recipient,
any disclosure, copying, distribution, or use of the contents
of this information is prohibited and may be unlawful. If you
have received this electronic transmission in error, please
reply immediately to the sender that you have received the
message in error, and delete it. Thank you.<br>
----------------------------------<br>
<br>
<br>
_______________________________________________<br>
Ale mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Ale mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Ale@ale.org">Ale@ale.org</a>
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a>
See JOBS, ANNOUNCE and SCHOOLS lists at
<a class="moz-txt-link-freetext" href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Jay Lozier
<a class="moz-txt-link-abbreviated" href="mailto:jslozier@gmail.com">jslozier@gmail.com</a></pre>
</body>
</html>