<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.gc-cs-link
{mso-style-name:gc-cs-link;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">You often have to wonder at the thinking of PHBs.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><br>
Letting someone go for a mistake they admitted sends a bad message to the rest of the staff. It is one thing to fire the guy that continually makes mistakes but a one off like the one I described and the one you described wouldn’t be valid. In fact in my
experience most big companies keep morons around far longer than they should and instead do things to avoid the mistakes they made. At one company they decided to quit using NIS because the same dude screwed up which host he did the yppush from on several
occasions. It would have made far more sense to get rid of the guy that repeatedly did this. They later fired one of their smartest admins because he cursed at someone. Attitude outweighs ability in some organizations.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ale-bounces@ale.org [mailto:ale-bounces@ale.org]
<b>On Behalf Of </b>Scott Plante<br>
<b>Sent:</b> Wednesday, September 12, 2012 11:59 AM<br>
<b>To:</b> Atlanta Linux Enthusiasts<br>
<b>Subject:</b> Re: [ale] Godaddy outage<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Arial","sans-serif";color:black">I was at a then Fortune 50 company and their Sys Admin "consultant" from CapGemini deleted their running ERP database. Luckily he realized what he did
and asked for help. I was there and convinced them to just hard power off the machine. When it booted, we were able to get the database segments out of lost+found, inspect the binary header at the top of each file to determine the order, rename the files and
bring up the database with no data loss or corruption. Poor guy was let go, which I thought was too bad because they'd have been screwed if he'd kept his mouth shut. A proper shutdown or even database stop and the files would have been lost, or at least harder
to recover. We were back up and running in <20 minutes.<o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Arial","sans-serif";color:black">
<hr size="2" width="100%" align="center" id="zwchr">
</span></div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif";color:black">From:
</span></b><span style="font-family:"Helvetica","sans-serif";color:black">"Jeff Lightner" <<a href="mailto:JLightner@water.com">JLightner@water.com</a>><br>
<b>To: </b>"Atlanta Linux Enthusiasts" <<a href="mailto:ale@ale.org">ale@ale.org</a>><br>
<b>Sent: </b>Wednesday, September 12, 2012 10:29:50 AM<br>
<b>Subject: </b>Re: [ale] Godaddy outage<br>
<br>
I was at a Fortune 500 once where the main ERP DB went belly up. That caused us to:<br>
1) Declare a disaster and invoke our DR plan which included sending folks and tapes to Philadelphia.<br>
2) Spend time on trying to recover the original Production.<br>
3) Build in house systems to try to figure out what caused the issue.<br>
4) Jump through dozens of other hoops.<br>
<br>
In the end we really couldn't pinpoint what had caused the issue though I suspected fat finger somewhere. The powers that be wouldn't accept this and they kept a team on it for over a year. That team eventually did put out an RCA but I always had my doubts.<br>
<br>
Later one of the DBAs did a fat finger causing another outage and immediately did a mea culpa. He asked if they wanted him to resign and they said no. We were all just so relieved that he had admitted the mistake and saved us all the extra headaches. (As
if recovering a major product DB all by itself isn't headache enough - it takes time even with good backups and logs.)<br>
<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a> <a href="mailto:[mailto:ale-bounces@ale.org]">
[mailto:ale-bounces@ale.org]</a> On Behalf Of JD<br>
Sent: Wednesday, September 12, 2012 9:56 AM<br>
To: Atlanta Linux Enthusiasts<br>
Subject: Re: [ale] Godaddy outage<br>
<br>
--<br>
I saw the JSC mission control primary NFS server for every workstation in the building failover during ascent making all flight monitoring applications static for 45 seconds. This happened while SRBs were firing just after the shuttle was launched. The console
cabling from the primary server was so tight that when the main engineer pulled the console 3 inches closer to get a better view, the cable disconnect caused a failover event.<br>
I didn't cause this, but was sitting a few yards away monitoring a different server during critical flight phases.<br>
--<br>
Spaces are critical. A month of work was destroyed by a single space error.<br>
* "rm -rf directory*" vs "rm -rf directory *"<br>
--<br>
Elsewhere, I have seen a few $2M/hr outages due to corrupted Oracle tables after a software vendor told a support guy to type a specific command on the running, production, dispatching system. That took about 7 hours to recover and months to convince upper
management that it was a fluke and not the fault of the guy doing the typing.<br>
--<br>
The problems that I've causes were usually due to bad scripting.<br>
* find+rm is dangerous.<br>
* rsync can destroy a system - especially if this is a backup just prior to an upgrade<br>
--<br>
<br>
I suspect we all have seen some pretty interesting outages over the years.<br>
<br>
<br>
On 09/11/2012 08:19 PM, <a href="mailto:simontek@gmail.com">simontek@gmail.com</a> wrote:<br>
> Worked at a data center in LA, netzero looped their switch, took out<br>
> the whole network. I assumed, they were an isp, and knew better. They<br>
> called to complain, I had the fun of telling them, I took them offline<br>
> til they fixed their issues. Not fun. Also babysat them, every time<br>
> they came in to work on stuff. Sent via BlackBerry from T-Mobile<br>
><br>
> -----Original Message----- From: Jeff Hubbs <<a href="mailto:jhubbslist@att.net">jhubbslist@att.net</a>> Sender:<br>
> <a href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a> Date: Tue, 11 Sep 2012 20:11:24 To:<br>
> <<a href="mailto:stephen.r.blevins@gmail.com">stephen.r.blevins@gmail.com</a>>; Atlanta Linux Enthusiasts<<a href="mailto:ale@ale.org">ale@ale.org</a>><br>
> Reply-To: Atlanta Linux Enthusiasts <<a href="mailto:ale@ale.org">ale@ale.org</a>> Subject: Re: [ale]<br>
> Godaddy outage<br>
><br>
> I once worked at a place where there was a guy who, not meaning to<br>
> cause trouble, created subdirectory after subdirectory on a Mac until<br>
> the OS wouldn't function anymore. We named it the "Copeland Worm" in his honor.<br>
><br>
><br>
> On 9/11/12 7:59 PM, Stephen R. Blevins wrote:<br>
>> Early in my IT career (early 1980's), I learned that "No malevolent<br>
>> cracker, no matter how malicious, can even begin to do the damage an<br>
>> authorized and well-meaning but incompetent user can do."<br>
>><br>
>> QED<br>
>><br>
>><br>
>> Stephen R. Blevins <a href="mailto:stephen.r.blevins@gmail.com">stephen.r.blevins@gmail.com</a><br>
>><br>
>> On 09/11/2012 02:30 PM, Michael H. Warfield wrote:<br>
>>> On Tue, 2012-09-11 at 13:53 -0400, Matt Hessel wrote:<br>
>>>> Well anonymous is claiming they took it down, I don't know if<br>
>>>> anyone at godaddy broke it. :)<br>
>>> NO!<br>
>>><br>
>>> First and foremost... "Anonymous" has not claimed any action. One<br>
>>> individual down in Brazil using a handle that has been associated<br>
>>> with Anonymous has claimed to have done this but stated they were<br>
>>> acting independently. The collective has not claimed this and it<br>
>>> remains unconfirmed.<br>
>>><br>
>>> Second... GoDaddy itself now claims it was not hackers and not a<br>
>>> DoS attack but a royal screwup in their routers that resulted in<br>
>>> corrupted routing tables. I'm not totally sure how much credibility<br>
>>> I will lend to that idea but, if true, this is one of the grandest<br>
>>> screwups since Microsoft dicked up their DNS years and years ago<br>
>>> with all their public name servers on a single network segment and<br>
>>> then cut them off from the private master name server with a firewall update.<br>
>>><br>
>>> I'm not sure which is worse. Being hammered by a collective of<br>
>>> malicious individuals out to get you or displaying a level of<br>
>>> technical incompetence and inability to follow RFCs and BCPs that<br>
>>> would put a technotard to shame! How did they manage to put all<br>
>>> their (DNS) eggs in one basket so that a single point of failure<br>
>>> could have such wide spread consequences??? Well, I guess they are on good company. MS has done it.<br>
>>> AT&T has done it. Others have done it. You would think they would<br>
>>> know better but they obviously do not.<br>
>>><br>
>>> Regards, Mike<br>
>>><br>
>>>> On Sep 11, 2012 1:41 PM, "Scott Plante" <<a href="mailto:splante@insightsys.com">splante@insightsys.com</a>><br>
>>>> wrote:<br>
>>>>> Yes, we use GoDaddy for registration but not DNS nor hosting and<br>
>>>>> we were unaffected. Our one client who was affected used them for<br>
>>>>> registration and DNS, but not hosting and they were affected. It<br>
>>>>> was just name resolution though, you could still access their<br>
>>>>> externally hosted site by IP of course. I don't know anyone who<br>
>>>>> was hosting with GoDaddy. You couldn't get to godaddy.com but I<br>
>>>>> didn't know their IP to try that.<br>
>>>>><br>
>>>>> I imagine someone's in big trouble, if not fired, over that one.<br>
>>>>><br>
>>>>> Scott<br>
>>>>><br>
>>>>> ------------------------------ *From: *"Brian Stanaland"<br>
>>>>> <<a href="mailto:brian@stanaland.org">brian@stanaland.org</a>> *To: *"Atlanta Linux Enthusiasts"<br>
>>>>> <<a href="mailto:ale@ale.org">ale@ale.org</a>> *Sent: *Monday, September 10, 2012 5:51:13 PM *Subject:<br>
>>>>> *Re: [ale] Godaddy outage<br>
>>>>><br>
>>>>> I know one group with DNS by GoDaddy but hosting elsewhere has<br>
>>>>> been affected. All machines are still reachable via IP address, of<br>
>>>>> course. Speaking of which, anyone know if GoDaddy hosted sites can<br>
>>>>> be reached by IP?<br>
>>>>><br>
>>>>> --Brian<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Mon, Sep 10, 2012 at 5:18 PM, Michael H. Warfield<br>
>>>>> <<a href="mailto:mhw@wittsend.com">mhw@wittsend.com</a>>wrote:<br>
>>>>><br>
>>>>>> On Mon, 2012-09-10 at 15:49 -0400, Scott Plante wrote:<br>
>>>>>>> You guys notice the Godaddy DNS outage? I have a customer' s<br>
>>>>>>> website<br>
>>>>>> down.<br>
>>>>>> <a href="http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millio">
http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millio</a><br>
>>>>>> ns-of-sites/<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
Been following this... Their DNS servers are impacted. Hosting servers<br>
>>>>>> indeterminate. Claims are made that #Anonymous3 down in Brazil<br>
>>>>>> is behind this for one reason or another but no one else from<br>
>>>>>> Anonymous has stepped up to the plate and claimed responsibility.<br>
>>>>>> Looks to be a loose cannon with a wild hair at this point...<br>
>>>>>><br>
>>>>>> If you are using them as a registrar but are managing your own<br>
>>>>>> DNS then you do not seem to be impacted at this time.<br>
>>>>>><br>
>>>>>> If you are using their DNS servers then you are probably impacted<br>
>>>>>> whether you are hosting with them or not.<br>
>>>>>><br>
>>>>>> If you are using their hosting services but managing your own<br>
>>>>>> DNS, please let us know. I have no data points on this curve.<br>
>>>>>><br>
>>>>>>> Scott<br>
>>>>>> Regards, Mike -- Michael H. Warfield (AI4NB) | <span class="gc-cs-link">(770) 985-6132</span> |<br>
>>>>>> <a href="mailto:mhw@WittsEnd.com">mhw@WittsEnd.com</a> /\/\|=mhw=|\/\/ |
<span class="gc-cs-link">(678) 463-0932</span> |<br>
>>>>>> <a href="http://www.wittsend.com/mhw/">http://www.wittsend.com/mhw/</a> NIC whois: MHW9 | An optimist<br>
>>>>>> believes we live in the best of all PGP Key: 0x674627FF |<br>
>>>>>> possible worlds. A pessimist is sure of it!<br>
>>>>>><br>
>>>>>> _______________________________________________ Ale mailing list<br>
>>>>>> <a href="mailto:Ale@ale.org">Ale@ale.org</a> <a href="http://mail.ale.org/mailman/listinfo/ale">
http://mail.ale.org/mailman/listinfo/ale</a> See JOBS,<br>
>>>>>> ANNOUNCE and SCHOOLS lists at<br>
>>>>>> <a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>> -- Government's view of the economy could be summed up in a few<br>
>>>>> short<br>
>>>>> phrases: If it moves, tax it. If it keeps moving, regulate it. And<br>
>>>>> if it stops moving, subsidize it. - *Ronald Reagan (1986)<br>
>>>>><br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br>
<br>
<br>
<br>
<br>
Athena(r), Created for the Cause(tm)<br>
Making a Difference in the Fight Against Breast Cancer<br>
<br>
---------------------------------<br>
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information
is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.<br>
----------------------------------<br>
<br>
<br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>