[ale] Little OT: Bad Linux Sysadmin Practices

[Lightner, Jeff]

After the "missing" backup issue they fully embraced the original plan they'd previously nixed.

SLAs weren't really in the vernacular until some time later and I'll have to say that once the company started focusing on SLAs (even between departments) more decisions were made based on SLA requirements than pure dollar requirements.

By the way in a former life I WAS a bean counter for a long time and had worked my way up to Financial Controller.   You are wrong to blame the bean counters for poor decisions like this - it is almost always operational managers trying to look good on paper that do this.   More than once in my former career I was the one pushing operations to make "smart" money decisions rather than "instant savings".    Many decisions made by operational types are made to meet this month's budget to the cost of this quarter's or this quarter's to the cost of this year's or this year's to long term stability.

It is even worse in publically traded companies where everything is judged on today's stock price rather than underlying company value.    There's a reason so many lost money in the Enron debacle despite the fact that what they were publishing in their required financial reports (especially the notes) would have let any person with a little "bean counter" training know how badly they were setup.   The problem is nobody wants to believe you when you tell them things are overvalued.  I remember how badly Warren Buffet was chastised by financial media pundits because he refused to by into the tech boom of the 90s.   The fact he had become a billionaire making smart decisions seemed to escape them.   When the bust finally hit and all the dot coms went belly up Berkshire Hathaway quietly went on about their business of making money based on value.   If I'd been Warren I'd have been using some of my billions to take out full page ads saying "I told you so!".  :-)

For years co-workers derided me for buying company stock at a place I worked but when the tech bust finally came that company stock turned out to be my second best investment because it did have intrinsic value.   (In fact when they sold out the company I was annoyed because I had to take the capital gains tax hit as they went private on the sell so I got cash rather than stock as in most mergers.)





-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Jay Lozier
Sent: Friday, October 12, 2012 10:03 AM
To: ale at ale.org
Subject: Re: [ale] Little OT: Bad Linux Sysadmin Practices

On 10/12/2012 09:30 AM, Lightner, Jeff wrote:
> I worked for a company that had a large data center in Little Rock which had both the telecom systems and the banking/mortgage systems (mainframe and open systems) for multiple telecom, banking and finance companies (at the time 20% of mortgages in the US went through these systems).
>
> On two separate occasions while doing supposedly "planned" maintenance they took down the whole data center accidentally.   In one of these events they had poorly wired one of the breaker cabinets and caused a short which made it fail.  No problem because they had brilliantly set up the power so that failures from one cabinet would shunt the load to the next one.  Oops - there's still a short so now we've fried that next cabinet.  No problem because it fails over to another one etc...  - OH WAIT....!
>
> It is truly gratifying when there is a major production outage that you as a sysadmin could NOT have prevented and do NOT have to be involved in resolving (at least until it comes time to power everything back up).
>
> At that same company a co-worker put together a very good backup policy on our first implementation of NetBackup but when they saw how much it was going to cost for all the tapes required to do the various retention levels management balked.   They nixed his plan and said they'd never need a backup more than 6 months old and "saved money" by not buying as many tapes and adjusting the retention policies.    8 months later when they asked for a backup from the first month due to a critical issue they asked why we didn't have it.  When we told them (and even showed to them in writing) how they had said they would never need more than 6 months did they accept responsibility for poor decision?  They did not - They instead said that the admins were not forceful enough in trying to convince them of the need for the original plan.    Sometimes you just can't win.
>
Did management have pointy hair? It sounds like they let the bean counters determine all policy whether there was overriding technical/commercial issue involved. I wonder what the contractual obligations were to the end users? The contracts probably specified minimum backups required and that at a minimum should have driven the backup policy. From your comments it sounds like PHB's ignored the contractual obligations (lawsuit for breach of contract)

<snip>


--
Jay Lozier
jslozier at gmail.com

_______________________________________________
Ale mailing list
Ale at ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo




Athena(r), Created for the Cause(tm)
Making a Difference in the Fight Against Breast Cancer

---------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.
----------------------------------