[ale] Bash Scripts for System Automation and Monitoring

[chip]

Instead of looking at it from a tools perspective, let's look at a use case.

For some reason you need to restart apache across 500 machines.  Would
you log in to each one and do it manually or would you write a script
for this?  Would it be python, perl, bash, or $other_language?  How
would you log in automatically?  If you need root/sudo perms to
restart apache, how would you deal with that (inputting your
password).  How would you make sure that the apache daemon restarted
properly and things are once again working.

Now you all of a sudden have 50 machines that just showed up in your
monitoring system has having run out of disk space.  They all happened
at the same time.  You log into one and see the problem and figure out
how to fix it.  You check a 2nd one and it's the same deal.  Now you
need to fix the other 48.  Script or manually?

Now you need to upgrade the FTP software running on 100 or so
machines.  The process is really straightforward except in a few cases
where a one-off config has been applied.  How can you easily tell
which ones have that one-off config and can you come up with some
scriptification to resolve those instances too?


To me this is what admin scripting boils down to.  It's not so much
being able to spot when things go wrong, but how do you fix them, and
fix them "at scale"?   If you have 10 boxes, no problem.  If you have
50, 100, 1000, it becomes a much larger issue.

--chip

On Tue, Mar 20, 2012 at 11:22 AM, James Sumners <james.sumners at gmail.com> wrote:
> Bash scripting, or the preferred shell of the location, is definitely
> a must. But that also means being familiar with the tool set available
> to you when you are actively interacting with Bash. Tools that I use
> frequently (in no particular order):
>
> * `nc` (netcat): a must if you're working with network services
> * `awk`
> * `sed`
> * `grep`
> * `cut`
> * `sort`
> * `screen`: e.g. https://bitbucket.org/jsumners/scripts/src/tip/screen_daemon.sh
> * `ps`
>
> That's a shorter list than I thought it would be. But it is off the
> top of my head. Still, you can do a lot with just those.
>
> I'd also recommend getting familiar with a scripting language that is
> a bit more robust than Bash. There are situations that I find are more
> easily handled with Python. I'm no Python guru (I code by manual with
> Python), but it certainly makes some things easier.
>
>
> On Tue, Mar 20, 2012 at 11:06, Jim Butler <jimbutler1234567890 at gmail.com> wrote:
>> Hi..
>> I have many years of experience with Linux, but I'm finding out lately
>> that apparently there is a good deal of mainstream knowledge that I've
>> missed somewhere. Most Linux System Admin positions require that you
>> know how to script so you can automate processes, and set up monitoring,
>> etc..
>> Usually when I wanted to monitor a system, I set up Nagios for host
>> monitoring, and then if I found a problem, I logged in manually to the
>> host in questionn and used tools like top, vmstat, iftop, iostat, etc.
>> for finding the cause of bottlenecks. But apparently when working on
>> large deployments like where you have large clusters or other types of
>> large setups, there need to be tools and/or scripts in place for
>> monitoring. I need to learn this stuff! I'm teaching myself BASH
>> scripting now, and so I have that under way. Also I know about using
>> MRTG and CACTI and the like, for monitoring network traffic on switches.
>> So let me ask everything this:
>> What "monitoring tools" or "automation tools" should a guy like me be
>> learning to use, in order to be a good and marketable System
>> Administrator? Where would you send a guy like me to learn the pieces he
>> is missing? If you were in my shoes, what would you do to get up to
>> speed most quickly? I'm looking for wisdom and guidance from Linux
>> System Admins who are, well, better than me and more experienced than me.
>>
>> Thank you in advance,
>> Jim Butler
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>
>
>
> --
> James Sumners
> http://james.roomfullofmirrors.com/
>
> "All governments suffer a recurring problem: Power attracts
> pathological personalities. It is not that power corrupts but that it
> is magnetic to the corruptible. Such people have a tendency to become
> drunk on violence, a condition to which they are quickly addicted."
>
> Missionaria Protectiva, Text QIV (decto)
> CH:D 59
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo



-- 
Just my $.02, your mileage may vary,  batteries not included, etc....