[ale] Bash Scripts for System Automation and Monitoring

[Chris Fowler]

On 03/20/2012 11:06 AM, Jim Butler wrote:
> What "monitoring tools" or "automation tools" should a guy like me be
> learning to use, in order to be a good and marketable System
> Administrator? Where would you send a guy like me to learn the pieces he
> is missing? If you were in my shoes, what would you do to get up to
> speed most quickly? I'm looking for wisdom and guidance from Linux
> System Admins who are, well, better than me and more experienced than me.

The biggest issue I've ran into and I think James brought it us is that 
there are times where clickity click click can not do what you need it 
to do fast enough.  This is when some scripting experience pays off big 
time.

A few years back I had to upgrade about 100 devices over modems.  The 
upgrade took 1 hour per device.  I wrote a script to do all the work and 
keep tabs on what devices were down or up and what upgrades failed or 
succeeded.  It also kept tabs on the device's version level.  After we 
ran this script a few times we had it all done.

The key here is that I spent an hour upfront to save a few days of man 
hours!   Expect is a very cool automation tool / language that allows 
you to script many operations.

Here is one such example:

   eval {
     my $exp = Expect->spawn("ssh $login\@$ip");
     $exp->log_stdout(0);
     $exp->log_file($b);
     $exp->expect(60, "password") or die "connect failure";
     $exp->send("$pass\r");
     $exp->expect(60, "]# ") or die "password failure";
     $exp->send("/sbin/show net 1\r");
     $exp->expect(60, "]# ");
     $exp->send("cat /etc/resolv.conf\r");
     $exp->expect(60, "]# ");
     $exp->send("cat /etc/resolv.conf > /tmp/resolv.old\r");
     $exp->expect(60, "]# ");
     $exp->send("ping www.google.com\r");
     $exp->expect(40,
       [ qr/could not resolve/ => sub {  $result = 1;} ],
       [ qr/pinging/ => sub {  $result = 2;} ],
       [ qr/no reply/ => sub {  $result = 2;} ],
     );

     $result_before = $result;
     $result = 0;

     $exp->send("echo 'nameserver $test_address' > /etc/resolv.conf\r");
     $exp->expect(60, "]# ");

     $exp->send("cat /etc/resolv.conf\r");
     $exp->expect(60, "]# ");

     $exp->send("ping www.google.com\r");
     $exp->expect(40,
       [ qr/could not resolve/ => sub {  $result = 1;} ],
       [ qr/pinging/ => sub {  $result = 2;} ],
       [ qr/no reply/ => sub {  $result = 2;} ],
     );

     $result_after = $result;

     # put it back!
     $exp->send("cat /tmp/resolv.old > /etc/resolv.conf\r");
     $exp->expect(60, "]# ");

     $exp->send("exit\r");
   };


Now why?  There are 100 devices behind 100 firewalls with admins that 
wear different thickness of tin foil hat.  We needed to make sure these 
devices could actually use the DNS they were programmed for to connect 
to a different host when the sites were moved.   This is just a 
snippet.  There is another script which connects to the database and 
runs this one with the proper information.   It then tallies it all up 
and generates a report the technicians can act upon.  We've ran this 
many times while waiting on the remote sites to adjust their firewalls.

I spent about an hour hacking that and the mother script out to do the 
surveys the customer needed.

Why did I use cat?  There is no cp!!! It is an embedded device.

Chris