[ale] RHEL 5 will not allow login from Console

[John Temple]

Bob and Mike,

Today was a pretty rough day. (Putting out other fires, since this is a webserver that is only used a by a few people it wasn't very high on the priority list. One person even said they wouldn't cry if it never came back.) 

To answer a few questions we are able to some what get into the system with a boot CD and linux rescue. From there I had only a moment to poke around but I was concentrating on trying to locate a file for someone else.
Yes the system is running in VMWare and SSH is dead we can only get in from the console of VMWare.

Tuesday I might have a chance to look around some more. Also my coworker was going to see if he could pull over a year old backup of the vdi.

Well touch back later when I have more details.

Thanks

On Sep 2, 2011, at 1:16 AM, Bob Toxen wrote:

> On Thu, Sep 01, 2011 at 04:06:04PM -0400, Michael H. Warfield wrote:
>> On Thu, 2011-09-01 at 15:34 -0400, John Temple wrote: 
>>> I have a RHEL 5 VM system that will not allow us to login from the console.
>>> We have tried to use both a valid user and root, for both of them after
>>> entering the username "Invalid Username" (or something like that) flashes
>>> and then we are returned to the login prompt. We have also tried booting
>>> into single user mode by editing the grub command line. No dice there
>>> either. Any suggestions on how to get the system back up?
> 
>> You say it did NOT prompt you for a password and failed immediately?
>> That sounds like a corrupted binary or something serious pretty deep in
>> the system.  Are you able to get in from other locations or are you just
>> flat out locked out?
> Check /bin/login for corruption or bad permissins (755 owned by root is
> normal) and /sbin/mingetty.  Also, check /etc/securetty.
> 
>>> A couple of things that we have noticed:
>>> 1. When the VM boots the system displays a couple of failures most noteably
>>> iptables and xinetd.
> 
>> Ewww...
> 
>>> 2. A few weeks ago a co-worker said that he had trouble with the system
>>> saying that it was in read only mode.
> 
>> That is generally indicative of file system corruption.
> Yup, it sounds like your system is seriously screwed up, clearly with
> some file system damage that could explain the lack of being able to
> log in.
> You could
> compare to backup with "tar -d" to diff against backup or reinstall.
> I assume you only can log in via ssh, which doesn't use /sbin/mingetty or
> /bin/login.
> 
>> You say it's a VM?  I take it, it must be one of the paravirtualized
>> VM's?  VMware, VirtualBox, XEN, or KVM?
> 
>> What I would suggest is laying hands on a good run-live forensic CD,
>> like the Network Secuirty Toolkit, NST, here:
> 
>> http://www.networkseckuritytoolkit.org
> 
>> They just came out with one based on Fedora 15.  The previous one was
>> based on Fedora 13 and is what I've been using the most.
> 
>> Boot your VM from the CD Image.  I think both VMware and VirtualBox
>> default to the hard drive, rather than the CD and you'll have to
>> interrup the BIOS and select the boot device.
> 
>> Get it up and running and then try running an fsck on the partitions
>> that it sees on the hard drive.  NST does start up LVM and you can fsck
>> LVM partitions too.
> 
>> If you have no errors, mount the partitions over a mount point in the
>> correct relative hierarchy (tedious, I know).  You can then chroot into
>> that mount point and you'll see your machine as if you had logged into
>> it (just that nothing is running) and you can poke around and check logs
>> and even manually start up run-time services and see how they behave.
>> You can run an rpm -V and do some verifying in there as well and see if
>> it finds anything to piss'n'moan about.
> 
>>> -- 
>>> John Temple
>>> cjtemple at gmail.com
> 
>> Regards,
>> Mike
>> -- 
>> Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
>>   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
>>   NIC whois: MHW9          | An optimist believes we live in the best of all
>> PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
> 
> Bob Toxen
> bob at verysecurelinux.com               [Please use for email to me]
> http://www.verysecurelinux.com        [Network&Linux security consulting]
> http://www.realworldlinuxsecurity.com [My book:"Real World Linux Security 2/e"]
> Quality Linux & UNIX security and SysAdmin & software consulting since 1990.
> Quality spam and virus filters.
> 
> "One disk to rule them all, One disk to find them. One disk to bring
> them all and in the darkness grind them. In the Land of Redmond where
> the shadows lie...and the Eye is everwatching"
> -- The Silicon Valley Tarot Henrique Holschuh with ... by Bob
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo