[ale] remote investigation

[James P. Kinney III]

Not fun, John.

If you can, plug in a dumb hub between the box and the outside world,
add a second box on that hub and turn on the net sniffers. You need to
determine if you have massive data exchange going on or is the problem
solely internal.

If the issue is massive data exchange, you have a security issue that
needs a hard drive backup, followed by a thorough system disinfection
running from known good binaries.

If the system is not getting massive IP traffic (or UDP for that
matter), then it is clearly internal. Linux doesn't just "break".
Something must be failing for that to happen. Restart your mail and web
servers one at a time. If the issue cleans up, then the one hat solved
the problem was spewing bad bits. Try running top to see if a process
has looplocked and is now eating up all your CPU time (think web script
in an infinite loop [been there, done that, ate all 16 CPU's on a Sun E
system once ;) ] ).

On Tue, 2003-09-02 at 08:37, John Wells wrote:
> Guys,
> 
> Came back from the Labor Day holiday and my mail server/web server is acting
> rather odd.
> 
> Services respond rather slowly, and sometimes not at all.  When services
> stop responding, I can still hit the router, so I know it has to be the
> server itself.
> 
> I'm currently logged in remotely and everything seems good, if not slow, but
> I expect it to freeze soon (it has a few times in the last hour or so).
> When it freezes, I can usually get a response after about 20 minutes or so.
> 
> The odd thing is, when services do "freeze" up, I can still telnet to a port
> on the machine, like 25 for smtp, and get a connection.  However, the SMTP
> server fails to respond and I just sit there.
> 
> I guess I'm kind of at a loss as to what sort of investigation I can do
> remotely.  I suppose the best way to see what's going on is to attempt to
> repeat the problem from home with a monitor connected and to see if it's
> actually doing anything during these timeouts, but I'd like to come home
> armed with any equipment that might be required.
> 
> Anyone had a similar experience in the past?  Does this sound like a
> possible bad NIC/harddrive/etc?  My first thought was that the box may have
> been compromised, but it'd be a wierd attack to let someone in every few
> minutes or so.  Netstat doesn't show anything unusual going on when I'm in,
> at least.
> 
> Any tests I could run against NIC/harddrive/etc to check for malfunctioning
> hardware?
> 
> Thanks for humoring the grasping at straws.  I'm frustrated, and clear
> thought is not currently an option... ;-)
> 
> John
> 
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
-- 
James P. Kinney III          \Changing the mobile computing world/
CEO & Director of Engineering \          one Linux user         /
Local Net Solutions,LLC        \           at a time.          /
770-493-8244                    \.___________________________./
http://www.localnetsolutions.com

GPG ID: 829C6CA7 James P. Kinney III (M.S. Physics) <jkinney at localnetsolutions.com>
Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C 6CA7 

 This is a digitally signed message part