<html><head></head><body><div>The root dir is NOT NFS mounted so that's a red-herring that you can't mount the /home later. If /var is not writeable, the system will hang as it can't log any more. Mounting requires a log entry</div><div><br></div><div>Since it's not happening all at once to all the machines it really smells like a local machine problem. Verify that the drive is not full. Check to see if the affected machines are on the power circuit. </div><div><br></div><div>Is it the same 2-3 each time? If so, run memtest and badblocks. If swap gets corrupted, Linux system lock up.</div><div><br></div><div>On Mon, 2016-03-28 at 10:54 -0500, Todor Fassl wrote:</div><blockquote type="cite"><pre>I have a mysterious problem with workstations in a shared use
environment. There are 2 labs in different buildings, onewith 6
workstations and one with 8. These workstations are used by a group of
about 30 grad student TAs. All are running ubuntu 15.10. Authentication
is via ldap and home directories are mounted via nfs. Every day, 2 or
3 of the machines go down. The earliest symptom I can find is that the
root filesystem is remounted read-only. Soon they stop responding to
ssh and snmp and they are essentially locked up. They still respond to
pings though.
I've caught the machines in the period where the root system is
read-only but I can still ssh to them. I've found that I cannot nfs
mount home directories on our file server. I can mount nfs shares on
other servers. And I can mount the same home directories if I go to
another workstation. Restarting nfs on the file server has no effect.
When I try to mount a home directory on an effected machine, the mount
just hangs. I ran it with strace and it just showed it was waiting --
for what, I'm not sure and I don't have a screen cap available at the
moment. I put a packet sniffer on the server and it showed it received a
single packet from the client and that's it.
There is nothing in the logs on the client. In fact, they simply stop at
some point in the process. At first I attributed this to the root
filesystem being read-only but it continues after I move /var to a
separate file system. At some point it just stops writing records to the
syslog but I don't know if it's before or after the root filesystem is
remounted read-only.
Many of the TAs also have identical workstations in their offices. None
of those machines seem to have this problem. The TAs do tend to walk
away from the workstations w/o logging out. But I wrote a script to kill
off their sessions and it didn't help. I had it send me an email
whenever it killed somebody's session and it doesn't seem to be
correlated with that. In other words, sometimes machines go down even if
everyone who has used it has remembered to log out.
I'm pretty desperate. Any ideas?
_______________________________________________
Ale mailing list
<a href="mailto:Ale@ale.org">Ale@ale.org</a>
<a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a>
See JOBS, ANNOUNCE and SCHOOLS lists at
<a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a>
</pre></blockquote><div><span><pre>--
James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
</pre></span></div></body></html>