<html><head></head><body>Back to the original issue:<br>
<br>
A tool like torque or slurm is really your best solution to intensive shared resources. It prevents 2 big jobs from eating the same machine and can also encourage users to code better to manage resources better so they can run more jobs.<br>
<br>
I have the same problem. One heavy gpu machine (4 tesla P100) only has 64 G ram. Student tried to load in 200+G of data into ram. <br>
<br>
A few crashes later he can run 2 jobs at once, each only eats 30G ram and one p100. <br><br><div class="gmail_quote">On October 4, 2017 6:32:32 PM EDT, Todor Fassl <fassl.tod@gmail.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">I manage a group of research servers for grad students at a university. <br />The grad students use these machines to do the research for their Ph.D <br />theses. The problem is that they pretty regularly kill off each other's <br />programs by using up all the ram. Most of the machines have 256G of ram. <br />One kid uses 200Gb and another 100Gb and one or the other, often both, <br />die. Sometimes they bringthe machines down by hogging the cpu or using <br />up all the ram. Well, the machines never crash but they might as well be <br />down.<br /><br />We really, really don't want to force them to use a scheduling system <br />like slurm. They are just learnng and they might run the same piece of <br />code 20 times in an hour.<br /><br />Is there a way to set a limit on the amount of ram all of a user's <br />processes can use? If so, we were thinking of setting it at 50% of the <br />on-board ram. Then it would take 3 students together to trash a machine. <br />It might still happen but it would be a lot more infrequent.<br /><br />Any other suggestions? Anything at all? Just keep in mind that we really <br />want to keep it easy for the students to play around.<br /><br /></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.</body></html>