[ale] System losing memory

[Jim]

timothy at meanor.net wrote:
> Try this:
> ps -eo pid,rss,vsize,args | sort -n -k 3,3
>   
Not really much difference from what top puts out.  It didn't find any 
processes that top didn't find. 
> This outputs PID, resident set size (all non-swapped memory process is using), vsize (virtual memory size of process), and command-line & args of each process, sorted by vsize.
>
> Do you have any oom_kill messages in any logs in /var/log/ ?  I can't remember if it actually says "oom_kill" in the message, or if it just says "killing process".
>   
Yes lots of them.  oom is trying to get rid of stuff, but doesn't seem 
to be getting any swap back.
> It's interesting that your system has swap completely utilized, yet almost all memory in use is pagecache.  I recall seeing this before on older Red Hat boxes, but I can't remember what causes this.
>   
I agree, something is preventing swap from getting freed up.  I killed 
most all of the user processes, mysqld and others.  No X, no apache and 
nothing really unusual.  I have a couple of other systems running pretty 
much the same compliment of software and none of them have ever had the 
problem.  The only difference is that this system is very busy.  One 
app  starts out at about 80 Mb and eventually rises to 110 Mb but 
doesn't go beyond that.  I've traced the memory to caches Boost uses 
primarily for smart pointers.  However killing that process doesn't 
affect the system hardly at all.
 
Output from suggested ps command for that process:

10936 5540 100352 /usr/local/bin/lserverd -f /etc/l
server.conf



Thanks,
Jim.