Memory is not freed to kernel

[ralight]

migrated from Bugzilla #486028
status CLOSED severity normal in component Mosquitto for 1.4
Reported in version unspecified on platform PC
Assigned to: Roger Light

On 2016-01-18 05:30:17 -0500, Pierre Fersing wrote:

We had an issue with Mosquitto that don't freed memory after using it. On a our server, this means that Mosquitto currently use 4 GB of memory... for an mostly empty broker.

We are in this situation because few days earlier (~ 3 days) our consumer stopped consuming message. Since we use persistent session Mosquitto kept those messages (in-memory + in persistence file). The file was about 4 GB which match the Mosquitto process consuming 4 Gb at that time. After our consumer processing the backlog,
the persistence file is now few Kb, but Mosquitto still use all 4 GB of RAM.

I'm not sure it's a memory-leak like a missing free(), the Mosquitto memory RSS on our server is not growing, it's stable at 4 GB.

While trying to reproduce this situation, I added some printf that showed the current RSS usage. Sometime the RSS was the same just before and just after a call to _mosquitto_free(store->payload).

My best guess would be that malloc/free don't always return memory to kernel (Google seems to confirm this). I didn't find more information on when free() really free the memory and when it does not.

It would be great if Mosquitto could freed that memory and not stay at memory peak forever.

Version used:

Mosquitto : 1.4.7
libc : Ubuntu Trusty libc6 2.19-0ubuntu6.6
Kernel : Ubuntu Trusty linux-image-virtual-lts-wily 4.2.0.23.17
uname : 4.2.0-23-generic

On 2016-01-18 16:30:25 -0500, Roger Light wrote:

Hi Pierre,

I've been very careful to ensure that there are no memory leaks in mosquitto. Some of the fixes in the 1.4.x releases were related to the case where memory was not leaked, but could have been freed because they were not needed any more. As far as I am aware now, all memory that is not needed is free()d as soon as possible.

What happens after this point is beyond my control. As you've suggested, free() does not release memory to the OS on most platforms. This means that on the next malloc(), memory is much quicker to acquire.

You could look into using an alternative malloc/free implementation that does return the memory to the OS, but first I'd suggest considering whether this is really a problem. If you have spikes of usage up to 4 GB, but most of the time are at a few KB, then the memory not in use should be swapped out to disk by your OS. Unless you have a shortage of swap space it should not influence other programs.

On 2016-02-11 16:32:41 -0500, Roger Light wrote:

I'm closing this as "not eclipse", because it's essentially nothing that mosquitto can do anything about - it's down to the malloc implementation.

If you disagree, please reopen the bug providing more details.