PyPlot memory deallocation problem

[flpf]

Hi,

when batch saving multiple figures with 'savefig()' a large amount of memory is
allocated and I am not able to free it afterwards without closing my julia session.

Basically I am doing:

for ff=101:150
    figure(1)
    PyPlot.clf()
    imshow(imagg,extent=[0,350,0,580],aspect=1);
    imshow(gridG[1:18,1:35,ff],interpolation="bicubic",cmap="jet",origin="lower",extent=[14,339,158,300],alpha=.5,aspect=.8)
    PyPlot.autoscale("off")
    plt.axis("off")
    plt.title("Absolute value @" *string(ff)* " Hz")
    savefig("~/pianoMeas1/klavierKlopfTest_"*string(ff)*".png");
    plt.close()
end

The memory usage before the loop is ~2gb, after the loop it is ~4gb.

I also tried

figurN=open("~/pianoMeas1/klavierKlopfTest_"*string(ff)*".png","w")
plt.savefig(figurN);
close(figurN)

giving the same result.

When overwriting existing files the memory usage stays constant so I believe the
"memory-overflow" occurs while allocating space for the file, but this is just a shot in the dark.

My julia version is:

versioninfo()
Julia Version 0.4.0-dev+2496
Commit 89f89e1* (2015-01-05 09:04 UTC)
DEBUG build
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Any ideas, pointers are appreciated as I am not shure where to start debugging this.

[stevengj]

What happens if you do the corresponding thing in Python?

[flpf]

I did not try it in Python because I am not familiar with it and just use Python packages
from julia (matplotlib and seaborn are the only two librarys I use).
I'll try to code the critical loop in Python and let you know how Python handles this.

[flpf]

I ran the script in Python and it seems to behave just fine.
The memory usage stays constant around ~ 1.6gb.

Because I'm not fluent in Python I am not a hundred percent shure that I'm doing the same thing but I guess the following code captures the problem.

Here is the code:

import numpy as np
import matplotlib.pyplot as plt

my_data=np.genfromtxt('~/testg.txt',delimiter=',');
my_plot_data=np.reshape(my_data,(18,35,10000));
for ff in np.linspace(1,400,400):
    plt.imshow(my_plot_data[1:18,1:35,int(ff)],interpolation="bicubic",cmap="jet",origin="lower",extent=[14,339,158,300],alpha=.5,aspect=.8)
    plt.savefig("test"+str(int(ff))+".png")

It runs much slower compared to the julia version but has no memory deallocation problems
as far as I can see.

[stevengj]

Is it just that there is some memory that hasn't been garbage-collected yet? You could try gc() in Julia after the loop.

Also, can you try deleting code from your Julia example until the problem goes away? That is, what is the minimal example that has a problem for you?

[flpf]

Yes, I called the garbage collecter outside and inside the loop with no effect. I also tried
setting all workspace variables to zero, which frees some memory but not the memory
allocated during the loop. I've noticed the problem in earlier versions of julia as well but
never looked into this because of one other thing that I just noticed, the large memory allocation only kicks in if the loop exceeds a certain count (between 50-100 iterations).

This is the minimal example in julia:

for ff=51:150
    imshow(gridG[1:18,1:35,ff],interpolation="bicubic",cmap="jet",origin="lower",extent=[14,339,158,300],alpha=.5,aspect=.8)
    imshow(imagg,extent=[0,350,0,580],aspect=1);
    figurN=open("~/pianoMeas1/klavierKlopfTest_"*string(ff)*".png","w")
    savefig(figurN);
    close(figurN)
end

One 'imshow()' + 'savefig()' runs smoothly, if I add the second 'imshow()' I get that
memory problem.

---

Edit: I added a second 'imshow()' to the Python version as well, it had no impact on the
memory allocation of Python.

[stevengj]

Wouldn't the equivalent of the Python version do savefig("~/pianoMeas1/klavierKlopfTest_"*string(ff)*".png")?

[flpf]

The Python test file ran in the home directory as did the julia version.
The only difference in that line is the usage of the Python string concatenation operator,
which is (+) instead of () in julia. The Python code prints similar ".png" files so I think the code snippets are comparable.

[stevengj]

But your Julia snippet passed a file handle to savefig, whereas the Python snippet just passed the filename. Why not pass the filename in both cases?

[flpf]

Sorry I haven't been clear on this. I've tried both variants in both languages. Both variants behave
the same in either language: julia->memory problem; Python ->no problem.

A second minimal example using the julia image from github:

using PyPlot
imagg=imread(download("https://camo.githubusercontent.com/e1ae5c7f6fe275a50134d5889a68f0acdd09ada8/687474703a2f2f6a756c69616c616e672e6f72672f696d616765732f6c6f676f5f68697265732e706e67"),format="png"); # Loading the julia image
for ff=1:300       
    imshow(imagg);       
    imshow(imagg);       
  savefig("~/tempFolder/imageWriteTest_"*string(ff)*".png");
end

The script is called from the home directory and writes the "*.png" files to a folder residing in the
same directory.
It eats up memory which is not freed afterwards. Maybe someone else could run it and see if it
causes similar problems or if this is a quirk of my setup.

[flpf]

I have an even more minimal minimal example which leads to the same memory
issues as described above:

using PyPlot
for ff=1:300
    plot(cos(2*pi*200*linspace(0,pi,40000)));
    plot(sin(2*pi*200*linspace(0,pi,40000)));
    savefig("/home/fl/tempFolder/imageWriteTest_"*string(ff)*".png");
end

[axsk]

I experience the same problem with Julia 0.4-dev on Windows 8.
In the given example the memory increase depends on the resolution of linspace and is around 4mb per iteration for the given 40.000.

The point here really seems to be the multiple plots, as the problem does not occur with single plot saves.
It persists when using subplot for different plots.

[kzapfe]

I have the same problem on IJulia with julia 3.7 and matplotlib 1.4.2, even after calling gc() each time I call a new figure. My code goes like this:

for t=1:tmax
    gc()
    PyPlot.ioff()
    figure()
    b=round(t/7022, 4)
    imagen=imshow(DatosSuaves[:,:,t], cmap="seismic", origin="upper", vmin=-200, vmax=200)
    cb=colorbar(imagen, ticks=[-200, 0, +200],fraction=0.046, pad=0.04 )
    cb[:set_label]("recorded voltage [µV]  ")
    cuac=savefig("GarbageCollector/LCF01-$t.png", dpi=72)
end

where DatosSuaves is a 64 by 64 by 5000 array. The memory is NEVER freed afterwards.

[aarchiba]

So, many of these examples contain other bugs that lead to growing memory usage. If you don't clear or close the figure and keep drawing things, they all remain in the plot and it grows. Also if you allocate a new figure each time (with figure()) but don't close any of them, that's a memory leak. So all the minimal examples appear to be examples of something else. The original is another story; I'm not sure what the problem there is.

[nsmith5]

I seem to be affected by this bug as well.

The following function is a minimal example. It allocates more and more memory everytime it's called and garbage collection does not remove it.

function make_figure(f::Matrix)
    fig = figure()
    ax = fig[:add_subplot](111)
    imag = ax[:imshow](f)
    fig[:savefig]("output.png")
    plt[:close](fig)
    return nothing
end

Example:

julia> using PyPlot
julia> f = randn(1024, 1024);
julia> make_figure(f)                # Allocates ~14 Mb
julia> make_figure(f)                # Allocates ~14 Mb again....
julia> gc()                          # No change in memory allocated
.
.
.

Tested on these two versions:

Julia Version 0.4.5
Commit 2ac304d* (2016-03-18 00:58 UTC)
Platform Info:
  System: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblasp.so.0
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Julia Version 0.5.0-dev+4846
Commit 59b2530* (2016-06-17 21:46 UTC)
Platform Info:
  System: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

Additionally, if you remove the savefig() line, the function still has a memory leak, it just leaks less each call (couple hundred Kb). Seems to be that there is some issue deallocating memory in python objects even if they are local variable in a function...

Edit: Not a Julia Issue

Upon further investigation the following python code has the same issue

def make_figure(arr):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    imag = ax.imshow(arr)
    plt.savefig("output2.png")
    plt.close(fig)
    return None

so my code is just not doing what I think it should be doing...

[stevengj]

@nsmith5, thanks for looking into it. If this is a Matplotlib issue, maybe you can file an issue there?

[nsmith5]

This seems to be a script versus repl thing I think. The following two scripts (python and julia) have no memory allocation problems.

• figure.py
• figure.jl

Run these scripts from the shell with a chosen number of plotting iterations doesn't result in any kind of memory leak.

$ export ITERATIONS=100
$ julia figure.jl $ITERATIONS
$ python figure.jl $ITERATIONS

@stevengj Yes, i'll look into it with Matplotlib.

[stevengj]

i.e. interactive vs. non-interactive mode in Matplotlib. (Running from the shell uses non-interactive mode.)

[abieler]

I do also experience this memory leak. I work in Jupyter notebook and have no automatic loop,
but instead I manually "loop", e.g. create plots, look at them, close figure, refine plot command, plot, look at it, close, refine etc...
After a while the notebook uses up all of my memory and gc() does not free any memory.

[stevengj]

@abieler, if the leak is in Matplotlib itself, there is nothing I can do. Someone needs to take one of the Python examples that exhibits the problem and report it upstream to Matplotlib.

[lucasb-eyer]

Hi all, this sounds an awful lot like ipython/ipython#7270. TL;DR: Try calling matplotlib.interactive(False) before the plotting, it might help as a workaround if this is the same issue.

[CrashLaker]

@lucasb-eyer nice! this "solved" my problem :)