Allow clearing gpu cache

[vladoovtcharov]

Allows releasing memory from pool from c and python api.
(Should resolve feature request #13482)

[szha]

Exposing this API as is would cause everything to stop for memory release which could be problematic for an asynchronous execution environment. Because of the impact on performance for not caching memory, a much more preferable option is to provide the option to completely turn off memory caching, so that people can use it and be sure that mxnet is not holding extra memory for special use cases.

[szha]

@vladoovtcharov thanks for the contribution. Instead of exposing ReleaseAll, would you mind adding an environment variable for MXNET_USE_GPU_MEM_POOL which defaults to 1, and use this value to decide whether Free should release the memory on the spot or return it to memory pool?

[vladoovtcharov]

Hi @szha, thanks for the feedback. I'm not sure if I completely understood though.
If the user wanted the memory to be freed right away couldn't they use a non-pooled storage manager (i.e. storage::NaiveStorageManagerstorage::GPUDeviceStorage()) or did you have something else in mind? (Although currently there is no value for MXNET_GPU_MEM_POOL_TYPE that would do this)

At least for my use case I was hoping to still use the memory pool/caching (for performance) but periodically pause and release the memory (so another process could use the gpu memory). I understand the release_all call could be expensive but it would not/should not be called regularly so overall it seems like it would effect performance less than using non pooled memory.

[szha]

@vladoovtcharov yes, they could use GPUDeviceStorage, and exposing the option in MXNET_GPU_MEM_POOL_TYPE is also fine. My general concern around exposing this type of knob at runtime is exactly to avoid supporting this type of use case, as it's hard to make any promise in this setting. For example, releasing objects in memory pool likely won't solve all your problem as there are other levels of caches. For now I think it's better to keep it as simple as on or off.

[vladoovtcharov]

I added an additional MXNET_GPU_MEM_POOL_TYPE (Unpooled) but I am worried that it is not the best thing to do.
As expected the performance is pretty bad when using unpooled memory (about 30% slower in my case for a simple classifier as in https://mxnet.incubator.apache.org/versions/master/tutorials/gluon/mnist.html)

It seems like using a pooled memory type with the ability to call ReleaseAll would be preferable in most use cases.

Also I'm not sure why the pooled memory managers would have any difference in what can be promised.
When using a storage manager that doesn't use pooling, when free gets called the memory is freed.
When using a storage manager that does use a pool, when free gets called the memory is recycled to the pool.
When ReleaseAll is called the recycled memory in the pool is freed.
So using the pooled manager with ReleaseAll would have the exact same guarantees as using an unpooled manager, but has the added benefit of improved performance.

[szha]

The problem is in the statement "the memory is freed". Not all memories are freed, and users shouldn't be concerned with how caching works in mxnet.

[szha]

I'm happy to take this into further consideration in 2.0 design (#9686). For now, let's not add the C API yet.

[vladoovtcharov]

@szha ok sounds good

[szha]

@vladoovtcharov thanks for understanding. Shall we get the Unpooled option merged first? If so, it would be great if you could look into the CI problems.

[vladoovtcharov]

Would it help to split into another pull request for just the unpooling? (One more formatting error to fix...)

[szha]

Yes, feel free to do so if it's easier :)

[anirudhacharya]

@mxnet-label-bot add [pr-work-in-progress]

[pinaraws]

@vladoovtcharov Did you get a chance to work on changes requested by @szha ?

[piyushghai]

@vladoovtcharov Gentle ping...

[Roshrini]

@vladoovtcharov Can you please resolve conflicts so that we can move forward with this PR?

[vladoovtcharov]

I went ahead and split into two merge requests, as requested. The second is here (#14716)

proper naming and doc addressed concerns.

[vladoovtcharov]

I'll update the c-api naming/documentation as well

[karan6181]

@vladoovtcharov Thanks for your contribution! Did you get a chance to update the docs ? Thanks!

[vladoovtcharov]

@karan6181 yes everything should be checked in and ready to be merged