fix(dsp): prevent cache instantiation when cache is disabled

[stevenhoelscher]

The environment variable DSP_CACHEBOOL does not completely disable the filesystem cache.

As a result, joblib.Memory is still instantiated, which will try to create a directory on the filesystem.

In an environment where a user does not have write access to the DSP_CACHEDIR, this will throw a surprising exception despite DSP_CACHEBOOL being set to false.

File "/var/lang/lib/python3.11/site-packages/dsp/modules/cache_utils.py", line 27, in <module>
    CacheMemory = Memory(location=cachedir, verbose=0)
  File "/var/lang/lib/python3.11/site-packages/joblib/memory.py", line 1020, in __init__
    self.store_backend = _store_backend_factory(
  File "/var/lang/lib/python3.11/site-packages/joblib/memory.py", line 132, in _store_backend_factory
    obj.configure(location, verbose=verbose,
  File "/var/lang/lib/python3.11/site-packages/joblib/_store_backends.py", line 452, in configure
    mkdirp(self.location)
  File "/var/lang/lib/python3.11/site-packages/joblib/disk.py", line 61, in mkdirp
    os.makedirs(d)

[stevenhoelscher]

Looks like this could be updated to

if cache_turn_on and cachedir2:

[stevenhoelscher]

It's not obvious what DSP_NOTEBOOK_CACHEDIR denotes. Because it seems to assign the NotebookCacheMemory a fs cache but then walks that back if DSP_CACHEBOOL is false.

Also note that if DSP_CACHEBOOL is true, but DSP_NOTEBOOK_CACHEDIR is not set, then the cache won't work because of joblib.Memory defaults:

If None is given, no caching is done and the Memory object is completely transparent.

https://joblib.readthedocs.io/en/latest/generated/joblib.Memory.html

[arnavsinghvi11]

Thanks @stevenhoelscher for the PR! This looks good and maintains the cache correctly. Is the PR ready to merge or pending the change you've mentioned here?