pickle.save crashes when saving an NDArray on MXNet with error Check failed: !IsMKLDNNData() on MKL v1.3.1 taken from PyPi

[piyushghai]

Using pickle.save on an NDArray crashes with the exception :

Traceback (most recent call last):
  File "test.py", line 77, in <module>
    save_pickle_dump('op.pkl', output)
  File "test.py", line 58, in save_pickle_dump
    pickle.dump(data, f)
  File "/usr/lib/python2.7/pickle.py", line 1376, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 621, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 194, in __reduce__
    return NDArray, (None,), self.__getstate__()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/mxnet/ndarray/ndarray.py", line 369, in __getstate__
    ctypes.byref(cptr)))
  File "/home/ubuntu/.local/lib/python2.7/site-packages/mxnet/base.py", line 251, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [01:26:04] src/ndarray/ndarray.cc:706: Check failed: !IsMKLDNNData() We can't generate TBlob for MKLDNN data. Please use Reorder2Default() to generate a new NDArray first

Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1b6f0d) [0x7f3aae810f0d]
[bt] (1) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1b72d8) [0x7f3aae8112d8]
[bt] (2) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2e7f465) [0x7f3ab14d9465]
[bt] (3) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1c53f8) [0x7f3aae81f3f8]
[bt] (4) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2ea21d0) [0x7f3ab14fc1d0]
[bt] (5) /home/ubuntu/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(MXNDArraySaveRawBytes+0x8f) [0x7f3ab12af47f]
[bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f3ac86dae40]
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f3ac86da8ab]
[bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f3ac88ea3df]
[bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7f3ac88eed82]

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.

Environment info (Required)

----------Python Info----------
('Version      :', '2.7.12')
('Compiler     :', 'GCC 5.4.0 20160609')
('Build        :', ('default', 'Nov 12 2018 14:36:49'))
('Arch         :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version      :', '18.1')
('Directory    :', '/usr/local/lib/python2.7/dist-packages/pip')
----------MXNet Info-----------
('Version      :', '1.3.1')
('Directory    :', '/home/ubuntu/.local/lib/python2.7/site-packages/mxnet')
('Commit Hash   :', '19c501680183237d52a862e6ae1dc4ddc296305b')
----------System Info----------
('Platform     :', 'Linux-4.4.0-1074-aws-x86_64-with-Ubuntu-16.04-xenial')
('system       :', 'Linux')
('node         :', 'ip-172-31-11-28')
('release      :', '4.4.0-1074-aws')
('version      :', '#84-Ubuntu SMP Thu Dec 6 08:57:58 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping:              4
CPU MHz:               3000.000
BogoMIPS:              6000.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              25344K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 ida arat pku
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0011 sec, LOAD: 0.5849 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0015 sec, LOAD: 0.0867 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0091 sec, LOAD: 0.5859 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0123 sec, LOAD: 0.0283 sec.
Timing for Gluon Tutorial(en): https://gluon.mxnet.io, DNS: 0.0997 sec, LOAD: 0.1171 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.2975 sec, LOAD: 0.5281 sec.

Package used (Python/R/Scala/Julia):
Python 2.7.12

Build info (Required if built from source)

Taken from PyPi using this command :
pip install mxnet-mkl

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)

Build config:
(Paste the content of config.mk, or the build command.)

Error Message:

(Paste the complete error message, including stack trace.)
As indicated above.

Minimum reproducible example

import mxnet as mx
import urllib
import os.path
from os.path import join
from collections import namedtuple
import pickle
Batch = namedtuple('Batch', ['data'])
 
print (mx.__version__)
 
file_path = "/tmp/"
 
 
def download(url, location):
    filename = url.split("/")[-1]
    filename = join(location, filename)
    if not os.path.exists(filename):
        urllib.urlretrieve(url, filename)
 
 
def get_model(prefix, epoch, location):
    print ("Downloading Model --- " + prefix)
    download(prefix + '-symbol.json', location)
    download(prefix + '-%04d.params' % (epoch,), location)
 
 
def _get_base_model(symbol, arg_params, aux_params, ctx, layer_name='flatten0'):
    """
        symbol: the pretrained network symbol
        arg_params: the argument parameters of the pretrained model
        num_classes: the number of classes for the fine-tune datasets
        layer_name: the layer name before the last fully-connected layer
        """
    all_layers = symbol.get_internals()
    net = all_layers[layer_name + '_output']
    print (net)
    base_model = mx.mod.Module(symbol=net, label_names=None, context=ctx)
    base_model.bind(for_training=False, data_shapes=[('data', (1, 3, 224, 224))])
    base_model.set_params(arg_params=arg_params, aux_params=aux_params)
    return base_model
 
 
def get_image(url):
    # download and show the image. Remove query string from the file name.
    fname = mx.test_utils.download(url, fname=url.split('/')[-1].split('?')[0])
    img = mx.image.imread(fname)
    if img is None:
        return None
    # convert into format (batch, RGB, width, height)
    img = mx.image.imresize(img, 224, 224) # resize
    img = img.transpose((2, 0, 1)) # Channel first
    img = img.expand_dims(axis=0) # batchify
    img = img.astype('float32') # for gpu context
    return img
 
def save_pickle_dump(file_handle, data):
    with open(file_handle, 'wb') as f:
        pickle.dump(data, f)
 
get_model('https://data.mxnet.io/models/imagenet/resnet/50-layers/resnet-50', 0, file_path)
print ("Loading model checkpoint ---")
sym, arg_params, aux_params = mx.model.load_checkpoint(join(file_path,'resnet-50'), 0)
 
model = _get_base_model(sym, arg_params, aux_params, mx.cpu())
 
model.data_shapes
 
img = get_image('https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg?raw=true')
 
train_data = mx.io.DataBatch(data=[img])
train_data.label = None
 
model.forward(train_data)
 
output = model.get_outputs()
 
#print output
 
#print output[0].asnumpy()
 
#print output[0].shape
 
save_pickle_dump('op.pkl', output)

Steps to reproduce

(Paste the commands you ran that produced the error.)

pip install mxnet-mkl
python test.py

test.py is the name of the file whose contents are shared above.

What have you tried to solve it?

Downloaded non mkl package. i.e.
pip install mxnet

[piyushghai]

@mxnet-label-bot add[NDArray, Bug, MKL, MKLDNN]

[pengzhao-intel]

Thanks @piyushghai to report the issue.

@TaoLv could you help take a look for this issue?

[TaoLv]

@piyushghai This issue has already been fixed on the master branch. Can you try nightly build?

pip install mxnet-mkl --pre

[piyushghai]

@TaoLv Thanks. I checked out the --pre package and it seems to work fine over there.
If you have an idea, can you point me to the commit/PR which seems to have fixed this issue ?

[TaoLv]

@piyushghai Fixed by this check in #12530:
https://github.com/apache/incubator-mxnet/blob/master/src/ndarray/ndarray.cc#L1598

[piyushghai]

@piyushghai Fixed by this check in #12530:
https://github.com/apache/incubator-mxnet/blob/master/src/ndarray/ndarray.cc#L1598

Thanks!

[piyushghai]

Closing this issue as the fix is already in master branch.