test_sparse_mathematical_core segmentation fault

[szha]

Description

unittest/test_sparse_operator.py::test_sparse_mathematical_core segmentation fault on centos-cpu

Occurrences

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-18656/18/pipeline/#step-259-log-1771

[leezu]

There can also be Bus error in create_sparse_array after test_sample_multinomial_num_outputs

"Bus errors are generally caused by applications trying to access memory that hardware cannot physically address" https://stackoverflow.com/questions/36970110/how-to-determine-the-cause-for-bus-error

[2020-11-07T05:38:26.553Z] tests/python/gpu/test_operator_gpu.py::test_normal_generator PASSED      [ 82%]

[2020-11-07T05:38:26.809Z] tests/python/gpu/test_operator_gpu.py::test_uniform_generator PASSED     [ 83%]

[2020-11-07T05:38:30.066Z] tests/python/gpu/test_operator_gpu.py::test_gamma_generator PASSED       [ 83%]

[2020-11-07T05:38:33.322Z] tests/python/gpu/test_operator_gpu.py::test_exponential_generator PASSED [ 84%]

[2020-11-07T05:38:35.831Z] tests/python/gpu/test_operator_gpu.py::test_poisson_generator PASSED     [ 84%]

[2020-11-07T05:38:39.094Z] tests/python/gpu/test_operator_gpu.py::test_negative_binomial_generator PASSED [ 85%]

[2020-11-07T05:39:11.490Z] tests/python/gpu/test_operator_gpu.py::test_multinomial_generator PASSED [ 85%]

[2020-11-07T05:39:11.490Z] tests/python/gpu/test_operator_gpu.py::test_with_random_seed PASSED      [ 86%]

[2020-11-07T05:39:11.490Z] tests/python/gpu/test_operator_gpu.py::test_random_seed PASSED           [ 86%]

[2020-11-07T05:39:11.490Z] tests/python/gpu/test_operator_gpu.py::test_unique_zipfian_generator PASSED [ 87%]

[2020-11-07T05:39:11.490Z] tests/python/gpu/test_operator_gpu.py::test_zipfian_generator PASSED     [ 87%]

[2020-11-07T05:40:47.841Z] tests/python/gpu/test_operator_gpu.py::test_shuffle PASSED               [ 88%]

[2020-11-07T05:40:47.841Z] tests/python/gpu/test_operator_gpu.py::test_randint PASSED               [ 88%]

[2020-11-07T05:40:47.841Z] tests/python/gpu/test_operator_gpu.py::test_randint_extremes PASSED      [ 89%]

[2020-11-07T05:41:34.440Z] tests/python/gpu/test_operator_gpu.py::test_randint_generator PASSED     [ 89%]

[2020-11-07T05:41:34.440Z] tests/python/gpu/test_operator_gpu.py::test_randint_without_dtype PASSED [ 90%]

[2020-11-07T05:41:34.440Z] tests/python/gpu/test_operator_gpu.py::test_sample_multinomial_num_outputs PASSED [ 90%]

[2020-11-07T05:42:00.951Z] Fatal Python error: Bus error

[2020-11-07T05:42:00.951Z] 

[2020-11-07T05:42:00.951Z] Current thread 0x00007fbb4be7f740 (most recent call first):

[2020-11-07T05:42:00.951Z]   File "/work/mxnet/python/mxnet/test_utils.py", line 505 in create_sparse_array

[2020-11-07T05:42:00.951Z]   File "/work/mxnet/python/mxnet/test_utils.py", line 543 in create_sparse_array_zd

[2020-11-07T05:42:00.951Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 596 in check_sparse_mathematical_core

[2020-11-07T05:42:00.951Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 726 in check_binary_op_with_scalar

[2020-11-07T05:42:00.951Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1133 in test_sparse_mathematical_core

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/python.py", line 1445 in runtest

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 210 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 237 in from_call

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 210 in call_runtest_hook

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 129 in call_and_report

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 99 in runtestprotocol

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 92 in pytest_runtest_protocol

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 271 in pytest_runtestloop

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 247 in _main

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 197 in wrap_session

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/config/__init__.py", line 93 in main

[2020-11-07T05:42:00.951Z]   File "/opt/rh/rh-python36/root/usr/bin/pytest", line 8 in <module>

[2020-11-07T06:01:37.785Z] Timeout (0:20:00)!

[2020-11-07T06:01:37.785Z] Thread 0x00007fbb4be7f740 (most recent call first):

[2020-11-07T06:01:37.785Z]   File "/work/mxnet/python/mxnet/test_utils.py", line 505 in create_sparse_array

[2020-11-07T06:01:37.785Z]   File "/work/mxnet/python/mxnet/test_utils.py", line 543 in create_sparse_array_zd

[2020-11-07T06:01:37.785Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 596 in check_sparse_mathematical_core

[2020-11-07T06:01:37.785Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 726 in check_binary_op_with_scalar

[2020-11-07T06:01:37.785Z]   File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1133 in test_sparse_mathematical_core

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/python.py", line 1445 in runtest

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 210 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 237 in from_call

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 210 in call_runtest_hook

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 129 in call_and_report

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 99 in runtestprotocol

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 92 in pytest_runtest_protocol

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 271 in pytest_runtestloop

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 247 in _main

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 197 in wrap_session

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/_pytest/config/__init__.py", line 93 in main

[2020-11-07T06:01:37.785Z]   File "/opt/rh/rh-python36/root/usr/bin/pytest", line 8 in <module>

[2020-11-07T08:26:36.330Z] Sending interrupt signal to process

https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/leezu-patch-3/4/pipeline