Training stuck after "Successfully opened dynamic library libcublas.so.10"

[akshathaarodi]

Hi,

I am trying to train the BERT-base model on a new dataset. The training is stuck after printing "Successfully opened dynamic library libcublas.so.10". Following this discussion, I replaced the calls to parallel_interleave with interleave. However, the training is still stuck. I am using a 32GB GPU with tensorflow 1.14. I am not sure how to debug this one. Did anyone face this problem? Is there any workaround?

2021-03-31 00:07:04.751806: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2021-03-31 00:07:04.751837: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2021-03-31 00:07:04.751881: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2021-03-31 00:07:04.751905: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2021-03-31 00:07:04.751931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2021-03-31 00:07:04.751972: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2021-03-31 00:07:04.752003: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2021-03-31 00:07:04.755988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2021-03-31 00:07:04.756628: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2021-03-31 00:07:04.931587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-31 00:07:04.931625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2021-03-31 00:07:04.932043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2021-03-31 00:07:04.942135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30591 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB-LS, pci bus id: 0000:06:00.0, compute capability: 7.0) 2021-03-31 00:07:04.950214: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5611aa9bf160 executing computations on platform CUDA. Devices: 2021-03-31 00:07:04.950233: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla V100-SXM2-32GB-LS, Compute Capability 7.0 2021-03-31 00:07:30.820369: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

Thank you!

[josubg]

Hello, I am getting the exact same problem with tensorflow 1.14. in a docker container based on tensorflow:1.14.0-gpu image and cuda 10.0 and a GPU with 10GB of RAM . The execution also stops at:

Successfully opened dynamic library libcublas.so.10

Is there any adidtional information we could provide to help identify this issue?

Thanks in advance.

[nupoorgandhi]

This is probably not the issue for others, but I was also seeing the training hang with tensorflow 1.14 after 
Successfully opened dynamic library libcublas.so.10
For me,  the problem was that my train set was empty, so it might be helpful to double check that.

[aymen-souid-github]

Does anyone have resolved this issue after "Successfully opened dynamic library libcublas.so.10" ?

2022-02-01 15:06:35.916081: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-02-01 15:07:28.841615: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2022-02-01 15:07:28.848147: W tensorflow/core/kernels/queue_base.cc:277] _0_padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(30, 20), b.shape=(20, 3000), m=30, n=3000, k=20
[[{{node width_scores/xw_plus_b/MatMul}}]]
[[strided_slice_6/_1889]]
(1) Internal: Blas GEMM launch failed : a.shape=(30, 20), b.shape=(20, 3000), m=30, n=3000, k=20
[[{{node width_scores/xw_plus_b/MatMul}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 58, in
tf_loss, tf_global_step, _ = session.run([model.loss, model.global_step, model.train_op])
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(30, 20), b.shape=(20, 3000), m=30, n=3000, k=20
[[node width_scores/xw_plus_b/MatMul (defined at /home/souid/coref/util.py:109) ]]
[[strided_slice_6/_1889]]
(1) Internal: Blas GEMM launch failed : a.shape=(30, 20), b.shape=(20, 3000), m=30, n=3000, k=20
[[node width_scores/xw_plus_b/MatMul (defined at /home/souid/coref/util.py:109) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node width_scores/xw_plus_b/MatMul:
width_scores/hidden_weights_0/read (defined at /home/souid/coref/util.py:107)
span_width_prior_embeddings/read (defined at /home/souid/coref/independent.py:379)

Input Source operations connected to node width_scores/xw_plus_b/MatMul:
width_scores/hidden_weights_0/read (defined at /home/souid/coref/util.py:107)
span_width_prior_embeddings/read (defined at /home/souid/coref/independent.py:379)

Original stack trace for 'width_scores/xw_plus_b/MatMul':
File "train.py", line 26, in
model = util.get_model(config)
File "/home/souid/coref/util.py", line 21, in get_model
return independent.CorefModel(config)
File "/home/souid/coref/independent.py", line 54, in init
self.predictions, self.loss = self.get_predictions_and_loss(*self.input_tensors)
File "/home/souid/coref/independent.py", line 283, in get_predictions_and_loss
candidate_mention_scores = self.get_mention_scores(candidate_span_emb, candidate_starts, candidate_ends)
File "/home/souid/coref/independent.py", line 382, in get_mention_scores
width_scores = util.ffnn(span_width_emb, self.config["ffnn_depth"], self.config["ffnn_size"], 1, self.dropout) # [W, 1]
File "/home/souid/coref/util.py", line 109, in ffnn
current_outputs = tf.nn.relu(tf.nn.xw_plus_b(current_inputs, hidden_weights, hidden_bias))
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 4066, in xw_plus_b
mm = math_ops.matmul(x, weights)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 2647, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5925, in mat_mul
name=name)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/souid/anaconda3/envs/arabic_coref/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()