-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training stuck after "Successfully opened dynamic library libcublas.so.10" #79
Comments
Hello, I am getting the exact same problem with tensorflow 1.14. in a docker container based on tensorflow:1.14.0-gpu image and cuda 10.0 and a GPU with 10GB of RAM . The execution also stops at:
Is there any adidtional information we could provide to help identify this issue? Thanks in advance. |
This is probably not the issue for others, but I was also seeing the training hang with tensorflow 1.14 after |
Does anyone have resolved this issue after "Successfully opened dynamic library libcublas.so.10" ? 2022-02-01 15:06:35.916081: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 During handling of the above exception, another exception occurred: Traceback (most recent call last): Errors may have originated from an input operation. Input Source operations connected to node width_scores/xw_plus_b/MatMul: Original stack trace for 'width_scores/xw_plus_b/MatMul': |
Hi,
I am trying to train the BERT-base model on a new dataset. The training is stuck after printing "Successfully opened dynamic library libcublas.so.10". Following this discussion, I replaced the calls to parallel_interleave with interleave. However, the training is still stuck. I am using a 32GB GPU with tensorflow 1.14. I am not sure how to debug this one. Did anyone face this problem? Is there any workaround?
2021-03-31 00:07:04.751806: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2021-03-31 00:07:04.751837: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2021-03-31 00:07:04.751881: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2021-03-31 00:07:04.751905: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2021-03-31 00:07:04.751931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2021-03-31 00:07:04.751972: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2021-03-31 00:07:04.752003: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2021-03-31 00:07:04.755988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2021-03-31 00:07:04.756628: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2021-03-31 00:07:04.931587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-31 00:07:04.931625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2021-03-31 00:07:04.932043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2021-03-31 00:07:04.942135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30591 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB-LS, pci bus id: 0000:06:00.0, compute capability: 7.0) 2021-03-31 00:07:04.950214: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5611aa9bf160 executing computations on platform CUDA. Devices: 2021-03-31 00:07:04.950233: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla V100-SXM2-32GB-LS, Compute Capability 7.0 2021-03-31 00:07:30.820369: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
Thank you!
The text was updated successfully, but these errors were encountered: