Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

train question #59

Open
Wangcongyu620 opened this issue Aug 19, 2023 · 1 comment
Open

train question #59

Wangcongyu620 opened this issue Aug 19, 2023 · 1 comment

Comments

@Wangcongyu620
Copy link

Wangcongyu620 commented Aug 19, 2023

When I run train.py in kitti6 dataset, I always encounter the following bugs, and I can't find the reason. Can you help me take a look?
ds_train: 3000
ds_val: 82
LOAD NET CHECKPOINT /home/wenbo/code_cy/READ-main/data/logs_offical/path/UNet_stage_0_epoch_net.pth

EPOCH 0

> TRAIN
EVAL MODE IN TRAIN
model parameters: 30193988
running on datasets [0]
model3d {'rgb': array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
...,
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]], dtype=float32), 'normals': None, 'uv2d': array([[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]], dtype=float32), 'faces': array([0, 1, 2], dtype=uint32), 'xyz': TrackedArray([[ -21.45231247, 13.24265003, -207.19848633],
[ -21.45231247, 13.24265003, -207.19848633],
[ -21.84580994, 14.13696384, -209.91720581],
...,
[ -30.93203926, 15.21014023, 684.93322754],
[ -30.93203926, 15.21014023, 684.93322754],
[ -30.93203926, 15.21014023, 684.93322754]]), 'xyz_c': TrackedArray([[0.41255814, 0.4299459 , 0.02919084],
[0.41255814, 0.4299459 , 0.02919084],
[0.41 , 0.44497895, 0.02623283],
...,
[0.3509302 , 0.46301866, 0.9998443 ],
[0.3509302 , 0.46301866, 0.9998443 ],
[0.3509302 , 0.46301866, 0.9998443 ]], dtype=float32), 'uv1d': array([ 0, 1, 2, ..., 2547319, 2547320, 2547321])}
proj_matrix was not set
total parameters: 50572564[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
[w] b'GLX: No GLXFBConfigs returned'
[w] b'GLX: Failed to find a suitable GLXFBConfig'
[x] Window creation failed
Traceback (most recent call last):
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/multiprocessing/queues.py", line 114, in get
raise Empty
_queue.Empty

The above exception was the direct cause of the following exception:Traceback (most recent call last):
File "/home/wenbo/code_cy/READ-main/train.py", line 546, in <module>
train_loss = run_train(epoch, pipeline, args, iter_cb)
File "/home/wenbo/code_cy/READ-main/train.py", line 281, in run_train
return run_epoch(pipeline, 'train', epoch, args, iter_cb=iter_cb)
File "/home/wenbo/code_cy/READ-main/train.py", line 256, in run_epoch
run_sub(dl, extra_optimizer)
File "/home/wenbo/code_cy/READ-main/train.py", line 146, in run_sub
for it, data in enumerate(dl):
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
idx, data = self._get_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
success, data = self._try_get_data()
File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 8638, 8658, 8678) exited unexpectedly
deleting buffers...

Process finished with exit code 1 @JOP-Lee Looking forward to your reply!

@VerseWei
Copy link

Hi,I encounter the same dilemma now, could you share your method for this problem with me? Many thanks for you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants