Training loss is not decreasing

[SamratSinghRathore]

Hello, I am using Unet decoder and HRNet encoder model but i realize that the training loss is not decreasing, instead it seems to remain the same. Also the training logs appear in separate lines which is a problem becuase 1 epoch take around 200 lines and not gets output in one line.

I have trained it for 10 epochs but the loss keeps jumping between 0.99 and 1

[mberkay0]

Greetings @SamratSinghRathore,
The divergence/non-convergence of a model may depend on more than one situation. I recommend trying different parameters and different encoders. You may also consider freezing specific layers of the encoder. Another option is to freeze the entire encoder and train only the decoder.
In addition, please ensure to pre-process your data according to the backbone network model you have chosen.
Best regards.

Note: The separate line verbose is due to the 'tqdm' library. This problem may occur in the Jupyter Notebook. I will update the 'tqdm.auto' module.
Perhaps you can use this for a quick solution:

from tqdm import tqdm

tqdm._instances.clear()

[SamratSinghRathore]

Hello, Thank you for the reply. Will implement and get back to you. Regards Samrat Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Berkay Mayalı ***@***.***> Sent: Saturday, June 24, 2023 6:24:21 PM To: mberkay0/pretrained-backbones-unet ***@***.***> Cc: SamratSinghRathore ***@***.***>; Mention ***@***.***> Subject: Re: [mberkay0/pretrained-backbones-unet] Training loss is not decreasing (Issue #5) Greetings @SamratSinghRathore<https://github.com/SamratSinghRathore>, The divergence/non-convergence of a model may depend on more than one situation. I recommend trying different parameters and different encoders. You may also consider freezing specific layers of the encoder. Another option is to freeze the entire encoder and train only the decoder. In addition, please ensure to pre-process your data according to the backbone network model you have chosen. Best regards. Note: The separate line verbose is due to the 'tqdm' library. This problem may occur in the Jupyter Notebook. I will update the 'tqdm.auto' module. Perhaps you can use this for a quick solution: from tqdm import tqdm tqdm._instances.clear() — Reply to this email directly, view it on GitHub<#5 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXX3EZGHLAZRX5XZ4SK3TXLXM3PP3ANCNFSM6AAAAAAZJJOIMM>. You are receiving this because you were mentioned.Message ID: ***@***.***>