KeyError: 'total_loss' in deep_boosted_regression application

[hasansari]

I have been trying to run the Deep Boosted Regression with some 3D datasets and get the following error after the first validation step. The training runs fine until then.

INFO:niftynet: validation iter 200, Loss=710.3148193359375, data_loss1=238.03733825683594, data_loss2=236.62986755371094, data_loss3=235.64764404296875 (79.673389s)
INFO:niftynet: cleaning up...
INFO:niftynet: iter 200 saved: //autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/models/pelvis_mrct/models/model.ckpt
INFO:niftynet: stopping sampling threads
ERROR:niftynet: Session failed to close after 30 seconds. Continuing after this point may leave your program in an undefined state.
Traceback (most recent call last):
File "net_run.py", line 8, in
sys.exit(main())
File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/init.py", line 147, in main
app_driver.run(app_driver.app)
File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 205, in run
loop_status=loop_status)
File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 331, in loop
ApplicationDriver.loop_step(application, iter_msg)
File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/application_driver.py", line 366, in loop_step
ITER_FINISHED.send(application, iter_msg=iteration_message)
File "/homes/8/hs584/.local/lib/python3.6/site-packages/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/homes/8/hs584/.local/lib/python3.6/site-packages/blinker/base.py", line 267, in
for receiver in self.receivers_for(sender)]
File "/autofs/space/atticus_001/users/hs584/Deeplearning/niftynet_env/NiftyNet/niftynet/engine/handler_performance.py", line 32, in update_performance_history
current_loss = console_content['total_loss']
KeyError: 'total_loss'

Any possible solutions for this? Thanks a lot

[danieltudosiu]

I am not the maintainer of that application, but looking into it it just misses the total_loss from the TF_SUMMARIES.

Please try and replace

            outputs_collector.add_to_collection(
                var=loss, name='LossSum',
                average_over_devices=True, summary_type='scalar',
                collection=TF_SUMMARIES)

with:

            outputs_collector.add_to_collection(
                var=loss, name='total_loss',
                average_over_devices=True, summary_type='scalar',
                collection=TF_SUMMARIES)