Train and Val directories created in main directory

[Adapter668]

Is there an existing issue for this?

• I have searched the existing issues

Bug summary

In the newest version, while training a new model, directories for train and val metrics are created in the main directory (InnerEye-DeepLearning) instead of in the directory of the model (InnerEye-DeepLearning/outputs/timestamp_model_name)

Code for reproduction

I didn't change the paths there

Actual outcome

Train and Val directories with matrics are in directory InnerEye-DeepLearning

Error messages

No errors

Expected outcome

Train and Val directories should be created in the directory of the model (InnerEye-DeepLearning/outputs/timestamp_model_name/Train) but it is not created

System info

Ubuntu 18.04.5 LTS
Conda environment created from environment.yml
d902e02

AB#8318

[Adapter668]

this occurs while using our own models, so it probably is problem in lightning container

[ant0nsc]

Hi @Adapter668 , thanks for reporting this. You are right, this changed recently. We previously had a hardcoded change working directory at the start of training, but that created a few downstream problems. We decided to remove that.
It is now the responsibility of the author of the LightningContainer to write to the correct folder. In the new world, your model should write all files to self.outputs_folder, instead of the current working directory.

@peterhessey , could you please double-check the documentation around that?

[Adapter668]

Hi, did you apply any solution here?

[ant0nsc]

@Adapter668 Did you try to ensure that your model writes all files into self.outputs_folder?