output_path may break postprocessing

[artemorloff]

after recent updates (~few weeks ago) Evaluation Tracker has been introduced and also the have been a change in storing outputs. Now outputs_path is not the FINAL destination, where the outputs are stored. There is one mode (sub) directory is created. This causes some problems:

• postprocessing becomes harder. There is one more dir inside output_path. You do not now its name till end of running the code. Or you have to include the code for finding model_name_sanitized inside of your processing script. Once this logic for defining model_name_sanitized changes, all scripts will fail
• there could be more errors for users, when someone uses the same output_path for running two different models. This way two diff subfolders are created
• if I accidentaly run the same lm_eval (for the same model and tasks + output_path), the files are not overwritten. The names now contain the exact time. So, there would be 2 copies of the same file with no reason

What are the future plans for using model_source param of Evaluation Tracker?

[thehir0]

#1842

Same issue

[KonradSzafer]

Hi @artemorloff!

• I have addressed problems raised here and in How to use Zeno #1842 in the new PR Results filenames handling fix #1926, where I have moved functions for handling the new results files to utils, so that they can be conveniently used in other parts of the codebase. For example, the Zeno script or the EvaluationTracker class.
• In general, it's now better to just provide the results path without the model name, and for each model a separate directory will be created inside, making it easier not to accidentally mix model results.
• Yes, now results won't be overwritten, and this is done intentionally, so it's harder to lose results. Some users prefer this option, e.g. here

[StellaAthena]

if I accidentaly run the same lm_eval (for the same model and tasks + output_path), the files are not overwritten. The names now contain the exact time. So, there would be 2 copies of the same file with no reason

If you don't desire this behavior, you can use the --use_cache flag to avoid rerunning tasks that are already run and have results stored.