model_comparator.py broken

[johnwee1]

Referencing this issue https://github.com/meta-llama/llama-recipes/pull/488/files where it seems that the evaluator class is no longer used.

I could not get the model_comparator.py script to run as it throws the same attribute error AttributeError: module 'lm_eval.tasks' has no attribute 'initialize_tasks' so my guess is that the script is broken.

I actually don't understand the point of the script, isn't it more straightforward to eval using predict-only at temperature 0 then directly compare the results? since at any non zero temperature the results would not tally across different backends.

thanks for taking the time to read.