--hf_hub_log_args causes IndexError

[johnwee1]

Brief description:

Trying to upload to HF causes the script to fail at the end (after running evaluations). The results are still saved in file, I just don't see the formatted results printed on the terminal. Removing the --hf_hub_log_args flag however fixes this issue. could it be due to recent changes? i just updated yesterday.

logfile:

Checking cached requests: 100%|███████████████████████████████████| 317/317 [00:00<00:00, 755.55it/s]



Running generate_until requests: 0it [00:00, ?it/s]

Running generate_until requests: 0it [03:34, ?it/s]

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/home/jovyan/lm-evaluation-harness/lm_eval/__main__.py", line 448, in <module>

    cli_evaluate()

  File "/home/jovyan/lm-evaluation-harness/lm_eval/__main__.py", line 432, in cli_evaluate

    evaluation_tracker.recreate_metadata_card()

  File "/home/jovyan/lm-evaluation-harness/lm_eval/loggers/evaluation_tracker.py", line 472, in recreate_metadata_card

    last_results_file = [

                        ^

IndexError: list index out of range

[KonradSzafer]

@haileyschoelkopf I think this can be closed now. From what I’ve checked, it’s related to the change of the extension from .json to .jsonl in the result files, which prevented the function from finding them - already fixed here:

lm-evaluation-harness/lm_eval/utils.py

Lines 162 to 166 in 2b26690

	def get_file_datetime(filename: str) -> str:
	"""
	Given the results and sample results filenames, extracts and returns the datetime.
	"""
	return filename[filename.rfind("_") + 1 :].replace(".jsonl", "")

[haileyschoelkopf]

Thanks for tracking this @KonradSzafer !