Error when chat template is not a string

[djstrong]

E.g. Mixtral has more than one template https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1/blob/main/tokenizer_config.json

Traceback (most recent call last):
  File "/venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/lm-evaluation-harness/lm_eval/__main__.py", line 369, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lm-evaluation-harness/lm_eval/utils.py", line 346, in _wrapper
    return fn(*args, **kwargs)
  File "/lm-evaluation-harness/lm_eval/evaluator.py", line 269, in simple_evaluate
    evaluation_tracker.general_config_tracker.log_experiment_args(
  File "/lm-evaluation-harness/lm_eval/loggers/evaluation_tracker.py", line 89, in log_experiment_args
    self.chat_template_sha = hash_string(chat_template) if chat_template else None
  File "/lm-evaluation-harness/lm_eval/utils.py", line 36, in hash_string
    return hashlib.sha256(string.encode("utf-8")).hexdigest()
AttributeError: 'dict' object has no attribute 'encode'

I guess just add str(). However, will it work later?

[haileyschoelkopf]

We should allow one to specify which chat template to use when there are multiple--potentially allowing for --apply_chat_template {template_name} as an arg format in addition to the current default?