Format of Personal Defined Dataset for Evaluation

[OscarC9912]

Hi developers,

I am wondering, if there's an instruction on how to create a self-defined eval dataset, like what the format should be, which could be executed by the lm_eval_harness ?

Thanks for your help ~

[haileyschoelkopf]

Hi! in our documentation we provide a description of how to add and define a new task.

For your own dataset, the only precondition is that you turn this into a Hugging Face dataset loadable with the datasets library—then in your task definition you can use all the column values.