You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At some point we should add options for PPL calculations, such as the ability to determine whether the windows overlap with each other, and .
Note that this heavily impacts the speed / compute required for evaluations in addition to scores.
Preferred implementation approach:
convert the generation_kwargs TaskConfig field to request_kwargs, leaving back-compatibility for providing generation_kwargs if request_kwargs is not in a given config
allow loglikelihood_rolling requests to take these request_kwargs, and implement new helper functions different from the current creation of context windows for loglikelihood rolling, and add unit tests for these helper fns
add some example task using these options
The text was updated successfully, but these errors were encountered:
At some point we should add options for PPL calculations, such as the ability to determine whether the windows overlap with each other, and .
Note that this heavily impacts the speed / compute required for evaluations in addition to scores.
Preferred implementation approach:
generation_kwargs
TaskConfig field torequest_kwargs
, leaving back-compatibility for providinggeneration_kwargs
ifrequest_kwargs
is not in a given configloglikelihood_rolling
requests to take these request_kwargs, and implement new helper functions different from the current creation of context windows for loglikelihood rolling, and add unit tests for these helper fnsThe text was updated successfully, but these errors were encountered: