Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task assigned to only one group when multiple groups are run #1436

Open
baberabb opened this issue Feb 17, 2024 · 0 comments
Open

Task assigned to only one group when multiple groups are run #1436

baberabb opened this issue Feb 17, 2024 · 0 comments
Labels
bug Something isn't working.

Comments

@baberabb
Copy link
Contributor

baberabb commented Feb 17, 2024

Not sure if this is an issue or if this is the expected way, but if we have a task assigned to two groups in the config and run both groups, for e.g with gsm8k:

group:
  - math_word_problems
  - new_group

and --tasks math_word_problems,new_group

the task_dict is:

... ('gsm8k', ('math_word_problems', <lm_eval.api.task.ConfigurableTask object at 0x1760df5c0>)), ...

Haven't tested if it's different in benchmarks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working.
Projects
Status: Backlog
Development

No branches or pull requests

2 participants