Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added leaderboard for arena hard auto #3437

Merged
merged 6 commits into from
Jul 31, 2024

Conversation

connorchenn
Copy link
Contributor

Why are these changes needed?

added feature to display arena hard leaderboard on chatbot arena page

Added code for a new tab to be displayed in the chatbot arena page. The code takes a csv file of the arena hard rankings, processes it, and displays it

Related issue number (if applicable)

Checks

  • I've run format.sh to lint the changes in this PR.
  • I've included any doc changes needed.
  • I've made sure the relevant tests are passing (if applicable).

@CodingWithTim
Copy link
Collaborator

let me take a look @connorchenn

@CodingWithTim
Copy link
Collaborator

@connorchenn Could you edit the mapping file instead of manually mapping them in the code? For cleaner practices.

info["glm-4-air"] = {
        "display": "GLM-4-Air",
        "link": "https://open.bigmodel.cn/",
        "org": "Zhipu AI",
    }
info["snorkel-mistral-pairrm-dpo"] = {
        "display": "Snorkel-Mistral-PairRM-DPO",
        "link": "https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO",
        "org": "Snorkel AI",
    }

@connorchenn
Copy link
Contributor Author

ok, I made the changes, I will add the new mapping file to the chatbot arena hf leaderboard

@CodingWithTim
Copy link
Collaborator

@connorchenn Thank you so much! A few things:

  1. Add arena-hard ranking to full leaderboard table
  2. Add knowledge cuff off to the the arena hard leaderboard
  3. Add last updated to the csv file and support automatic update in the title
  4. Update the element id for the leaderboard.
  5. Add link to paper in title.
  6. Update Llama-3 score on Arena Hard Repo.

And we should be good for merging.

@CodingWithTim
Copy link
Collaborator

@connorchenn I think this looks good pretty much. Just one thing, the column name for arena-hard-auto is too crammed. Let's change it to arena-hard-auto and make the space for the column name larger so it fits in one line?
Screenshot 2024-07-15 at 4 58 19 PM

@CodingWithTim
Copy link
Collaborator

CodingWithTim commented Jul 16, 2024

@connorchenn Could you fix the code format? It is not passing the check.

@CodingWithTim CodingWithTim merged commit d310369 into lm-sys:main Jul 31, 2024
2 checks passed
bluechanel pushed a commit to bluechanel/FastChat that referenced this pull request Aug 19, 2024
@connorchenn Thanks for the contribution. This is amazing work!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants