Request for Repo: "Open Source, Daily Auto-Generated SOTA LLM Model Benchmarks"

[Marviel]

✨ Open Source Daily Auto-Generated SOTA LLM Model-Comparisons Repository

(Sorry for posting here, just not sure where to ask)

Does ^^ this already exist?

If So, Where??

• EcoSystem Graphs Does not appear to have benchmarks.

It's hard to keep up.

All the open source LLM/AI repositories are becoming impossible for basically any human to keep pace with.

There are many awesome "snippets" that are posted in public channels, but not all models hold up well and generalize after practice.

💭 Feature Requests

Here's what I'd like to see: (Please add your own in comments)

License

• GPLv3.0 Affero

Update Cadence

• Daily Evaluation Runs -- auto-updating the Github Repo with up-to-date Evaluation Results as described below.

Eval Result DB

Description

Each time the cron job is run (daily) the evaluations should be written into a database.

Properties

The Output of the Cron Job should be a set of entries into the EvalDB, showing:
(1) The prompt / Input
(2) The model, including its known current parameters and limitations at the time of the eval run
(3) The output

Up-To-Date Model DB

Model DB Description

There should be an updated database of models which is displayed on the README.md

Model DB Properties

This should include the following:

API or Self-Hosted

This is critical for both speed and price reasons.

Modes & Mode Parameters

• Text
  • Chat vs. Standard Completion
  • Cost-Per-Token
  • Average Speed-Per-Token
  • Context Length
  • Unicode-Support (most models?)
• Image
  • Max Context Size
  • Cost-Per-Pixel(?)
• Video
  • Max Context Length
  • Max Input

Included Training Datasets

Tags

A free-form field for anything that doesn't fit into the above schema.

Does Something Like This Exist?

?????

What else do we need?

?????

I'm trying to make something to fill this niche myself and will link here shortly.

[SierotkaM]

śr., 29 mar 2023, 17:33 użytkownik Luke Bechtel ***@***.***> napisał:

…

✨ Open Source Daily Auto-Generated SOTA LLM Model-Comparisons Repository It's hard to keep up. All the open source LLM/AI repositories are becoming impossible for basically any human to keep pace with. There are many awesome "snippets" that are posted in public channels, but not all models hold up well and generalize after practice. 💭 Feature Requests Here's what I'd like to see: (Please add your own in comments) License - GPLv3.0 Affero Update Cadence - Daily Evaluation Runs -- auto-updating the Github Repo with up-to-date Evaluation Results as described below. Eval Result DB Description Each time the cron job is run (daily) the evaluations should be written into a database. Properties The Output of the Cron Job should be a set of entries into the EvalDB, showing: (1) The prompt / Input (2) The model, *including its known current parameters and limitations at the time of the eval run* (3) The output Up-To-Date Model DB Model DB Description There should be an updated database of models which is displayed on the README.md Model DB Properties This should include the following: API or Self-Hosted This is critical for both speed and price reasons. Modes & Mode Parameters - Text - Chat vs. Standard Completion - Cost-Per-Token - Average Speed-Per-Token - Context Length - Unicode-Support (most models?) - Image - Max Context Size - Cost-Per-Pixel(?) - Video - Max Context Length - Max Input Included Training Datasets Tags A free-form field for anything that doesn't fit into the above schema. Does Something Like This Exist? ????? What else do we need? ????? I'm trying to make something to fill this niche myself and will link here shortly. — Reply to this email directly, view it on GitHub <#505>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6PJHHVHLTHCIFXBPR24VYDW6RI6NANCNFSM6AAAAAAWMCEU6A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>