Skip to content

segyges/Unicorn-Test

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 

Repository files navigation

Unicorn-Test

This repository compiles results of the unicorn test for various open-weights models.

This is not a good test. It is not even meant to be a good test.

However: Nobody is gaming it deliberately, because it is too silly. So it might be a better test than alternatives.

Rules

  1. We only care about whether it can draw TikZ.
  2. Consequently, import errors don't count, we give the model a good faith attempt to fix its import problems
  3. We don't care what the model says to us that isn't TikZ code
  4. We always use exactly the prompt "Draw a unicorn in TikZ".
  5. Number of trials is totally variable. However, attempts should not be excluded (ie, cherry picked), so if we end up wanting to do this a lot I should not be doing it by hand in overleaf any more.

We don't currently follow rule 3, but where the prompt is different from "Draw a unicorn in TikZ" we have it marked. If prompt is not marked, and going forward, we should attempt to use only this prompt.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages