Add better test coverage for models

[haileyschoelkopf]

We should improve our test coverage of the various model classes supported.

Ideally we'd have at least https://github.com/EleutherAI/lm-evaluation-harness/blob/main/tests/models/test_huggingface.py tests for every model class. This would require, for API models, mocking out the API call with a dummy one, and probably something similar for VLLM, which currently requires CUDA to be run so we don't have the tests run on the CI.

Help or contributions on this welcome! Else, I'll look into this hopefully soonish.

[raamana]

Hi Hailey, I am interested in contributing to this great project and helping. Let me take spend some time on the docs and library to learn how it is setup and how the tests are designed, so I can help improve the test coverage. I will ask you more questions as I get going and run into any knowledge blocks :)

[raamana]

Hi @haileyschoelkopf , actually would you or one of your colleagues point me to a video or design doc that explains how the code base is setup and how to navigate it? from my quick look, it is complex even for an experienced coder like me, so a little kickstart wtih an existing developer would make it easier. I am happy to do a video chat also if that's easier.

[haileyschoelkopf]

Hey @raamana , I don't have the bandwidth to do a video chat unfortunately, but we'll be putting out a preprint soon that has in part a description of the high-level moving pieces / design!

If you try to understand the codebase but hit some snags, feel free to ping on discord in the #lm-thunderdome channel of EleutherAI's discord (link should be in readme) and I or someone else will probably be able to respond! It would be helpful to know what components are most confusing or should be better documented somewhere.