Support InternLM

[monatis]

InternLM opensourced a new 7B-parameter model, and their published evaluation scores seem to be very promising.

According to their readme:

InternLM has open-sourced a 7 billion parameter base model and a chat model tailored for practical scenarios. The model has the following characteristics:

• It leverages trillions of high-quality tokens for training to establish a powerful knowledge base.
• It supports an 8k context window length, enabling longer input sequences and stronger reasoning capabilities.

A quick look at their source code reveals that it's based on GPTNeoX, saying the code is modified to accomodate minor architectural changes. It should be fairly easy to support InternLM with little to no modifications to the GPTNeoX example here. I'm mobile now, but I'll give it a try to it if anyone else has not yet.