Building into Godot game engine

[st4rl1ght1337]

Hey, I'm not sure if this is the best place to post this. I couldn't find an IRC or discord channel associated with the project. I need to build ggml into my Godot game, using this https://docs.godotengine.org/en/stable/tutorials/scripting/gdextension/gdextension_cpp_example.html or this https://docs.godotengine.org/en/3.0/development/cpp/custom_modules_in_cpp.html

I'm not sure why the example documents have python dependencies, does the ggml binary require python to run? Confused as I thought it was all in C++. If someone could point me in the right direction it would be helpful

[slaren]

The python files in the examples are used to convert the model files to ggml format, they are not required to run ggml.

[st4rl1ght1337]

The python files in the examples are used to convert the model files to ggml format, they are not required to run ggml.

Ah I see, thanks.

[st4rl1ght1337]

So related question
I basically just need to call whatever the equivalent of these methods are:

Load model
Prompt model

Anyone know where that is in the source code?

[slaren]

ggml doesn't provide directly functions to do these things directly, it is a low level library used to build and run computation graphs. You would need to study the source code of the example of the model that you want to use and figure a way to adapt it to your needs.
If you need a higher level library, you should look into https://github.com/ggerganov/llama.cpp, but currently it only supports LLaMA models.

[st4rl1ght1337]

ggml doesn't provide directly functions to do these things directly, it is a low level library used to build and run computation graphs. You would need to study the source code of the example of the model that you want to use and figure a way to adapt it to your needs. If you need a higher level library, you should look into https://github.com/ggerganov/llama.cpp, but currently it only supports LLaMA models.

I've been using https://github.com/LostRuins/koboldcpp but it's too high level for me. So for instance, what I want to do is call something like the -p flag ./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "This is an example" as a method in the game code.

One of the issues is it seems like llama.cpp can't run the models I'm trying to run

[fire]

I've been advising @st4rl1ght1337, I think the best approach is pick an end model like whisper.cpp / llama.cpp and integrate it directly.

[brightening-eyes]

you can implement a class based on Node from godot-cpp and directly integrate your model there. things like llama.cpp and whisper.cpp can be used as well.
then in your scene you will instanciate your class, and in your gdScript / C# code, you can call the model.

[st4rl1ght1337]

I've been advising @st4rl1ght1337, I think the best approach is pick an end model like whisper.cpp / llama.cpp and integrate it directly.

It seems like fire had success compiling ggml into the godot engine as a module and is in the process of getting whisper to work. I think his repo is up now

For me, I need to run the Pygmalion 6B model, which is based on GPTJ. There is an example of running a GPTJ model in the docs, so I'll see if I can call it using fire's compilation.

@brightening-eyes I did try my hand at compiling KoboldCpp into Godot, but both KoboldCpp and ggml are written in C without classes, so I wasn't sure what class to define in my register_types.cpp ClassDB::register_class field, which specifies something like

/* register_types.cpp */

#include "kobold-caller.cpp"

void initialize_ggml_module(ModuleInitializationLevel p_level) {
    if (p_level != MODULE_INITIALIZATION_LEVEL_SCENE) {
            return;
    }
    ClassDB::register_class<kobold_caller>();
}

void uninitialize_ggml_module(ModuleInitializationLevel p_level) {
    if (p_level != MODULE_INITIALIZATION_LEVEL_SCENE) {
            return;
    }
   // Nothing to do here in this example.
}

I tried programming kobold_caller as a C++ wrapper class to access the main functions in expose.cpp of kobold.cpp (see LostRuins/koboldcpp#350), but I haven't had any success in getting the compilation to succeed. Expose.cpp returns many errors / warnings in compilation when called in my wrapper class and I'm a bit too noobish in C/C++ to accurately troubleshoot them. I get the feeling I may have an easier time of it trying to define wrapper classes for ggml as the codebase is comparatively smaller

I think Fire's repo probably has this defined, I'll pull it up

https://github.com/V-Sekai/v-sekai.whisper/blob/main/register_types.cpp

I guess what I'm trying to figure out is how to use something like whisper.cpp or llama.cpp in order to call the ggml GPTJ example. In so far as I've been told llama.cpp can't load gptj ggml files, but I think @fire said they didn't have any problems loading it

[brightening-eyes]

@st4rl1ght1337
you need to make a class, and then make _bind_methods etc (I exactly dont remember the name) there.
in constructor, you need to create the structs, and in the destructor, free them.
then, create your methods that calls into ggml, llama.cpp, KoboldCpp, etc.
in _bind_methods (which is static I suppose), you need to register your methods, signals etc. for example, when a token is processed, you can emit a signal and control it in your gdScript and align your text generation in game frames I think.