Add an API example using server.cpp similar to OAI.

[jwj7140]

adding an API example that provides responses similar to OpenAI's chat completion and completion.
This example is about 30% faster than existing similar examples because they are based on llama-cpp-python, which is slightly slower than llama.cpp.
This example must be used with server.cpp.

Run this code, and write like this in python:
openai.api_base = "http:https://***.***.***.***:8081"
Then almost all OpenAI api code is compatible with llama.cpp.

[SlyEcho]

There is some formatting problem that the CI picked up, can you take care of it?

[jwj7140]

um... Should I make new branch and PR again?
I don't know about github system well. sorry

[SlyEcho]

No, if you open the failing check it will tell you where it needs fixing.

examples/server/api_like_OAI.py:
	Wrong line endings or no final newline
	Not all lines have the correct end of line character
	73: Trailing whitespace

3 errors found

Some whitespace issues, should be easy.

[jwj7140]

yep thank you

[SlyEcho]

Is it possible to get a binding for also /v1/ prefixed endpoints?

@app.route('/completions', methods=['POST'])
@app.route('/v1/completions', methods=['POST'])
def completion():

It seems like this would give better compatibility?

[howard0su]

Shall we change server example to match response of OAI?

[jwj7140]

@SlyEcho Actually I think that it doesn't seem to matter much, but I'll add it. thank you.
server.cpp seems to return a "truncated" contents in many times(with "truncated:true" logs). Can you tell me what is "truncated" and why it is showed?

[jwj7140]

@howard0su Well, I think the current server example has the most comfortable forms. OAI api is just an experimental attempt.

[SlyEcho]

Can you tell me what is "truncated" and why it is showed?

Truncated indicates the automatic context management when the context runs out of space. Some explanation here: #1838.

It is just an indicator to the API's user, this way they could create some kind of method to handle the lost information in the context. For example they could use another LLM to summarize the previous chat and then replace the earlier chat messages leaving more space for new questions and answers.

[SlyEcho]

Probabilities are now returned from the server in #1962

Would it be hard to add them in to this?

[jwj7140]

great! Should I make PR again?

[SlyEcho]

We can add it later as well, it's up to you, otherwise we can merge it as-is.

[jwj7140]

Okay I'll add it later. It is okay to merge. thanks

[SlyEcho]

It would be nice to add all this to the C++ API. Maybe in the future. But I will try to use this wrapper so I get an idea of what what possible changes to the server are needed.

[ANMahmood]

Hi,
I am just trying to get python api_like_OAI.py running but having the following issue:
In one terminal, the server loads fine:

./server --model ~/Downloads/LLM_Applications/LocalAI/models/wizardcoder-python-34b-v1.0.Q8_0.gguf --ctx-size 16000 --threads 20 --alias gpt-4 --embedding --host 127.0.0.1 --port 8000
llama server listening at http:https://127.0.0.1:8000

{"timestamp":1696079264,"level":"INFO","function":"main","line":1623,"message":"HTTP server listening","hostname":"127.0.0.1","port":8000}

second terminal starts api_likeOAI.py but when I send requests to this server, it can't find the model (gpt-4):

python3 api_like_OAI.py 
 * Serving Flask app 'api_like_OAI'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http:https://127.0.0.1:8001
Press CTRL+C to quit
127.0.0.1 - - [30/Sep/2023 21:39:44] "GET /v1/models HTTP/1.1" 404 -
127.0.0.1 - - [30/Sep/2023 21:39:45] "GET /v1/models HTTP/1.1" 404 -
127.0.0.1 - - [30/Sep/2023 21:39:46] "GET /v1/models HTTP/1.1" 404 -

I am sending requests to gpt-4 as that's the alias set with server. What am I doing wrong?

[cebtenzzre]

I am just trying to get python api_like_OAI.py running but having the following issue:

If you open a new issue with that information, you are more likely to get a response.

[jwj7140]

@ANMahmood

I am sending requests to gpt-4 as that's the alias set with server. What am I doing wrong?

This example does not have "/models" endpoint that exist in actual OpenAI's API because server.cpp supports only one model to run. So...if there are any problems with your project because of this issue, removing code may be helpful.