feat: OpenAI: JSON mode

[Butch78]

Hi @santiagomed,

Congrats again on 100 Stars ⭐⭐

I tried to implement OpenAi's new JSON mode, unfortunately, I couldn't get my test to work, I'm not sure why but I think I'm using the response_format: Option<ResponseFormat> on the Payload struct incorrectly. Any help would be greatly appreciated!

[santiagomed]

Hi @Butch78! Thanks again! I've been a bit busy but will review this when I have time sometime this week!

[Ap3lsin4k]

All tests seem to fail for the same reason.

thread 'pipeline::simple::test::test_generate' panicked at orca/src/pipeline/simple.rs:281:54:
called `Result::unwrap()` on an `Err` value: error decoding response body: missing field `id` at line 8 column 1

Did API change? Did they previously send body: {id: ...}?
I don't see the id with value in the logs.
Just from the error message I infer that we expect id in the response, and OpenAI removed the field.

I assume we have a structure to represent OpenAI's Response body. We may remove the field called id.

[Ap3lsin4k]

I suggested a change to test, consistent naming, increase type safety as well as pointed to possible bug fix.

[Ap3lsin4k]

I like how concrete this "{" is. It is easy to read.

There is a comment above, is something not obvious from the test name. I would C\consider an assert instead of a comment.
Something like

assert!(response.is_valid_json())

I would leave the first assert too assert!(response.to_string().starts_with("{"));

[] is a JSON. May open AI return list of dictionaries in JSON?
If so the comment is not accurate better without it.

[Ap3lsin4k]

second thought: I remember Uncle Bob's suggestion source code should be general, tests be concrete.
Maybe assert_eq!(response, "{ \"country\": "berlin"}")

[Ap3lsin4k]

Nitpick:
Hmm, I am wondering if it will be better without Option...

Suggested change

	response_format: Option<ResponseFormat>,
	response_format: ResponseFormat,

Maybe it is possible to override the default zero value. So users don't have to set it, at the same time may read more about default format.

[santiagomed]

@Butch78 I was pretty sick last week and then it was Thanksgiving, so didn't have time to review. I will try and review it this week. Sorry for the delay!

[Butch78]

I hope you are feeling better and had a great Thanksgiving 🦃 @santiagomed, I've been busy myself but I tried to simplify the chat-completions functionality :)

I still wasn't able to get my teststo pass though :/ It's something to do with the ResponseFormat but I can't figure out what breaks the chat compeletions

[santiagomed]

Hi @Butch78 . I found the issue. The raw response body returns this:

{
  "error": {
    "message": "'json_object' is not of type 'object' - 'response_format'",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

This is what the payload currently looks like:

Payload {
    model: "gpt-3.5-turbo-1106",
    prompt: None,
    max_tokens: 1024,
    temperature: 1.0,
    stop: None,
    messages: [
        Message {
            role: User,
            content: "What is the capital of France?",
        },
        Message {
            role: Assistant,
            content: "Paris",
        },
        Message {
            role: User,
            content: "What is the capital of Germany?",
        },
    ],
    stream: false,
    response_format: JsonObject, // serializes to "json_object"
}

You are just passing the enum to the payload which serializes to "json_object", where it should serialize to { "type": "json_object" }!

[Butch78]

Thank you @santiagomed for you help!
The word JSON must be present in the Promp when using JSON mode:https://platform.openai.com/docs/guides/text-generation/json-mode.

I just modified the prompt to include it but maybe in the future when this implemented:
#16