Chore Support base64 embedding format

[henomis]

Issue: #473

This PR adds the support for base64 embedding format. As discussed in the issue, this format is not officially documented, however it is used in the official python library.

The following go example can be used to test the change:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/sashabaranov/go-openai"
)

func main() {

	client := openai.NewClient(os.Getenv("OPENAI_API_KEY"))

	res, err := client.CreateEmbeddings(
		context.Background(),
		openai.EmbeddingRequest{
			Input:          "Your text string goes here",
			Model:          openai.AdaEmbeddingV2,
			EncodingFormat: openai.EmbeddingEncodingFormatBase64,
		},
	)
	if err != nil {
		panic(err)
	}

	data, err := json.MarshalIndent(res, "", "  ")
	if err != nil {
		panic(err)
	}

	fmt.Println(string(data))
}

The following example is provided to compare the result using the python implementation:

import openai
import os

openai.api_key = os.getenv("OPENAI_API_KEY")

text = "Your text string goes here"

response = openai.Embedding.create(
    model="text-embedding-ada-002", 
    input=text, 
    kwargs={"encoding_format": "base64"},
    )
print(response)

[codecov]

Codecov Report

Merging #485 (e36e114) into master (25da859) will increase coverage by 0.13%.
Report is 1 commits behind head on master.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #485      +/-   ##
==========================================
+ Coverage   97.30%   97.43%   +0.13%     
==========================================
  Files          18       18              
  Lines         780      820      +40     
==========================================
+ Hits          759      799      +40     
  Misses         15       15              
  Partials        6        6              

Files Changed	Coverage Δ
embeddings.go	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

[sashabaranov]

Shouldn't we update res otherwise?

[henomis]

Oh, yes! My bad! I'll fix it

[sashabaranov]

Thank you for the update! It's a bit hard to figure out res manipulations here. Could we please get rid of the named res return here?

[henomis]

Yeah! It wasn't so clear, but I left it as I found it to minimize the modifications. However, now named returns have been removed.

[sashabaranov]

Ahhh, I didn't realize how much more problematic would it make if err != nil returns!

Sorry for bothering you so much, feelks like we've stumbled upon a tricky piece here :D I think that the var embeddingResponse any = &response creates a level of indirection which is not trivial to understand.

What do you think of this approach?

func (c *Client) CreateEmbeddings(
	ctx context.Context,
	conv EmbeddingRequestConverter
) (res EmbeddingResponse, err error) {
	baseReq := conv.Convert()
	req, err := c.newRequest(ctx, http.MethodPost, c.fullURL("/embeddings", baseReq.Model.String()), withBody(baseReq))
	if err != nil {
		return
	}

	if baseReq.EncodingFormat != EmbeddingEncodingFormatBase64 {
		err = c.sendRequest(req, &res)
		return
	}

	base64Response := &EmbeddingResponseBase64{}
	err = c.sendRequest(req, base64Response)
	if err != nil {
		return
	}

	res, err = base64Response.ToEmbeddingResponse()
	return
}

[henomis]

This was one of my first ideas. Even though it will duplicate the code on calling twice sendRequest is the one with a good readability.

[henomis]

Code and tests have been refactored.

[sashabaranov]

Thank you so much!

[sashabaranov]

Yeah, this test would not catch res not being updated, could you please update test server here so it would return non-empty EmbeddingResponse?

[sashabaranov]

Seems like we have a problem with non-base64 cases here

[sashabaranov]

@henomis seems like the test coverage is failing now

[henomis]

It's the coverage for this piece of code

err = c.sendRequest(req, base64Response)
if err != nil {
	return
}

in my first implementation c.sendRequest() was the last instruction of the function (like on the other tests files where sendRequest has been used) and the coverage was ok.

[henomis]

Ok, inserted a test case to make sendRequest fail

[sashabaranov]

@henomis thank you so much!