Support OpenAI's Batch API

[djstrong]

How complex would be adding OpenAI's Batch API? For now, GPT-4 benchmark of Polish tasks would cost me $1500 :(

[xksteven]

It's not very difficult for the most part it's replacing these lines

such as

                response = oa_completion(
                    client=self.client,
                    chat=True,
                    messages=inps,
                    model=self.model,
                    **kwargs,
                )

with the following

with open("open_ai_batch.jsonl", "a") as f:
    f.write(json.dumps(inps))

# Then after the json file is written
....
# upload the file
batch_input_file = client.files.create(
    file=open("batchinput.jsonl", "rb"),
    purpose="batch"
)

batch_input_file_id = batch_input_file.id

client.batches.create(
    input_file_id=batch_input_file_id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={
      "description": "nightly eval job"
    }
)

We need a new way (function) to get back and check the results

# check results
client.batches.retrieve(batch_input_file_id)

# retrieve results once done
content = client.files.content("output_file")

The devil is in the details though as the jsonl file can only be called for 1 model type and how do you handle multiple files/batches etc. For the simple case of evaluating 1 model 1 batch it's pretty simple change.