Vision Eval for engineering examination question bank.

[MoreTore]

I have an engineering exam bank of about 1000 questions with simple illustrations. I have the questions already in JSONL format but some of them rely on the image to answer correctly.

Here is an example question without images.

{"question": "The exciter field current meter can be described as:", "options": ["a. A meter that measures and records the amount of power produced by the generator.", "b. A measurement of the reactive power being generated by the AC generation system.", "c. An indicator of the AC kilowatts the generator is producing.", "d. A measurement of the inductive power being generated by the AC generation system.", "e. Giving an indication of the DC current that is being supplied to the generators field windings."], "answer": "e"}

Answer stripped from prompt
GPT-3.5 output:

The exciter field current meter can be described as giving an indication of the DC current that is being supplied to the generator's field windings. Therefore, the correct option is (e) "Giving an indication of the DC current that is being supplied to the generator's field windings." The exciter field current meter is used to monitor and control the DC current supplied to the generator's field windings, which in turn determines the strength of the magnetic field produced and the amount of power that can be generated by the generator.

GPT-4 output:

The exciter field current meter can be described as:

e. Giving an indication of the DC current that is being supplied to the generators field windings.

I would like to make an eval for questions like this:

[jwang47]

Seems like this could be a good eval! Currently our models don't support taking in images. We can revisit this if it changes later.

[ryleymcc]

Seems like this could be a good eval! Currently our models don't support taking in images. We can revisit this if it changes later.

But wait, I thought GPT-4 allows images as input per the OpenAI release presentation.

[jwang47]

That's correct, GPT-4 supports images as input. However, currently the API we expose to the GPT-4 model doesn't support images.

[MoreTore]

Ok so its just a matter of time