Multimodal transcribers (v2)

[ZanSara]

Multi modal transcribers convert image/audio/video documents into text documents.

The main question about these components, however, is what input should they deal with in order to be able to function both in Indexing and in query scenarios.

• input path, output document: Works well for indexing, clumsy for query (document needs to be converted back to string)
• input document, output document: Same as above
• input path, output string: works for query, doesn't work for indexing (metadata is likely lost, for example whisper timestamps)

Currently WhisperTranscribers for v2 follow the path --> document pattern, but that makes them ugly to use in query pipelines.

Once we decide on a strategy, all transcribers should work similarly:

Tasks

Beta Give feedback

1. ImageTranscriber (v2)
2. AudioTranscriber (v2)
3. VideoTranscriber (v2)

Existing work:

• LocalWhisperTranscriber (v2) #4909
• RemoteWhisperTranscriber (v2) #4910