code on text-video qa

[cdqncn]

Dear authors,

I was wondering if you could release the code on text-video qa(e.g., dataloader and how you process the videos).

Thanks!

[LiJunnan1992]

Hi, you can refer to the code here for dataloading of text-video qa: https://github.com/salesforce/ALPRO. Thanks!

[cdqncn]

Thanks for your reply, this code is classification, I just want to learn how to generation for QA in BLIP.

[LiJunnan1992]

We use the VQA model to generation answers:

BLIP/models/blip_vqa.py

Line 85 in 48211a1

question_output = self.text_encoder(question.input_ids,

To handle videos, we simply concatenate frame features and pass them to the text decoder.

[BlueCat7]

@cdqncn Hi, have you reproduce the author result in zero-shot video QA. I tried to do it, but failed.

[dipta007]

We use the VQA model to generation answers:

BLIP/models/blip_vqa.py

Line 85 in 48211a1

question_output = self.text_encoder(question.input_ids,

To handle videos, we simply concatenate frame features and pass them to the text decoder.

@LiJunnan1992 By concat, do you mean, after getting the image (frame) encoding, all the encodings are concatenated and then passed that raw concatenated embeddings to the decoder? Thanks for such awesome repo by the way.