A question about the optimizer:

[HrealcodeH]

There are two functions in VALOR/optim/misc.py, one is build_optimizer and the other is build_optimizer_for_VQA. Is the second one specifically for the VQA task, while the first one is for other tasks? Which function did you use to obtain the results listed in the paper?

[HrealcodeH]

I also have another question. The results reported in the paper are on the test set, correct? (assuming the dataset used has a separate test set). Were those results obtained by first training the model and saving the checkpoint with the best performance on the validation set, then evaluating that checkpoint on the test set? Or were the test set results obtained by directly using the test set in place of a validation set during training? I would greatly appreciate it if you could clarify this. Please let me know if I can provide any other details.

[HrealcodeH]

I'm sorry, I didn't notice you had provided the json file for the datasets. Please ignore the 2nd question. But I'm still wondering if the build_optimizer_for_VQA in VALOR/optim/misc.py is used?

[TXH-mercury]

I'm sorry, I didn't notice you had provided the json file for the datasets. Please ignore the 2nd question. But I'm still wondering if the build_optimizer_for_VQA in VALOR/optim/misc.py is used?

Hi @HrealcodeH , 'build_optimizer_for_VQA' is not used in the latest version of VALOR and please just ignore it. It is used when VQA is modeled as a classification task instead of generation task at the earlier development stage of VALOR.

[HrealcodeH]

I'm sorry, I didn't notice you had provided the json file for the datasets. Please ignore the 2nd question. But I'm still wondering if the build_optimizer_for_VQA in VALOR/optim/misc.py is used?

Hi @HrealcodeH , 'build_optimizer_for_VQA' is not used in the latest version of VALOR and please just ignore it. It is used when VQA is modeled as a classification task instead of generation task at the earlier development stage of VALOR.

@TXH-mercury Thank you for your previous answer, it was very helpful. I have a new question now: the audiocaps in the processed json files you provide for most finetuning datasets appear to be empty. Could you please add the audiocaps?

[TXH-mercury]

audiocaps is added