Does vistion tower trained during starge 2 (Visual Instruction Tuning)?

[GoGoJoestar]

I find @torch.no_grad() in CLIPVisionTower.forward(), so it won't flow gradient to CLIP while training.

LLaVA/llava/model/multimodal_encoder/clip_encoder.py

Lines 45 to 57 in c121f04

	@torch.no_grad()
	def forward(self, images):
	if type(images) is list:
	image_features = []
	for image in images:
	image_forward_out = self.vision_tower(image.to(device=self.device, dtype=self.dtype).unsqueeze(0), output_hidden_states=True)
	image_feature = self.feature_select(image_forward_out).to(image.dtype)
	image_features.append(image_feature)
	else:
	image_forward_outs = self.vision_tower(images.to(device=self.device, dtype=self.dtype), output_hidden_states=True)
	image_features = self.feature_select(image_forward_outs).to(images.dtype)
	return image_features

However, here is a key: "mm_vision_tower_lr": 2e-06," in model's config.json file, and in the LLaVA-NEXT blog on May 25th, the vision tower are training during stage-2 with lr=2e-6.

Were the previous models trained according to this strategy? Will training CLIP be better when training for downstream task?

[2U1]

I think the training code isn't open yet for the LLaVA-NEXT.

[PangziZhang523]

Print out the gradient，while requires_grad=True，the parameter.grad=None?