Wrong dimension for L2 normalization?

[dzabraev]

When ReduceDim

mmt/model/model.py

Lines 715 to 725 in 0d848cd

	class ReduceDim(nn.Module):
	def __init__(self, input_dimension, output_dimension):
	super(ReduceDim, self).__init__()
	self.fc = nn.Linear(input_dimension, output_dimension)
	def forward(self, x):
	x = self.fc(x)
	x = F.normalize(x)
	return x

is applied to video expert embeddings it do F.normalize on shape (batch_size, num_tokens, embedding_dim), and reduce by default along dim=1 (num_tokens)

mmt/model/model.py

Lines 431 to 434 in 0d848cd

	if self.vid_inp in ['both', 'temp', 'all']:
	for mod in self.modalities:
	layer = self.video_dim_reduce[mod]
	experts_feats[mod] = layer(experts_feats[mod])

But here it is applied to shape (batch_size, embedding_dim)

mmt/model/model.py

Lines 423 to 426 in 0d848cd

	for mod in self.modalities:
	layer = self.video_dim_reduce[mod]
	mnp_experts[mod] = layer(pooled_experts[f'{mod}_avgpool'])
	maxp_experts[mod] = layer(pooled_experts[f'{mod}_maxpool'])

So normalization along embedding_dim axis.

Is it mistake or not?

[gabeur]

Good catch, thank you!
Indeed it was a mistake.
I have pushed a fix.
Fixing the bug seems to improve performance slightly, not significantly.