I believe there is an implementation error in SAN

[waystogetthere]

Hello, I believe that this line is an error:

san/san/__init__.py

Line 73 in bb9aaea

attended_matrix = self.multi_head[k](input_space) * input_space

I think it should be:

attended_matrix = self.softmax(self.multi_head[k](input_space)) * input_space

as shown in the paper:

[SkBlaz]

Hello! Thanks for the issue/report, nice catch. I'd be grateful if you opened a PR with the change - the left-out softmax was a remnant of refactoring that made the source a bit more user friendly (and it appears it's starting to pay off!)

[waystogetthere]

Hello,
Thanks for your fast reply! Merry Xmas & Happy New year!
Yeah, I would make a pull request soon.

I think the 'self.multi_head[k]' is the k-th head attention which is Linear Layer with weight and bias: $W_{l_{att}}^k$ and $b^k_{l_{att}}$. So the softmax is needed

[SkBlaz]

Great, thanks. Indeed, to be aligned with the paper, the activation is required. Surprisingly, current version (multilinear blocks basically) also seems to work, might be worth further exploration at some point.

[SkBlaz]

The suggested change was merged to master.