MM-32950: Reliable WebSockets: Basic single server

[agnivade]

This PR adds reliable websocket support for a single server.

Below is a brief overview of the three states of a connection:

Normal:

• All messages are routed via web hub.
• Each web conn has a send queue to which it gets pushed.
• A message gets pulled from the queue, and before it
  gets written to the wire, it is added to the dead queue.

Disconnect:

• Hub Unregister gets called, where the connection is just
  marked as inactive. And new messages keep getting pushed
  to the send queue.

If it gets full, the channel is closed and the conn gets removed
from conn index.

Reconnect:

• We query the hub for the connection ID, and get back the
  queues.
• We construct a WebConn reusing the old queues, or a fresh one
  depending on whether the connection ID was found or not.
• Now there is a tricky bit here which needs to be carefully processed.
  On register, we would always send the hello message in the send queue.
  But we cannot do that now because the send queue might already have messages.

Therefore, we don't send the hello message from web hub, if we reuse a connection.

Instead, we move that logic to the web conn write pump. We check if
the sequence number is in dead queue, and if it is, then we drain
the dead queue, and start consuming from the active queue.
No hello message is sent here.

But if the message does not exist in the dead queue, and the sequence number
is actually something that should have existed, then we set
a new connction id and clear the dead queue, and send a hello message.
The client, on receiving a new connection id will automatically
set its sequence number to 0, and make the sync API calls to manage
any lost data.

https://mattermost.atlassian.net/browse/MM-32590

NONE

[agnivade]

Webapp changes here: mattermost/mattermost-webapp#7921

[agnivade]

I have tested this manually for all possible scenarios by manually tweaking the code to introduce errors.

Also tested in race mode, and ran a small load test to verify any edge cases.

[streamer45]

Looks very good 🎉. I really appreciated the thorough commenting.
Gave this a first pass and left a few comments.

[agnivade]

I really appreciated the thorough commenting.

Definitely, I needed the comments to keep track of things myself. If I didn't comment some thing, the next day, I would do the same thing, and then remember why I did not choose to do something.

I think the end result is more or less manageable. Initially, it looked like it would get way more complicated. :P

[agnivade]

@streamer45 - Take a look now. The API layer is significantly cleaned up.

[isacikgoz]

Looks good to me, awesome work @agnivade 🎉 I have minor things but none of the are blocking.

[streamer45]

Awesome 👍 , just one question remaining :)

[streamer45]

Looks great. Brilliant work 👍

[agnivade]

@saturninoabril - This needs to be tested along with mattermost/mattermost-webapp#7921 and https://github.com/mattermost/enterprise/pull/973.

Please set "EnableReliableWebSockets": true in service settings, and test websocket related functionalities for any regression.

Thanks!

[saturninoabril]

Thanks @agnivade! I'll wait for mattermost/mattermost-webapp#7921 to be ready for testing. Once ready, please update all the repos and I'll get the build image for testing.

[agnivade]

/update-branch

[agnivade]

@saturninoabril - You should be good to go now. Thanks!

[saturninoabril]

Tested and passed. Thanks @agnivade!