No reconnection when candle socket stream gets dropped

[thomasd3]

Binance has a limit about the number of streams will it support on a socket. Right now, it stands at 128.

I'm grabbing the candles of the 139 futures with 8 timeframes per future, which means 139 * 8 = 1112 streams.

If you max out the number of streams / socket, Binance will just close the socket without telling you anything. And being at the max is not a good idea because a reply to a ping will open a new stream for the socket, and it will be dropped just like that.

I'm putting 120 streams per sockets, so I have: 139 futures * 8 time frames / 120 streams = 9 socket connections.

This means 15 futures / socket, and it works, I can stream all the data without problem.

But then I notice some streams are dropping sometimes. It's not regular, but rather random.
And when they're dropped, there is no connection / disconnection event sent.

I do not know if the library doesn't reconnect the socket, or if it reconnects but doesn't resend the subscription message. But, from my perspective, I can only know something is amiss when some futures don't have messages for a while.

I am wondering if Binance dropping the connection is detected. They don't send a notification saying it's being closed when you exceed the number of streams.

[thomasd3]

What I'm wondering about is that if I keep below the 128 streams / socket, and open multiple sockets, I shouldn't get any drops, but they still drop regularly over time.

[thomasd3]

Here is more info.

My subscriptions are:

[22:17:48.851] subscribed to 1000XECUSDT,1INCHUSDT,AAVEUSDT,ADAUSDT,ALGOUSDT,ALICEUSDT,ALPHAUSDT,ANKRUSDT,ANTUSDT,APEUSDT,API3USDT,ARPAUSDT,ARUSDT,ATAUSDT,ATOMUSDT,AUDIOUSDT,AVAXUSDT,AXSUSDT,BAKEUSDT,BALUSDT,BANDUSDT,BATUSDT,BCHUSDT,BELUSDT,BLZUSDT,BNBUS
DT,BNXUSDT,BTCDOMUSDT,BTCUSDT,BTSUSDT,C98USDT,CELOUSDT,CELRUSDT,CHRUSDT,CHZUSDT,COMPUSDT,COTIUSDT,CRVUSDT,CTKUSDT,CTSIUSDT,CVCUSDT,DARUSDT,DASHUSDT,DEFIUSDT,DENTUSDT,DGBUSDT,DOGEUSDT,DOTUSDT,DUSKUSDT,DYDXUSDT,EGLDUSDT,ENJUSDT,ENSUSDT,EOSUSDT,ETCUSDT,ETHU
SDT,FILUSDT,FLMUSDT,FLOWUSDT,FTMUSDT,FTTUSDT,GALAUSDT,GALUSDT,GMTUSDT,GRTUSDT,GTCUSDT,HBARUSDT,HNTUSDT,HOTUSDT,ICXUSDT,IMXUSDT,IOSTUSDT,IOTAUSDT,IOTXUSDT,JASMYUSDT,KAVAUSDT,KLAYUSDT,KNCUSDT,KSMUSDT,LINAUSDT,LINKUSDT,LITUSDT,LPTUSDT,LRCUSDT,LTCUSDT,MANAUS
DT,MASKUSDT,MATICUSDT,MKRUSDT,MTLUSDT,NEARUSDT,NEOUSDT,NKNUSDT,OCEANUSDT,OGNUSDT,OMGUSDT,ONEUSDT,ONTUSDT,OPUSDT,PEOPLEUSDT,QTUMUSDT,RAYUSDT,REEFUSDT,RENUSDT,RLCUSDT,ROSEUSDT,RSRUSDT,RUNEUSDT,RVNUSDT,SANDUSDT,SFPUSDT,SKLUSDT,SNXUSDT,SOLUSDT,SRMUSDT,STMXUS
DT,STORJUSDT,SUSHIUSDT,SXPUSDT,THETAUSDT,TOMOUSDT,TRBUSDT,TRXUSDT,UNFIUSDT,UNIUSDT,VETUSDT,WAVESUSDT,WOOUSDT,XEMUSDT,XLMUSDT,XMRUSDT,XRPUSDT,XTZUSDT,YFIUSDT,ZECUSDT,ZENUSDT,ZILUSDT,ZRXUSDT trade stream
[22:17:50.318] subscribed to 1000XECUSDT,1INCHUSDT,AAVEUSDT,ADAUSDT,ALGOUSDT,ALICEUSDT,ALPHAUSDT,ANKRUSDT,ANTUSDT,APEUSDT candle stream
[22:17:51.634] subscribed to API3USDT,ARPAUSDT,ARUSDT,ATAUSDT,ATOMUSDT,AUDIOUSDT,AVAXUSDT,AXSUSDT,BAKEUSDT,BALUSDT candle stream
[22:17:52.990] subscribed to BANDUSDT,BATUSDT,BCHUSDT,BELUSDT,BLZUSDT,BNBUSDT,BNXUSDT,BTCDOMUSDT,BTCUSDT,BTSUSDT candle stream
[22:17:54.258] subscribed to C98USDT,CELOUSDT,CELRUSDT,CHRUSDT,CHZUSDT,COMPUSDT,COTIUSDT,CRVUSDT,CTKUSDT,CTSIUSDT candle stream
[22:17:55.558] subscribed to CVCUSDT,DARUSDT,DASHUSDT,DEFIUSDT,DENTUSDT,DGBUSDT,DOGEUSDT,DOTUSDT,DUSKUSDT,DYDXUSDT candle stream
[22:17:56.908] subscribed to EGLDUSDT,ENJUSDT,ENSUSDT,EOSUSDT,ETCUSDT,ETHUSDT,FILUSDT,FLMUSDT,FLOWUSDT,FTMUSDT candle stream
[22:17:58.281] subscribed to FTTUSDT,GALAUSDT,GALUSDT,GMTUSDT,GRTUSDT,GTCUSDT,HBARUSDT,HNTUSDT,HOTUSDT,ICXUSDT candle stream
[22:17:59.724] subscribed to IMXUSDT,IOSTUSDT,IOTAUSDT,IOTXUSDT,JASMYUSDT,KAVAUSDT,KLAYUSDT,KNCUSDT,KSMUSDT,LINAUSDT candle stream
[22:18:01.012] subscribed to LINKUSDT,LITUSDT,LPTUSDT,LRCUSDT,LTCUSDT,MANAUSDT,MASKUSDT,MATICUSDT,MKRUSDT,MTLUSDT candle stream
[22:18:02.368] subscribed to NEARUSDT,NEOUSDT,NKNUSDT,OCEANUSDT,OGNUSDT,OMGUSDT,ONEUSDT,ONTUSDT,OPUSDT,PEOPLEUSDT candle stream
[22:18:03.663] subscribed to QTUMUSDT,RAYUSDT,REEFUSDT,RENUSDT,RLCUSDT,ROSEUSDT,RSRUSDT,RUNEUSDT,RVNUSDT,SANDUSDT candle stream
[22:18:05.088] subscribed to SFPUSDT,SKLUSDT,SNXUSDT,SOLUSDT,SRMUSDT,STMXUSDT,STORJUSDT,SUSHIUSDT,SXPUSDT,THETAUSDT candle stream
[22:18:06.394] subscribed to TOMOUSDT,TRBUSDT,TRXUSDT,UNFIUSDT,UNIUSDT,VETUSDT,WAVESUSDT,WOOUSDT,XEMUSDT,XLMUSDT candle stream
[22:18:07.841] subscribed to XMRUSDT,XRPUSDT,XTZUSDT,YFIUSDT,ZECUSDT,ZENUSDT,ZILUSDT,ZRXUSDT candle stream

The first one is trades where I can subscribe to all the tickers at once and this is never a problem.
Then all the other lines represent a socket connection each and the candles they are subscribed to. I use 8 timeframes per symbol and 10 symbols per candle, so 80 streams per socket. This is well below the limit (which I thought was 120, but it's 200).

And this runs well for many hours...

And then this happened ~10 hours later:

08:01:03.660] ⚠i KLine socket connection lost
[08:01:04.929] ⚠, KLine socket connection restored
[08:01:15.763] processed 128 candles,   0 kvwaps,     0 trades; rabbit: tpm 0ms, candles lag  0, kvwaps lag  0, trades lag  0
[08:01:15.763] missing candles: BANDUSDT, BATUSDT, BCHUSDT, BELUSDT, BLZUSDT, BNBUSDT, BNXUSDT, BTCDOMUSDT, BTCUSDT, BTSUSDT
[08:01:15.763] missing kvwap: BANDUSDT, BATUSDT, BCHUSDT, BELUSDT, BLZUSDT, BNBUSDT, BNXUSDT, BTCDOMUSDT, BTCUSDT, BTSUSDT
[08:01:15.763] missing trades: 1000XECUSDT, 1INCHUSDT, AAVEUSDT, ADAUSDT, ALGOUSDT, ALICEUSDT, ALPHAUSDT, ANKRUSDT, ANTUSDT, APEUSDT, API3USDT, ARPAUSDT, ARUSDT, ATAUSDT, ATOMUSDT, AUDIOUSDT, AVAXUSDT, AXSUSDT, BAKEUSDT, BALUSDT, BANDUSDT, BATUSDT,
BCHUSDT, BELUSDT, BLZUSDT, BNBUSDT, BNXUSDT, BTCDOMUSDT, BTCUSDT, BTSUSDT, C98USDT, CELOUSDT, CELRUSDT, CHRUSDT, CHZUSDT, COMPUSDT, COTIUSDT, CRVUSDT, CTKUSDT, CTSIUSDT, CVCUSDT, DARUSDT, DASHUSDT, DEFIUSDT, DENTUSDT, DGBUSDT, DOGEUSDT, DOTUSDT, DUSKUSDT
, DYDXUSDT, EGLDUSDT, ENJUSDT, ENSUSDT, EOSUSDT, ETCUSDT, ETHUSDT, FILUSDT, FLMUSDT, FLOWUSDT, FTMUSDT, FTTUSDT, GALAUSDT, GALUSDT, GMTUSDT, GRTUSDT, GTCUSDT, HBARUSDT, HNTUSDT, HOTUSDT, ICXUSDT, IMXUSDT, IOSTUSDT, IOTAUSDT, IOTXUSDT, JASMYUSDT, KAVAUSDT
, KLAYUSDT, KNCUSDT, KSMUSDT, LINAUSDT, LINKUSDT, LITUSDT, LPTUSDT, LRCUSDT, LTCUSDT, MANAUSDT, MASKUSDT, MATICUSDT, MKRUSDT, MTLUSDT, NEARUSDT, NEOUSDT, NKNUSDT, OCEANUSDT, OGNUSDT, OMGUSDT, ONEUSDT, ONTUSDT, OPUSDT, PEOPLEUSDT, QTUMUSDT, RAYUSDT, REEFU
SDT, RENUSDT, RLCUSDT, ROSEUSDT, RSRUSDT, RUNEUSDT, RVNUSDT, SANDUSDT, SFPUSDT, SKLUSDT, SNXUSDT, SOLUSDT, SRMUSDT, STMXUSDT, STORJUSDT, SUSHIUSDT, SXPUSDT, THETAUSDT, TOMOUSDT, TRBUSDT, TRXUSDT, UNFIUSDT, UNIUSDT, VETUSDT, WAVESUSDT, WOOUSDT, XEMUSDT, X
LMUSDT, XMRUSDT, XRPUSDT, XTZUSDT, YFIUSDT, ZECUSDT, ZENUSDT, ZILUSDT, ZRXUSDT

I consider something missing when there is no data for that symbol for 1 minute.

So it looks like we go disconnected, reconnected right away but the re-subscription is not working.

[JKorf]

Hi Thomas, this is rather hard to debug for me as well. The library should reconnect and resubscribe all topics when the reconnection is successful. If a resubscription on a reconnected socket fails the socket will be closed and resubscription will be tried again.
So in theory everything should be restored to the way it was.

Using the latest version, if you use Debug log level you should be able to log pretty much everything that's going on with the sockets, but not the data which is send/received. Is it in options to run with that? (I understand that previously running in Trace log level wasn't an option because of all the data being logged)