[BUG] server disconnects the TCP connection while TLS connection is not complete, client will dead loop

[CastleOnTheHill]

mosquitto/lib/net_mosq.c

Line 576 in ff97fbf

if (err == SSL_ERROR_SYSCALL) {

mosquitto version: 1.6.7
openssl version: 1.1.1k
platform: linux 4.14

This problem still exists in the latest version。

Hi, I'm working with mosquitto and i found: When the TCP connection is complete, but the TLS connection is not complete. If the server disconnects the TCP connection, In this case, SSL_get_error returns the error code SSL_ERROR_SYSCALL.
In the current implementation, if the error code is SSL_ERROR_SYSCALL, the mosquitto_loop function returns success and continues to invoke SSL_connect in the next mosquitto_loop,

Because the TCP connection is disconnected, SSL_connect immediately returns the failure error code SSL_ERROR_SYSCALL and invoke the next mosquitto_loop again. As a result, the program enters a dead loop.
This dead loop causes high CPU usage. In fact, the mosquitto_loop does not return failure until the mqtt keepalive time expires.

According to https://www.openssl.org/docs/man1.1.1/man3/SSL_get_error.html

SSL_ERROR_SYSCALL
Some non-recoverable, fatal I/O error occurred. The OpenSSL error queue may contain more information on the error. For socket I/O on Unix systems, consult errno for details. If this error occurs then no further I/O operations should be performed on the connection and SSL_shutdown() must not be called.

SSL_ERROR_SYSCALL is fatal error，when SSL_get_error return SSL_ERROR_SYSCALL, we should return an error to the user。

[CastleOnTheHill]

I merge Backport SSL connect fixes. in my local workspace。program
passed the above situation successfully, and the code is indeed a lot cleaner。

[ralight]

Great, thank you very much for confirming.