Can't establish secure SSL TLS v1.2 connection among mosquitto mqtt broker and the client

[SamShekhovtsov]

The mosquitto mqtt broker (latest verison available) is configured to use TLS v1.2, it is verified that the server works fine using the TLS encryption with live clients like the following:
https://testclient-cloud.mqtt.cool/
MQTT-Explorer http:https://mqtt-explorer.com/
The client application is setup to support the TLS v1.2 encryption. The client application can connect to the live MQTT brokers like AWS Amazon MQTT broker and the https://www.hivemq.com/ broker. Everything works fine with the TLS v1.2 encryption, that data exchange is protected.
When trying to pair the mosquitto mqtt broker and the client application - the TLS handshake failure is experienced:

OpenSSL Error[0]: error:140370E5:SSL routines:ACCEPT_SR_KEY_EXCH:ssl handshake failure

when I try to debug the SSL/TLS using the Wireshark, I see that there is a Client Hello, after that the Server Hello, then Certificate, Server Key Exchange, Server Hello Done and it stops at that point.

After the deep investigation, it looks like the mosquitto MQTT broker is using the LibreSSL (libssl) library and not the OpenSSL (one of the forks) as it stated.
There are two possibilties: (1) server is indeed sending something bad in the ServerKeyExchange to this client, but other clients don't see the problem because they don't use the DHE (at all), don't use the same type of DHE (FF vs EC), or don't use the same curve/group for DHE, or get a different signature type. (2) server is sending valid ServerKeyExchange that triggers a bug in client, but other servers don't because they don't use DHE, don't use the same DHE, or the same curve/group, or the same signature.