No data sent/recv when using mosquitto_connect_async with my own loop

[doragasu]

I have an application that uses asynchronous operation without the threaded interface. I need to work like this because I have a library managing file descriptors using epoll, and I need to use it to manage the mosquitto socket.

This was working flawlessly with mosquitto 1.4.8 and 1.4.10. But since I updated to version 1.5.1, it stopped working. And the same happens with 1.5.3. I have coded the following minimum [non] working example:

#include <mosquitto.h>
#include <stdio.h>
#include <sys/epoll.h>

// Set the following parameters as needed
#define USERNAME	"my_username"
#define PASSWORD	"my_password"
#define HOST		"my_broker.com"
#define CLIENT_ID 	USERNAME
#define PORT		8883
#define KEEPALIVE	60

#define EVENTS 		(EPOLLIN | EPOLLOUT | EPOLLERR | EPOLLHUP | EPOLLRDHUP | EPOLLET)

static void connect_cb(struct mosquitto *mosq, void *priv, int rc)
{
	printf("%s: %i received (%s)\n", __func__, rc, mosquitto_connack_string(rc));
}

static void disconnect_cb(struct mosquitto *mosq, void *priv, int rc)
{
	printf("%s: %i received (%s)\n", __func__, rc, mosquitto_connack_string(rc));
}

static void log_cb(struct mosquitto *mosq, void *priv, int level, const char *msg)
{
	printf("%s: %i %s\n", __func__, level, msg);
}

static struct mosquitto* mosq_init(void)
{
	struct mosquitto *mosq = NULL;
	const char *client_id = CLIENT_ID;
	int err = 0;

	err = mosquitto_lib_init();
	if (!err) {
		mosq = mosquitto_new(client_id, true, NULL);
		err = mosq == NULL?-2:0;
	}
	if (!err) err = mosquitto_username_pw_set(mosq, USERNAME, PASSWORD);
	if (!err) err = mosquitto_tls_set(mosq, NULL, "/etc/ssl/certs", NULL, NULL, NULL);
	if (!err) {
		mosquitto_connect_callback_set(mosq, connect_cb);
		mosquitto_disconnect_callback_set(mosq, disconnect_cb);
		mosquitto_log_callback_set(mosq, log_cb);
	}
	printf("%s: code %d\n", __func__, err);

	if (err && mosq) {
		mosquitto_destroy(mosq);
		mosq = NULL;
	}

	return mosq;
}

int main(void)
{
	struct mosquitto *mosq = NULL;
	int mosq_socket = -1;
	int err = -1;
	int epfd = -1;
	struct epoll_event ev;

	// Init and connect
	mosq = mosq_init();
	if (mosq) err = mosquitto_connect_async(mosq, HOST, PORT, KEEPALIVE);

	if (!err) {
		epfd = epoll_create1(0);
		err = epfd < 0?-3:0;
	}
	if (!err) {
		mosq_socket = mosquitto_socket(mosq);
		ev.data.fd = mosq_socket;
		ev.events = EVENTS;
		err = epoll_ctl(epfd, EPOLL_CTL_ADD, mosq_socket, &ev);
	}

	// Loop
	while (!err) {
		err = epoll_wait(epfd, &ev, 1, 250);
		if (err < 0) err = -4;
		else if (err > 0) {
			// Event
			err = -5;
			if (ev.events & EPOLLIN) {
				printf("read triggered\n");
				mosquitto_loop_read(mosq, 1);
				err = 0;
			}
			if (ev.events & EPOLLOUT) {
				if (mosquitto_want_write(mosq)) {
					printf("write triggered\n");
					mosquitto_loop_write(mosq, 1);
				}
				err = 0;
			}
		}
		mosquitto_loop_misc(mosq);
	}

	printf("%s: ERROR %d\n", __func__, err);

	return err;
}

If I compile and run this code using mosquitto 1.4.10, I get the following output:

mosq_init: code 0
log_cb: 16 Client my_username sending CONNECT
write triggered
read triggered
write triggered
read triggered
write triggered
read triggered
write triggered
read triggered
log_cb: 16 Client my_username received CONNACK
connect_cb: 0 received (Connection Accepted.)

Everything works, and if I capture the packets, I can see the cert and key exchange prior to the CONNACK. But when I run the program using mosquitto 1.5.3, the "write_triggered" message is printed once and the connection never completes. If I monitor the network using tcpdump, I can see that the TCP handshake is completed, but the TLS negotiation never starts:

TCP      76     35026 → 8883 [SYN, ECN, CWR] Seq=0 Win=26280 Len=0 MSS=1314 SACK_PERM=1 TSval=1714539719 TSecr=0 WS=128
TCP      76     8883 → 35026 [SYN, ACK, ECN] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=44011310 TSecr=1714539719 WS=256
TCP      68     35026 → 8883 [ACK] Seq=1 Ack=1 Win=26368 Len=0 TSval=1714539778 TSecr=44011310

And that's all. If I remove the EPOLLET from EVENTS, the "write triggered" message is continuously printed, but the connection is never completed, the cert/key exchange never happens. So it looks like mosquitto_loop_write() is not working as expected.

Am I doing anything wrong, or is there a regression?

[doragasu]

Is my approach correct? There are some things that I am not sure about:

1. Should I use mosquitto_connect() or mosquitto_connect_async()?
2. Should I call mosquitto_loop_start()? With version 1.4.10 it worked without calling it...

I have been experimenting doing variations of the program above, and it looks like it works if I call mosquitto_loop_start() independently of using connect or connect_async, but I would like to know which one is the correct approach, and why it worked with version 1.4.10 and it does not work now.

[ralight]

I've found the commit that introduces the problem, it's not you at fault :) I still need to diagnose the underlying problem, that is a task for next week.

For reference - if you're using mosquitto_loop_start(), you should not use any of the other loop functions, the library is doing the work for you.

[doragasu]

Thanks for the feedback. I was currently working around the bug by calling mosquitto_loop() during the connection stage, and then mosquitto_loop_{read,write,misc}() once the connection is completed. It's ugly but works until this gets fixed.

[ralight]

I think this is fixed on the fixes branch, could you give it a try?

[doragasu]

Thanks, I will give it a try as soon as I have a slot.

[doragasu]

Finally I got time to test this. I have tried with version 1.5.4 (that I can see the fix has been merged into). The example above works if I connect using mosquitto_connect(). But if I leave the example as is (using mosquitto_connect_async()), it still does not work. Maybe this is the correct behavior, and I should use moquitto_connect().

Thank you very much for the fix!