Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http_dir slow TTFB #1936

Closed
sectokia opened this issue Dec 9, 2020 · 9 comments
Closed

http_dir slow TTFB #1936

sectokia opened this issue Dec 9, 2020 · 9 comments
Milestone
2.0.1

Comments

@sectokia
Copy link

sectokia commented Dec 9, 2020 •
edited

The time to first byte from when a GET is set to http_dir for a website element, to when it responds is almost always 100mS, or in steps of 100mS. If a page has for instance 20 references to images that don't exist, mosquitto takes over 2 seconds to answer them all.

I am not smart enough to figure out why. But it seems to me the call to wsl_service without a timeout of -1 eventually results in a poll() call in wsl_service with a -1 timeout that blocks, and is only waken because of a deeper 100mS timer in the lws netlink code. I think the value is meant to be 0?
@ralight
Copy link
Contributor

ralight commented Dec 9, 2020

What version of Mosquitto, and what version of libwebsockets are you using?

@sectokia
Copy link
Author

sectokia commented Dec 9, 2020 •
edited

I have tried:
mosquitto-1.6.9-install-windows-x64.exe
mosquitto-2.0.0-install-windows-x64.exe
mosquitto version 1.6.12 via ppa:mosquitto-dev/mosquitto-ppa for ubuntu for windows through windows subsystem for linux.

All do the same thing: They have FFTB of 100mS, 200mS or 300mS for every GET request.

Example conf:

listener 8083
protocol websockets
allow_anonymous true
http_dir www

Example is to serve a HTML file with the content below, and observe how long it takes browser to get response to GET for all the non existing images:

<!DOCTYPE HTML>
<HTML>
<BODY>
<IMG SRC="01.PNG"/>
<IMG SRC="02.PNG"/>
<IMG SRC="03.PNG"/>
<IMG SRC="04.PNG"/>
<IMG SRC="05.PNG"/>
<IMG SRC="06.PNG"/>
<IMG SRC="07.PNG"/>
<IMG SRC="08.PNG"/>
<IMG SRC="09.PNG"/>
<IMG SRC="10.PNG"/>
<IMG SRC="11.PNG"/>
<IMG SRC="12.PNG"/>
<IMG SRC="13.PNG"/>
<IMG SRC="14.PNG"/>
<IMG SRC="15.PNG"/>
<IMG SRC="16.PNG"/>
<IMG SRC="17.PNG"/>
<IMG SRC="18.PNG"/>
<IMG SRC="19.PNG"/>
<IMG SRC="20.PNG"/>
</BODY>
</HTML>

@ralight
Copy link
Contributor

ralight commented Dec 9, 2020

I'm afraid there is nothing I can do about this problem right now. libwebsockets does not allow applications access to its listening sockets, which makes it impossible to integrate with existing applications without this delay.

The problem is that Mosquitto has to wait for any of its listening sockets, or any of the client sockets to be active. It does this with a 100ms timeout by default, so if nothing is active then the loop will run anyway. Ideally this timeout would actually be much higher (we can calculate when the next event will arrive that we need to wake up for). The only sockets that we cannot wait for at the same time are the libwebsockets listening sockets, that accept new connections. So in our wait, we may have to be paused for 100ms if nothing else has happened with the sockets we are watching, at which point the loop continues and we process any libwebsockets connections that have arrived in that time period.

We could drop our timeout value, which would give better responsivity for incoming websockets connections, but it would mean excessive and unnecessary CPU usage.

I am looking into removing libwebsockets and replacing it with an alternative, with the aim of getting around this problem and hopefully improving general websockets performance along the way.

@sectokia
Copy link
Author

sectokia commented Dec 10, 2020 •
edited

I think it could be done with LWS:

Setup a LWS_CALLBACK_ADD_POLL_FD and LWS_CALLBACK_DEL_POLL_FD callbacks as part of lws_create_context, then LWS would callback each time it needs to add/remove a FD to poll on.

Those FD's could be inserted/removed from the pollfds used in mux_handle main loop.

If poll shows a LWS FD is ready, service it with lws_service_fd()

@ralight
Copy link
Contributor

ralight commented Dec 10, 2020

Heh, lws said it wasn't possible to get access to the listener fds when I asked. I shouldn't have believed it!

@ralight ralight modified the milestones: 2.1, 2.0.1 Dec 10, 2020
@sectokia
Copy link
Author

sectokia commented Dec 10, 2020 •
edited

I couldn't get this to work.
I think issue is that LWS_WITH_EXTERNAL_POLL is not enabled so the LWS_CALLBACK_ADD_POLL_FD is never issued by libwebsocket

Does LWS_WITH_EXTERNAL_POLL have to be set in make file or something?

@ralight
Copy link
Contributor

ralight commented Dec 11, 2020

Yes, unfortunately it is turned off by default in more recent versions of libwebsockets and is going to be removed. Turning that option on when building libwebsockets should do the job. I would also suggest looking at version 2.4.2, it has proved to be a stable and reliable version, whereas 3.x introduced problems and I know of people who struggle with 4.x.

I've uploaded Windows versions for 2.0.2 to the download page, perhaps you could give those a try?

@sectokia
Copy link
Author

Thanks a lot!

@sectokia
Copy link
Author

I'll did around some more another day but I think 2.0.2 win32 download doesn't have the fix working.

fAuernigg pushed a commit to fAuernigg/mosquitto that referenced this issue Jan 4, 2021
@ralight @fAuernigg
Fix websockets listeners not causing the main loop not to wake up.
df1ace3 
Closes eclipse#1936. Thanks to sectokia.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.0.1
Development

No branches or pull requests
2 participants
@ralight @sectokia