Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve handling of long messages #98

Open
Wobbley opened this issue Apr 12, 2019 · 19 comments
Open

Improve handling of long messages #98

Wobbley opened this issue Apr 12, 2019 · 19 comments

Comments

@Wobbley
Copy link

Wobbley commented Apr 12, 2019 •
edited
Loading

Hey

Long messages seem to get shortened. Example: https://gist.github.com/Wobbley/b71a19fe9e433247e5352a2c438d18c9

As you can see it simply cuts off the message. I checked and it displays fine in MQTTFX. Am I just looking in the wrong window? I used the copy value function from the window on the right.

OS: Ubuntu 18.04 LTS
Version: 0.2.3
@thomasnordquist
Copy link
Owner

Currently, messages are truncated to 10kb.

  • MQTT Explorer keeps a record of all topics/messages, I added the limit to prevent the app from running out of memory.
  • Value preview would become unbearably slow if it would try to display a 5MB binary/text.

Silently truncating the messages is not good enough anymore, could be solved with a configuration option or at least a notification when a message has been truncated.
What is your expected message size?

@thomasnordquist
Copy link
Owner

thomasnordquist commented Apr 12, 2019 •
edited
Loading

If you need an immediate solution have a look at:

MQTT-Explorer/backend/src/index.ts

Lines 47 to 53 in 50b2362

connection.onMessage((topic: string, payload: Buffer, packet: any) => {
let buffer = payload
if (buffer.length > 10000) {
buffer = buffer.slice(0, 10000)
}
backendEvents.emit(messageEvent, { topic, payload: Base64Message.fromBuffer(buffer), qos: packet.qos, retain: packet.retain })

Change it and execute yarn build and yarn start

@thomasnordquist thomasnordquist changed the title Long messages get shortened Improve handling of long messages Apr 15, 2019
@Wobbley
Copy link
Author

Wobbley commented Apr 15, 2019 •
edited
Loading

I assumed there was some performance related reasons for truncating the message. A note of some kind that the message has been truncated would be nice.

The message is big, and I understand the need for truncating it. More specifically it's two MP3 files (about 5 seconds long) and some more metadata. Is it possible to truncate it "per field" somehow? I don't mind if the huge base64 message is truncated, problem is that I don't get to see the rest of the content, hence confirm that the structure is correct.

MQTT Explorer works great for all of the other queues, so this might just be an edge case not worth pursuing.

@thomasnordquist
Copy link
Owner

The very first thing to do is notify the user of truncated content and add a (volatile?) option to increase the limit (10k / 100k / 1M / 5M).

Beyond ~1M / ~5M the preview and plot should truncate the content for performance reasons.

The Chrome Debugger handles long strings like this:
Bildschirmfoto 2019-04-15 um 13 52 45

Long fields in JSON like this:
Bildschirmfoto 2019-04-15 um 13 58 31

Redux Dev Tools trunk a little differently (infix truncation, keeping "...", {...}, [...] to indicate content type)
Bildschirmfoto 2019-04-15 um 14 01 05

Destructuring the JSON and truncating large fields would be the coolest but I'd need to rewrite the JSON preview with an AST parser. I might have to do it either way for Plotting JSON values.

@Wobbley
Copy link
Author

Wobbley commented Apr 15, 2019

Notifying the user is great. An option to increase it is even better. Insomnia gives you the option to ignore the buffer, but gives you a clear cut warning that it will reduce performance, maybe even killing the application. It would be great not having to change application just to view those specific messages, but rather give it a go, and bear the consequences.

I would say that the Redux Dev Tools version is preferable, but I understand if that's a major re-write.

@thomasnordquist
Copy link
Owner

Except for the notification maybe, none of this will make it in the next release.
I'm basically polishing up the current release, removing all the "paper-cuts" that have accumulated over time.

Preview: #87

@squawk979
Copy link

Was caught out by this, initially thought I had a bug. Agree the minimum is to warn user the message has been truncated. Be great if we could increase the limit via settings as I do have to analyse larger messages from time to time which means switching tools :(

@thomasnordquist
Copy link
Owner

The limit needs to be better, but I'm not yet sure how.

Some ideas

  1. Static setting in the sidebar
    1.1. Should this setting be permanent? (remembered through restarts)
    1.2. Remember settings except "unlimited"
  2. Big warning in a cut off topic with the ability to increase the limit for this specific topic.
  3. Increase limit for the last x topics a user has selected (coupling between view state and the tree-model is pretty high)
  4. dynamic, based on current Ram usage (something like: <256MB unlimited, >256MB 100kb limit, > 512MB 10kblimit). This would be the most complicated approach, but most users would never see that there is a limit.

@thomasnordquist thomasnordquist mentioned this issue Oct 24, 2019
Closed
@Wobbley Wobbley closed this as completed Sep 16, 2020
@Wobbley Wobbley reopened this Sep 16, 2020
@Wobbley
Copy link
Author

Wobbley commented Sep 16, 2020

I was looking through my stale issue, and closing some of them. Should I just close this one as well? Still using your tool by the way, and got some co-workers to use it as well. Keep up the good work!

@sooslaca
Copy link

Hi @Wobbley ,

I'm interested in solving this one too, right now I see it's truncated at 20k.

Thank you

@luebbe luebbe mentioned this issue Aug 26, 2021
Closed
@saper-2
Copy link

saper-2 commented May 28, 2022 •
edited
Loading

I just stumbled upon this recently when I was digging around zigbee2mqtt... (they put in one topic list of zigbee devices in network in JSON array of objects that have a lot of data - I have only 8 zigbee devices, and the topic with list have already ~21230 characters, I saw ppl having 20 and even more devices - how long they have this topic 😅 ).

Upvote for configurable value limits.
@thomasnordquist - maybe a table or list in settings: where we can enter that exact topics of interest with their value max-length?

@RomeoBucur
Copy link

Hi Thomas, I.ve tried to extend mqtt explorer(installed under Windows 10 Enterprise) limit to 50KB , and I'm still trying to find in Application/Dev Tools in *.js files (under Page) but I didn't find yet MQTT-Explorer/backend/src/index.ts. Installed version is MQTT-Explorer 0.4.0-beta1. Can I kindly ask you to give more details to limit can be extended? Thanks a lot in advance.

@GO74
Copy link

GO74 commented Nov 8, 2022

I could need a solution to this as well. Not sure how to apply the 'solution' as well.

@danielgoepp
Copy link

+1 for this! It doesn't look like this has been addressed as of early 2023 here. I'm working with payloads in the 50K to 100K range (zigbee network maps) and this is a real bummer of a limitation.

@ispysoftware
Copy link

+1 for this. At least add a warning that it's been truncated.

@RomeoBucur
Copy link

RomeoBucur commented Mar 14, 2023 via email

@saper-2
Copy link

saper-2 commented Mar 14, 2023

+1 for this! It doesn't look like this has been addressed as of early 2023 here. I'm working with payloads in the 50K to 100K range (zigbee network maps) and this is a real bummer of a limitation.

Yes, I do was working with Zigbee2MQTT too and for just ~10 devices I get payload about 226k (when z2m start it publish devices info+ config, and do it from time to time too).

At least information that payload is truncated would be sufficient.

@Snaaio Snaaio mentioned this issue Mar 21, 2023
Open
@abstract-entity
Copy link

+1

@lordmundi
Copy link

+1

also ran into this today :( i think it would be great if we had some way to increase the max message size. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests