Connection WebChannel transport errored when querying more than 9998 rows... #5197
Comments
|
Have also tested this with |
|
@yuchenshi - Saw you worked on a similar issue #4336 ...have any insights here? |
|
@tnoetzel This is a more complicated issue in the Firestore emulator and we don't have any updates yet. |
|
@yuchenshi - Any suggested workarounds in the meantime? |
|
+1 |
1 similar comment
|
+1 |
|
+1. After reaching +10k rows on one of the collections, the emulator stopped working properly 😭 |
|
Bump since I am getting this error too. Any status on this? I believe it's pretty blocking issue. |
|
Repro testing: I'm able to insert 125,000 rows via the batch interface without issue. After 10k rows though the Emualtor Suite UI seems to have problems. I'm using the modular API - does switching to modular help when the UI is not open? import/expert of the 125k rows appears to work fine also. I'm not able to reproduce the querysnapshot issue without the UI. I'll continue investigating the UI issue - can anyone confirm this is reproducible without the UI (as in, no browser is open with the Emulator Suite UI, but it is still running). |
|
I encountered a similar issue (hundreds of repetitions of this error series over and over in the debug log): I encountered this error while doing bulk setting of documents into a Firebase Emulator instance. If no browser windows are open to look at the Firestore UI, doing the following (chunks of parallel writes) operates blazingly-fast: But after that process concludes, if I open the Emulator UI window to look at the new data, the web UI just gets a "loading" indicator, and the debug log floods with those BackChannel errors. If I run the data importing script while the Emulator UI is open, it goes relatively quickly through the first few batches, but once it's past about 1,500 records, it slows down, and the debug log floods with those BackChannel errors. The inserts do continue to go in, but go slower and slower with each batch. So this seems to be an issue with the Emulator suite trying to communicate over websockets; when it knows there's a client waiting for all messages it's unable to serve them up fast-enough to keep the backlog clear? |
|
I am no longer able to continue development on my project because I need live data to develop the next feature. At least, please provide us with a work-around, it seems to be an issue with gRPC. My data is just about 20 MB, yeah! that small! But one collection (the one that crashes) has about 16k entries in it. I have repeatedly been disappointed with the Emulators Suite lately. It usually takes ages before any viable fix, no work-around, nothing. I'm seriously starting to consider other alternatives. |
I know the pain, we ended up going thru a months long project to migrate to postgres. |
|
Thank you for the feedback. Some possible workarounds here:
I'll look at putting in a short term fix soon, but essentially the UI needs to handle data windowing better. |
|
@christhompsongoogle I'm glad to hear someone is going to do something about it, at last. But I'm sad for all the people that have been here since November 1st, 2022 because of this. I'm equally sad for other persons who have invested in Firebase, not knowing it wasn't their program that had bug, but it's the Emulators Suite, I do feel their pain on this one. To be honest, I'm looking forward for a real fix, not a short term fix. I hear that Google Developers are pretty thorough when it comes to requirements for correct, efficient and fast algorithms. Anyway, I feel like there are too many tradeoffs that have been made on the Emulators Suite, not just on this one problem, but on different bugs as well. Please give us some real fix, please 🥹 |
This is not constructive. If you want to rant take it elsewhere please. 10k documents in a single collection is outside what I would consider "typical" use for an emulator and the UI - this is not a production scaled product and so there will necessarily be scalability limits. The project is [mostly] open source - if anyone would like to improve anything I'd be happy to collaborate on or review any PRs. |
|
@christhompsongoogle - While I agree about the ranting, I strongly disagree about it being an atypical use for an emulator. Many companies use emulators to test locally with production copies of data. It's an extremely common use case. FWIW, best I can tell this has nothing to do with the emulator UI per se... even if you don't open the emulator UI the issue still occurs (though I suppose it could be because the emulator UI is still technically running... is there a flag to prevent the emulator UI from running to test whether that's really the issue?) ... As a sidebar, while I again agree that ranting isn't constructive, I can understand the frustration. Given that a) Firebase is a Google product, b) vague references to "internal bugs" are common in many of these issues, and c) @yuchenshi said this is "a more complicated issue in the Firestore emulator", suggesting somebody contribute to the repo because it's "mostly open source" feels less like a genuine suggestion and more like slapping the customer. As I'm sure. you know, Google is rapidly developing a reputation for abandoning its products. It'd be a shame to have that happen with Firebase. |
|
The only reproducible case I'm able to find is having over 10k documents while the UI is open in the browser. There's another possibly-related bug where having the UI open in the browser slows down batch inserts significantly. For clarity, I'm not able to reproduce the bug by inserting ~15k documents and then listening via onSnapshot to the collection. This leads me to the conclusion that this is just the UI fetching all documents in a single page and listening on each of them. Improving the behavior between onSnapshot and the emulator might be necessary but I don't have any working example to test it or even demonstrate that it's a problem. To test without the UI, try the following:
|
I think slight correction to this: I agree it seems to be the presence of a front-end visitor is triggering the issue, but the issue is happening on the back-end. Since the front-end is listening on a websocket connection, I think the issue arises when the front-end connects and sends some "I'm here listening; let me know of any updates", and then the back-end tries to bundle up 10,000+ messages and send them all at once to the websocket connection. The error from the back-end seems the buffer/queue on the back-end is getting over-filled; that it cannot clear the queue fast enough. Not sure if the back-end when it hears the "someone is listening on the front-end" message checks the state of that collection and then sends notifications for all the records at once (as opposed to paginating/chunking the notifications to the websocket handler), and the socket handler can't handle ingesting messages in chunks larger than 10,000 at once? Or is it that the front-end as part of the initial handshake over the socket is "I'm interested in collection FOO; send me all the records in that collection" (and it should be paginating instead)? |
As a developer, my key desire is to have a well-documented way to test on non-production environments. Whether that's a clear way to have a real Firebase separate project for each developer of a project to do local development off of, without the risk of affecting the production Firebase project's data, or having locally-running emulated Firebase features. If the Emulator tools are only going to be targeting 10,000 or less records, I think that's fine (useful for "toy" projects or proof of concepts), but then there needs to be a separate means to do production development/testing safely too. The "hosting" feature has the ability to create "Preview channels" which serve this need. If the "Firestore" feature had a similar option, I think that would be fine too. But until that's the case, having clear direction on whether Google's going to consider the Emulated tools as only targeting small/toy implementations, or meant to be robust pre-prod testing tools would be good. |
|
The emulators are a tool, but right now they are not a complete replacement for a preprod environment. There are some key differences in behavior that should still be tested with a live project like indexes, transactions, and load testing: |
Is there a guide somewhere that lays out how to (safely, cheaply) use a "preprod environment" within Firebase? I presume it's just "create a new Firebase project" (probably with a second billing plan), but then the code repository needs to have two different projects in its Firebase config file, and it's just one accidental |
|
Terraform is a tool that we've recently added support for: |
"Recently added support for" and "is a robust tool we recommend for production use" are two very different things. For clarity, for developers making infrastructure choices for their projects, is Terraform the suggested/default/most-supported tool that the Firebase team recommends using for this functionality, that will be fully-supported going forward? |
|
Terraform is an industry standard that Google has long supported and Firebase is prioritizing support for Terraform going forward. Not all features are supported in Firebase yet so be sure to check that it will meet your needs before starting. |
|
So, summarizing the current state of things that I see (please correct me if I'm wrong):
So, from that I gather that "soon" could be a very long time (for the maintaining team to prioritize this), and if the community wants to prioritize it, they have to develop the change themselves. I have not done development on the core Firebase Tools core repo, but will look into that (I've done library development using |
|
FWIW, I still think there's a completely valid use case for supporting more than 10K rows locally... developing with production data is useful in many contexts and developers are substantially more likely to catch issues if they're using that data locally rather than just in a preproduction environment where they test before a final deployment. Given that Firebase is geared so heavily towards startups I wouldn't expect most Firebase users to have the types of sophisticated development / testing processes necessary to catch a high percentage of bugs on staging... It's also substantially faster if you're dealing with fixing an issue that only occurs with large amounts of data (but doesn't have to do with Firebase directly) to not have to try to fix it and then deploy to staging to see if your changes actually worked. This was literally my case where I was trying to debug an issue related to client-side virtualization of large lists, but was prevented because the emulator couldn't handle it. It also sounds like a "temporary" fix is at least within reach, so @MidnightLightning maybe let's not write off the improvement just yet. @christhompsongoogle - if the "short-term fix" is obvious to you but you don't have time to implement, might be useful to at least point @MidnightLightning in the right direction... |
|
@christhompsongoogle Thanks for getting firebase/firebase-tools-ui#935 in! I see that change got merged into that project's main branch, but the |
|
The latest UI version was picked up by I created a release in the firebase-tools-ui repo so it's clear what version we're on - thanks for pointing it out. |
|
I've now updated my local installation to use With the new version, for data that's at rest (a collection that already has 10,000+ items), visiting the Emulator page for Firestore loads properly now (rather than showing a loading spinner on the front-end and the "back channel error" messages on the back-end), and for both unfiltered and filtered views of the collection, it maxes out at 10,000 items listed. For data that's in motion, there still seems to be an issue though. If there is a script that's doing a large series of inserts updates (using the Admin SDK to seed an initial data set or sync data from another source), if during the mass inserts a browser tab is open to the Emulator Firestore page, the back-end starts throwing the same "too many pending messages in the back channel" errors about sending more than 10,000 messages, and the web browser tab doesn't show any obvious loading indicators, but I did experience it crashing due to an "out of memory" exception if I let it run long enough. So it seems this fix corrects the front-end calling the back-end, but when the back-end is calling the front-end through the websocket connection, this issue can still arise? |
|
Is anyone else listening in on this ticket able to confirm or not if this fully fixes the problem in your development environment, or you still get the "too many pending messagings in the back channel (10001)" errors in some circumstances? |
|
Checking in @christhompsongoogle; are you able to verify on your end whether this change fixed both querying at-rest collections, and sending update messages to the front-end? |
|
There are still some ways to trigger this, notably with very large firestore imports. My fix was to resolve the issue with the UI becoming unusable after 10k rows in a single collection, but a future follow-up change will still be needed to resolve the issue completely. |
|
@christhompsongoogle @MidnightLightning I get hundreds of these messages: A bit later I get: And if I use transactions I get this: From my point of view, this has nothing to do with the UI - as I didn't have it open and it happens to me quite soon after starting the emulator. I have a relatively large database (around 1GB) and maybe it is special that I have subscribed to around 100 collection streams from several test-devices in my Flutter project. I run the emulator separately on a server like workstation - I assigned the Java Process 24GB RAM - which the process uses right after start. Thank you for renewed interest in this issue, I am sorry that I can't provide a reproducible example - but I hope my trail of exceptions leads you in the right direction. To add to the topic about what the emulator should or should not support: I can imagine it is probably hard to port a very scalable cloud database into a "not so scalable but still usable"-local database without actually releasing the sourcecode of Firestore itself - but I still I feel the local development copy is a very important part of the development cycle which I always found a rather frustrating expericene with Firestore. But I really enjoy the rest of the product, so yeah, thank you anyway :-) |
[REQUIRED] Environment info
firebase-tools: 11.15.0
Platform: macOS 13.0
[REQUIRED] Test case
Since upgrading to Ventura, using firebase web
@firebase/firestore: Firestore (8.10.1), I'm getting errors when trying to listen to ~15K rows of data. The backup I'm using is only 18.3MB of data (I have 64GB of ram), so I don't think this is a memory issue.With some testing, this seems to start happening when trying to pull any more than 9998 rows.
In the console, I'm seeing
@firebase/firestore: Firestore (8.10.1): Connection WebChannel transport erroredand numerous 400 request errors.In the logs (
firestore-debug.log) shows:[REQUIRED] Steps to reproduce
These both result in the console error :
Adding
.limit(9998)seems to prevent these errors.[REQUIRED] Expected behavior
Data should query without error.
[REQUIRED] Actual behavior
... etc...
Or, from the
--debugflag:The text was updated successfully, but these errors were encountered: