-
Notifications
You must be signed in to change notification settings - Fork 236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poor performance of EMF Exporter causes metric lost in the long run #388
Comments
The performance is improved from 30s -> 200ms after applying changes from open-telemetry/opentelemetry-collector-contrib#2572
|
Close this issue because the fix is merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened
Metrics exported by EMF exporter were lost by 1-3 minutes during the long run, causing gaps in the dashboard. Followed by a false burst, it automatically recovered from the stuck.
Usually metrics were reported at a constant rate, but when the collector "stucked", it was unsteady.
After adding more debugging logs, we found it took 30 seconds to digest one chunk of metrics by average, e.g. metrics from one Pod in terms of using prometheus receiver.
It took 3 minutes to finish the same process at the time of metrics lost.
Root cause
Although metrics are grouped before the pushing happens, every logEventBatch is sent one by one, causing high network latencies, which drags down the performance.
How to solve it
Pushing EMF logs in batch.
The text was updated successfully, but these errors were encountered: