-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[receiver/filelog] Limit files consumed per polling cycle #18477
Conversation
Foresight Summary
View More Details⭕ build-and-test-windows workflow has finished in 5 seconds (40 minutes 44 seconds less than
|
Job | Failed Steps | Tests | |
---|---|---|---|
windows-unittest-matrix | - 🔗 | N/A | See Details |
windows-unittest | - 🔗 | N/A | See Details |
✅ telemetrygen workflow has finished in 55 seconds (2 minutes 2 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
build-dev | - 🔗 | N/A | See Details |
publish-latest | - 🔗 | N/A | See Details |
publish-stable | - 🔗 | N/A | See Details |
✅ check-links workflow has finished in 1 minute 23 seconds (1 minute 10 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
changed files | - 🔗 | N/A | See Details |
check-links | - 🔗 | N/A | See Details |
✅ changelog workflow has finished in 2 minutes 56 seconds (48 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
changelog | - 🔗 | N/A | See Details |
✅ prometheus-compliance-tests workflow has finished in 3 minutes 28 seconds (5 minutes 15 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
prometheus-compliance-tests | - 🔗 | N/A | See Details |
✅ build-and-test workflow has finished in 42 minutes 16 seconds (16 minutes 4 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
unittest-matrix (1.19, internal) | - 🔗 | ✅ 561 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, internal) | - 🔗 | ✅ 561 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, processor) | - 🔗 | ✅ 1472 ❌ 0 ⏭ 0 🔗 | See Details |
correctness-metrics | - 🔗 | ✅ 2 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, processor) | - 🔗 | ✅ 1472 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, extension) | - 🔗 | ✅ 546 ❌ 0 ⏭ 0 🔗 | See Details |
correctness-traces | - 🔗 | ✅ 17 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, extension) | - 🔗 | ✅ 546 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, receiver-0) | - 🔗 | ✅ 2574 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, receiver-0) | - 🔗 | ✅ 2574 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, exporter) | - 🔗 | ✅ 2452 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, receiver-1) | - 🔗 | ✅ 1958 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, receiver-1) | - 🔗 | ✅ 1958 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, exporter) | - 🔗 | ✅ 2452 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.19, other) | - 🔗 | ✅ 4676 ❌ 0 ⏭ 0 🔗 | See Details |
unittest-matrix (1.18, other) | - 🔗 | ✅ 4676 ❌ 0 ⏭ 0 🔗 | See Details |
integration-tests | - 🔗 | ✅ 55 ❌ 0 ⏭ 0 🔗 | See Details |
setup-environment | - 🔗 | N/A | See Details |
check-collector-module-version | - 🔗 | N/A | See Details |
check-codeowners | - 🔗 | N/A | See Details |
checks | - 🔗 | N/A | See Details |
lint-matrix (receiver-0) | - 🔗 | N/A | See Details |
lint-matrix (receiver-1) | - 🔗 | N/A | See Details |
lint-matrix (processor) | - 🔗 | N/A | See Details |
lint-matrix (exporter) | - 🔗 | N/A | See Details |
lint-matrix (extension) | - 🔗 | N/A | See Details |
lint-matrix (internal) | - 🔗 | N/A | See Details |
lint-matrix (other) | - 🔗 | N/A | See Details |
build-examples | - 🔗 | N/A | See Details |
unittest (1.19) | - 🔗 | N/A | See Details |
unittest (1.18) | - 🔗 | N/A | See Details |
lint | - 🔗 | N/A | See Details |
cross-compile (darwin, amd64) | - 🔗 | N/A | See Details |
cross-compile (darwin, arm64) | - 🔗 | N/A | See Details |
cross-compile (linux, 386) | - 🔗 | N/A | See Details |
cross-compile (linux, amd64) | - 🔗 | N/A | See Details |
cross-compile (linux, arm) | - 🔗 | N/A | See Details |
cross-compile (linux, arm64) | - 🔗 | N/A | See Details |
cross-compile (linux, ppc64le) | - 🔗 | N/A | See Details |
cross-compile (windows, 386) | - 🔗 | N/A | See Details |
cross-compile (windows, amd64) | - 🔗 | N/A | See Details |
build-package (deb) | - 🔗 | N/A | See Details |
build-package (rpm) | - 🔗 | N/A | See Details |
windows-msi | - 🔗 | N/A | See Details |
publish-stable | - 🔗 | N/A | See Details |
publish-dev | - 🔗 | N/A | See Details |
publish-check | - 🔗 | N/A | See Details |
✅ load-tests workflow has finished in 8 minutes 3 seconds (8 minutes 39 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
loadtest (TestTraceAttributesProcessor) | - 🔗 | ✅ 3 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestIdleMode) | - 🔗 | ✅ 1 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestMetric10kDPS|TestMetricsFromFile) | - 🔗 | ✅ 6 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) | - 🔗 | ✅ 8 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) | - 🔗 | ✅ 10 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) | - 🔗 | ✅ 12 ❌ 0 ⏭ 0 🔗 | See Details |
loadtest (TestBallastMemory|TestLog10kDPS) | - 🔗 | ✅ 19 ❌ 0 ⏭ 0 🔗 | See Details |
setup-environment | - 🔗 | N/A | See Details |
✅ e2e-tests workflow has finished in 11 minutes 50 seconds (8 minutes 6 seconds less than main
branch avg.) and finished at 9th Feb, 2023.
Job | Failed Steps | Tests | |
---|---|---|---|
kubernetes-test | - 🔗 | N/A | See Details |
*You can configure Foresight comments in your organization settings page.
pkg/stanza/fileconsumer/config.go
Outdated
// Poll file limit can be 0 to signal unlimited or it must be greater than or equal to | ||
// max concurrent files to ensure we can read at least the configured max concurrent files in a polling cycle. | ||
if c.PollFileLimit < 0 || (c.PollFileLimit < c.MaxConcurrentFiles && c.PollFileLimit != 0) { | ||
return fmt.Errorf("`poll_file_limit` must be 0 or greater than or equal to `max_concurrent_files`") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we separate this into two checks/errors? The user will only violate one or the other at a time, so having a more specific message for each case would be clearer.
Signed-off-by: Corbin Phelps <[email protected]>
Signed-off-by: Corbin Phelps <[email protected]>
Signed-off-by: Corbin Phelps <[email protected]>
Signed-off-by: Corbin Phelps <[email protected]>
Signed-off-by: Corbin Phelps <[email protected]>
…the intent Signed-off-by: Corbin Phelps <[email protected]>
f8a6813
to
3f36608
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple nits, but looks good otherwise.
Signed-off-by: Corbin Phelps <[email protected]>
Description: Added new configuration
poll_file_limit
parameter to the filelog receiver.Validation Logic:
poll_file_limit
cannot be negativepoll_file_limit
can be 0 to indicate unlimitedpoll_file_limit
must be greater than or equal tomax_concurrent_files
to allow at least one full batchIn cases that there are more files discovered than the batch size this parameter allows limiting how many can be processed per polling cycle.
Link to tracking Issue: #18476
Testing: Added unit tests for ingestion rate and configuration tests.