-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Race condition in testbed load generator when parallel is used #32326
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
codeboten
pushed a commit
that referenced
this issue
May 1, 2024
Fixes a race condition in testbed's default load generator (`ProviderSender`). The fix changes the behavior of how logging works for the `ProviderSender`. Before this PR, the code will try to log errors if the previous error is not the same and it would try to do this across goroutines. Now, each error will be logged if the previous error is not the same for each goroutine. Alternatively, we can also build a logger using bloom filter to try to log each error once though I am not sure if that would be required. This PR offers a quick fix while keeping the behavior reasonably close to the current behavior. Closes #32326 **Link to tracking Issue:** #32326 **Testing:** Follow the steps in the tracking issue.
rimitchell
pushed a commit
to rimitchell/opentelemetry-collector-contrib
that referenced
this issue
May 8, 2024
Fixes a race condition in testbed's default load generator (`ProviderSender`). The fix changes the behavior of how logging works for the `ProviderSender`. Before this PR, the code will try to log errors if the previous error is not the same and it would try to do this across goroutines. Now, each error will be logged if the previous error is not the same for each goroutine. Alternatively, we can also build a logger using bloom filter to try to log each error once though I am not sure if that would be required. This PR offers a quick fix while keeping the behavior reasonably close to the current behavior. Closes open-telemetry#32326 **Link to tracking Issue:** open-telemetry#32326 **Testing:** Follow the steps in the tracking issue.
cparkins
pushed a commit
to AmadeusITGroup/opentelemetry-collector-contrib
that referenced
this issue
Jul 11, 2024
Fixes a race condition in testbed's default load generator (`ProviderSender`). The fix changes the behavior of how logging works for the `ProviderSender`. Before this PR, the code will try to log errors if the previous error is not the same and it would try to do this across goroutines. Now, each error will be logged if the previous error is not the same for each goroutine. Alternatively, we can also build a logger using bloom filter to try to log each error once though I am not sure if that would be required. This PR offers a quick fix while keeping the behavior reasonably close to the current behavior. Closes open-telemetry#32326 **Link to tracking Issue:** open-telemetry#32326 **Testing:** Follow the steps in the tracking issue.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
testbed
What happened?
Description
If the
ProviderSender#Start
is called to run with more than one goroutine then a race condition occurs due to the shared variableProviderSender#prevError
.Steps to Reproduce
ProviderSender#Start
with load options havingLoadOptions#Parallel
value greater than1
. Example:make test
orgo test -v -race ./...
Expected Result
No race condition and the tests pass.
Actual Result
Race conditions found by the race-detector.
Collector version
v0.98.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
No response
Log output
Additional context
Possible solution: Make
prevError
local to thegenerate{Log,Metrics,Trace}
and remove it fromProviderSender
-- not sure if I am missing something here but that should be enough. If we require theprevError
to be a field in theProviderSender
then we can useatomic.Value
.The text was updated successfully, but these errors were encountered: