Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachprod: fails with concurrent update on map error #124400

Closed
nameisbhaskar opened this issue May 19, 2024 — with Slack · 1 comment · Fixed by #124401
Closed

roachprod: fails with concurrent update on map error #124400

nameisbhaskar opened this issue May 19, 2024 — with Slack · 1 comment · Fixed by #124401
Assignees
@nameisbhaskar
Labels
X-blathers-untriaged blathers was unable to find an owner

Comments

@nameisbhaskar Slack
Copy link
Contributor

nameisbhaskar commented May 19, 2024 •
edited

nightly failed with error:
05:51:29 === RUN acceptance/gossip/restart [metrics: https://go.crdb.dev/roachtest-grafana/teamcity-15320942/acceptance-gossip-restart/1716097889536/1716099689536
05:51:30 fatal error: concurrent map writes
05:51:30
05:51:30 goroutine 2812 [running]:
05:51:30 github.com/cockroachdb/cockroach/pkg/roachprod.updatePrometheusTargets.func1(_, {{0xc002c314a0, 0x2b}, {0x44aa200, 0xedddb899c, 0xc002ffc4d0}, {0x0, 0x0, 0x0}, 0x274a48a78000, ...})
05:51:30 github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:811 +0x2c8
05:51:30 created by github.com/cockroachdb/cockroach/pkg/roachprod.updatePrometheusTargets in goroutine 2445
05:51:30 github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:802 +0x27f

Jira issue: CRDB-38873
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented May 19, 2024

Hi @nameisbhaskar, please add a C-ategory label to your issue. Check out the label system docs.

There was no T-eam label on the issue. Please add one so that the issue can be triaged by a team.

While you're here, please consider adding an A- label to help keep our repository tidy.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added the X-blathers-untriaged blathers was unable to find an owner label May 19, 2024
@nameisbhaskar nameisbhaskar self-assigned this May 19, 2024
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 19, 2024
@nameisbhaskar
roachprod: handle nodeIPPorts map update atomicity
817189e 
when creating cluster config service discovery
for each node happens concurrently, and the
information is added to a common map. If the
map getting updated at the same time, roachprod
panics for concurrent update.
the fix is to have a mutex to protect the same.

informs: cockroachdb#124400
Epic: none
@nameisbhaskar nameisbhaskar mentioned this issue May 19, 2024
Merged
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 19, 2024
@nameisbhaskar
roachprod: handle nodeIPPorts map update atomicity
3af4f18 
when creating cluster config service discovery
for each node happens concurrently, and the
information is added to a common map. If the
map getting updated at the same time, roachprod
panics for concurrent update.
the fix is to have a mutex to protect the same.

Fixes: cockroachdb#124400
Epic: none
craig bot pushed a commit that referenced this issue May 20, 2024
@nameisbhaskar
Merge #124401
5d01328 
124401: roachprod: handle nodeIPPorts map update atomicity r=srosenberg a=nameisbhaskar

when creating cluster config service discovery
for each node happens concurrently, and the
information is added to a common map. If the
map getting updated at the same time, roachprod
panics for concurrent update.
the fix is to have a mutex to protect the same.

Fixes: #124400
Epic: none

Co-authored-by: Bhaskarjyoti Bora <[email protected]>
@craig craig bot closed this as completed in 36ac171 May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nameisbhaskar nameisbhaskar

Labels
X-blathers-untriaged blathers was unable to find an owner
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachprod: handle nodeIPPorts map update atomicity nameisbhaskar/cockroach
1 participant
@nameisbhaskar