You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a cluster of about 40 nodes and we have 2 nodes that refuse to leave. Using force-leave (with and without -prune) removes them briefly but then they get rebroadcast back into the node list. These nodes have been dead for weeks if not months.
Reproduction Steps
Unknown how to reproduce but we got here doing regular green/blue deployments for the past number of years:
Running on AWS AL2023 on EC2's in an on-demand ASG.
Log Fragments
Can attach debug log payload to ticket but basically once the force-leave is issued, very shortly after we see an EventMemberJoin message come through with the dead node getting re-added.
The text was updated successfully, but these errors were encountered:
Overview of the Issue
We have a cluster of about 40 nodes and we have 2 nodes that refuse to leave. Using force-leave (with and without -prune) removes them briefly but then they get rebroadcast back into the node list. These nodes have been dead for weeks if not months.
Reproduction Steps
Unknown how to reproduce but we got here doing regular green/blue deployments for the past number of years:
Consul info for both Client and Server
Client info
Server info
Operating system and Environment details
Running on AWS AL2023 on EC2's in an on-demand ASG.
Log Fragments
Can attach debug log payload to ticket but basically once the force-leave is issued, very shortly after we see an EventMemberJoin message come through with the dead node getting re-added.
The text was updated successfully, but these errors were encountered: