CASSANDRA-19705: Reconfigure CMS after replacement, bootstrap and move operations

[krummas]

Thanks for sending a pull request! Here are some tips if you're new here:

• Ensure you have added or run the appropriate tests for your PR.
• Be sure to keep the PR description updated to reflect all changes.
• Write your PR title to summarize what this PR proposes.
• If possible, provide a concise example to reproduce the issue for a faster review.
• Read our contributor guidelines
• If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:

<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

[beobal]

There's a comment on reconfigureCMS saying it's only to be used for interactive purposes due to its lack of retries. I think this comment is simply outdated as retries is baked into the processor implementations now.

[krummas]

yeah either way I think we're good here - if the reconfiguration fails we'll alert the operator via metrics and nodetool cms

[beobal]

I think it might actually be better to unconditionally reconfigure as it removes the window between checking if we need to and submitting. It's currently safe to do so anyway as a no-op will add 2 entries to the log, something like:

20 |  72058140398682507 | 2024-06-20 12:40:49.681000+0000 |         ADVANCE_CMS_RECONFIGURATION | AdvanceCMSReconfiguration{idx=0, current=FinishReconfiguration(), diff=Diff{additions=[], removals=[]}, activeTransition=null}
19 |  72058140398682506 | 2024-06-20 12:40:49.631000+0000 | PREPARE_COMPLEX_CMS_RECONFIGURATION | PrepareCMSReconfiguration#Complex{replicationParams=ReplicationParams{class=org.apache.cassandra.locator.MetaStrategy, datacenter1=3}}

but we could reduce this to just the prepare entry if we move the check to PrepareCMSReconfiguration::execute

if (newCms.equals(currentCms))
{
    logger.info("Proposed CMS reconfiguration resulted in no required modifications at epoch {}", prev.epoch.getEpoch());
    return Transformation.success(prev.transformer(), LockedRanges.AffectedRanges.EMPTY);
}

[beobal]

if we do make a change to submit the reconfig unconditionally, it would be good to still log something like this in the event that the submission fails, maybe at WARN?