endpoints cannot be changed from notReadyAddresses to addresses

[chymy]

What happened?

I have a k8s clusters that have encountered the following situation:
1、All master nodes are full of memory burst
2、After the memory of all master nodes is restored, the Ready type status of pod is true, but some endpoints in the subsets still remain in notReadyAddresses and do not recover.
service:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-05-29T07:42:43Z"
  name: nginx-service
  namespace: default
  resourceVersion: "412201"
  uid: 17f43ff9-16fe-4507-a593-ede1a29a1255
spec:
  clusterIP: 123.123.95.164
  clusterIPs:
  - 123.123.95.164
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx

enpoints:

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2024-06-12T02:41:41Z"
 creationTimestamp: "2024-05-29T07:42:43Z"
  name: nginx-service
  namespace: default
  resourceVersion: "468584"
  uid: 85b1134f-7f5d-4087-b76f-e7aa5371c016
subsets:
- notReadyAddresses:
  - ip: 172.22.161.17
    nodeName: 192.168.3.89
    targetRef:
      kind: Pod
      name: nginx
      namespace: default
      uid: 6022f2d6-ead8-4882-9a76-8e51901731ee
  ports:
  - port: 80
    protocol: TCP

pod

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-05-29T07:42:43Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-05-29T07:42:49Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-05-29T07:42:49Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-05-29T07:42:43Z"
    status: "True"
    type: PodScheduled

3、The issue is resolved only by restarting kube-controller-manager.

/area controller-manager

What did you expect to happen?

When all master node memory is restored, endpoints change from notReadyAddresses to addresses

How can we reproduce it (as minimally and precisely as possible)?

Refer to the description of "What happened?"

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
1.28.3

Cloud provider

None

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

/sig network

[aroradaman]

/assign

[aojea]

looks like a duplicate of this #117193

[QuantumEnergyE]

#125638

[aojea]

can we please aggregate all of them in one , having information spreaded across issues is making it more difficult to track, it seems we know the root cause, we have a workaround #117193 (comment)

it is missing someone coming with a proposal to fix it

[aojea]

/close

duplicate of #117193 (comment) and #125638

@QuantumEnergyE I want to have only one issue for this, is there any special motivation you want to use #125638 instead of #117193?

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close

duplicate of #117193 (comment) and #125638

@QuantumEnergyE I want to have only one issue for this, is there any special motivation you want to use #125638 instead of #117193?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.