2.6-head | Can't add a port type clusterIP to a deployment created in the old UI

[sgapanovich]

Rancher Server Setup

• Rancher version: v2.6-ba4347297ef8984f629650979a1b3a470592f4da-head
• Installation option (Docker install/Helm Chart): Docker Install
• Proxy/Cert Details:n/a

Information about the Cluster

• Kubernetes version: v1.21.5
• Cluster Type (Local/Downstream): Downstream
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Provisioned RKE cluster in EC2

Describe the bug

Can't add a port type clusterIP to a deployment created in the old UI

To Reproduce

• Navigate to the old UI
• Go to workloads
• Click "Deploy" (type: deployment, "Scalable deployment of")
• Add a port (type: clusterIP)
• Click "Launch"
  NOTE: at this point you have a deployment created and a service created

• Navigate to the new UI
• Select your cluster
• Go to Workloads
• Go to Deployments
• Select "Edit Config" next to your deployment
• Add a port
  type: clusterIP

• Open Devtools so you can see what request payload the UI sends
• Click "Save"
  NOTE: UI sends both ports

Result
The newly passed port is not added to the service

Expected Result

It is expected that the clusterIP will have another port added
There is a workaround if you

• Go to workloads again
• Go to Deployments
• Click "Edit Config" next to your deployment
  You will notice that the dropdown was reset to “Do not create a service”

• Select "Cluster IP"
• Click "Save"
  NOTE: the UI sends exactly the same request payload like it did the first time but this time the service is created

Had no ability to test/reproduce it if somebody upgrades from 2.5 to 2.6

[sgapanovich]

@kinarashah the issue i mentioned today

[brendarearden]

TLDR; the old ui is using PUT on edit and the new UI is using POST on edit, which is causing an error because the "test" service already exists.

I wasn't able to see the exact same recreation, but I had a similar experience.

Following the reproduction steps, I went to the old ui, created a deployment:

• deployment name: test
• add ClusterIP port with port 8080.
• Left all other values as default values

At this point I see my pod and service up and running

I then navigate to the new UI to edit the deployment.

I see that the Service Type on my ClusterIP is set to Do not create a service

I add another port with:

• Service Type: ClusterIP
• Name: another
• Container Port: 8081
• Protocol: TCP

I then click save and see the following error:

Here is the POST body, you can see that both ports are not included in the POST.

{type: "service", spec: {ports: [{name: "another", protocol: "TCP", port: 8081, targetPort: 8081}],…},…}
metadata: {name: "test", namespace: "default",…}
annotations: {field.cattle.io/targetWorkloadIds: "['default/test']", management.cattle.io/ui-managed: "true"}
name: "test"
namespace: "default"
ownerReferences: [{apiVersion: "apps/v1", controller: true, kind: "Deployment", name: "test",…}]
0: {apiVersion: "apps/v1", controller: true, kind: "Deployment", name: "test",…}
spec: {ports: [{name: "another", protocol: "TCP", port: 8081, targetPort: 8081}],…}
ports: [{name: "another", protocol: "TCP", port: 8081, targetPort: 8081}]
0: {name: "another", protocol: "TCP", port: 8081, targetPort: 8081}
name: "another"
port: 8081
protocol: "TCP"
targetPort: 8081
selector: {workload.user.cattle.io/workloadselector: "apps.deployment-default-test"}
workload.user.cattle.io/workloadselector: "apps.deployment-default-test"
type: "ClusterIP"
type: "service"

If I cancel out of the edit, and go back into the deployment edit screen I see both ports still listed, even though the creation was not successful

Running kubectl describe svc test still returns the same info

And checking the get response for the edit page, I only see one port listed for the test service.

I then click create, leaving the Service Type: Do Not Create Service and it does not error but the new clusterIP is not created.

If I attempt to change them both to ClusterIP:

Then I get the following error:

But the POST contains both Ports

{type: "service",…}
metadata: {name: "test", namespace: "default",…}
spec: {ports: [{name: "8080tcp02", protocol: "TCP", port: 8080, targetPort: 8080},…],…}
ports: [{name: "8080tcp02", protocol: "TCP", port: 8080, targetPort: 8080},…]
0: {name: "8080tcp02", protocol: "TCP", port: 8080, targetPort: 8080}
1: {name: "another", protocol: "TCP", port: 8081, targetPort: 8081}
selector: {workload.user.cattle.io/workloadselector: "apps.deployment-default-test"}
type: "ClusterIP"
type: "service"

If I do the same edit in the old UI, I see that it is utilizing PUT instead of POST and both ports get added as expected.

[sgapanovich]

@brendarearden the issue with "service already exists" has been fixed. Please see first issue here #4159 and test notes in the comment section

[brendarearden]

I was able to recreate this issue, and I did confirm that I saw PUT for the Deployment has both sets of ports on the container:

But after digging into it a little bit I saw that the way the service gets generated in the rancher code is from the template spec annotations.
https://github.com/rancher/rancher/blob/8a9d383baac34a4d518741831f56a64c9d2988bf/pkg/controllers/managementagent/workload/workload.go#L88

If I put a breakpoint at this spot, I see that when making edits in the old ui, both Cluster IP ports are listed in the workload.TemplateSpec.Annotations[PortsAnnotation], but when editing in the new ui, only the original port shows up in the annotation. So it would appear that the Cluster IP that was added during edit is not making its way into the field.cattle.io/ports annotation on the template spec.

[mantis-toboggan-md]

@brendarearden it looks like you've tested with 2.6.1-rc1 it should be working on 2.6.1-rc2; HOWEVER, the new UI doesn't leverage those annotation at all (didn't even know they were a thing when I added this to the dashboard 9 mo ago tbh). I'm unsure if I need to start adding them for workloads made/edited in the new UI; as far as I can tell that alone isn't enough to make services with workloads.

When you add workload ports in the old UI the workload POST contains info about those services and the backend creates them. When you add workload ports in the new UI, the workload POST is just the kubernetes workload config -- it has spec.template.spec.containers[].ports[] but they don't explicitly define associated services -- then separate POST requests create the services. "service already exists" bug was happening because the new UI wasn't detecting the services created the old way, and was trying to make new ones with the same name instead of editing (POST instead of PUT as you already found).

It sounds like workload.TemplateSpec.Annotations['field.cattle.io/ports'] (this appears in spec.template.metadata.annotations in dashboard/steve) is being used by the backend to make services but merely adding that label when creating a workload through steve is insufficient: I made a deployment in the old UI with a nodeport defined, this created ClusterIP and Nodeport services. If I clone that workload in the new UI, and ensure the annotation has the right workload name in it, then intentionally bypass the new UI's form of service creation, the service's aren't automatically created by the backend.

If you think there's something essential about the annotation that I've overlooked then let me know and I'll get to work adding it to the dashboard workload create process.

[gaktive]

At least there's a workaround for 2.6.1 users. Should release note the workaround for people upgrading until we can sort this out in an upcoming release.

[gaktive]

@brendarearden did you see Nancy's comment asking you about the annotation?

[brendarearden]

@gaktive I had not, thank you for re-@ing me! @mantis-toboggan-md with the new information you have provided, I will walk through the same steps you used and see if there is anything that I missed in regards to the annotations.

[mantis-toboggan-md]

This is ties into to a backend issue rancher/rancher#34981

[catherineluse]

I'm closing this issue as stale due to its age, lack of activity, and the fact that the workaround exists. If this fix is still needed, we can reopen the ticket.