Route in unknown status in v1.14

[dicolasi]

What version of Knative?

0.14.x

Expected Behavior

Route should be created and be in Ready status.

Actual Behavior

Route stuck in unknown status.

Name:         hasura
Namespace:    hasura
Labels:       argocd.argoproj.io/instance=property-xyz-hasura
              serving.knative.dev/service=hasura
Annotations:  serving.knative.dev/creator: system:serviceaccount:argocd:argocd-application-controller
              serving.knative.dev/lastModifier: system:serviceaccount:argocd:argocd-application-controller
API Version:  serving.knative.dev/v1
Kind:         Route
Metadata:
  Creation Timestamp:  2024-05-22T12:40:48Z
  Finalizers:
    routes.serving.knative.dev
  Generation:  1
  Owner References:
    API Version:           serving.knative.dev/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Service
    Name:                  hasura
    UID:                   a5e3e3ff-838a-4576-a0b7-ae0334735a6c
  Resource Version:        799848
  UID:                     8fe489e0-78b3-4863-9b5e-146f4f8cd8c1
Spec:
  Traffic:
    Configuration Name:  hasura
    Latest Revision:     true
    Percent:             100
Status:
  Address:
    URL:  http:https://hasura.hasura.svc.cluster.local/
  Conditions:
    Last Transition Time:  2024-05-22T12:42:09Z
    Status:                True
    Type:                  AllTrafficAssigned
    Last Transition Time:  2024-05-22T12:40:48Z
    Status:                Unknown
    Type:                  CertificateProvisioned
    Last Transition Time:  2024-05-22T12:40:49Z
    Status:                True
    Type:                  IngressReady
    Last Transition Time:  2024-05-22T12:42:09Z
    Status:                Unknown
    Type:                  Ready
  Observed Generation:     1
  Traffic:
    Latest Revision:  true
    Percent:          100
    Revision Name:    hasura-00001
  URL:                http:https://hasura.hasura.svc.cluster.local/
Events:
  Type     Reason                 Age                 From              Message
  ----     ------                 ----                ----              -------
  Warning  InternalError          91s (x11 over 91s)  route-controller  route: "hasura" does not own Service: "hasura"
  Warning  FinalizerUpdateFailed  91s                 route-controller  Failed to update finalizers for "hasura": Operation cannot be fulfilled on routes.serving.knative.dev "hasura": the object has been modified; please apply your changes to the latest version and try again
  Normal   FinalizerUpdate        91s                 route-controller  Updated "hasura" finalizers
  Normal   Created                91s                 route-controller  Created placeholder service "hasura"
  Normal   Created                90s                 route-controller  Created Ingress "hasura"
  Warning  InternalError          60s (x4 over 60s)   route-controller  failed to add route annotation to Namespace=hasura Name="hasura-00001": revision.serving.knative.dev "hasura-00001" not found

downgrading to v1.13 solved the problem.

Steps to Reproduce the Problem

Here the knative configuration

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  registry:
    imagePullSecrets:
      - name: container-registry-credentials
  ingress:
    kourier:
      enabled: true
  config:
    certmanager:
      issuerRef: |
        kind: ClusterIssuer
        name: letsencrypt-production
    domain:
      svc.cluster.local: ""
    network:
      ingress-class: "kourier.ingress.networking.knative.dev"
      autoTLS: Enabled
      autocreateClusterDomainClaims: "true"
      httpProtocol: Redirected
    autoscaler:
      stable-window: "60s"
      enable-scale-to-zero: "true"
      pod-autoscaler-class: "kpa.autoscaling.knative.dev"
    features:
      autodetect-http2: "Enabled"

The kubernetes version 1.28.9

[dprotaso]

To confirm - what version of cert manager are you using?

[dprotaso]

Can you also paste an example knative service that you see failing when you deploy to your environment?

[dicolasi]

Cert manager

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
resources:
  - https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.yaml

[dicolasi]

Here an example of service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hasura
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
        autoscaling.knative.dev/max-scale: "1"
        autoscaling.knative.dev/metric: "rps"
        autoscaling.knative.dev/target: "25"
    spec:
      containers:
        - image: hasura/graphql-engine:v2.39.2
          ports:
            - containerPort: 8080
          env:
            - name: POSTGRES_USERNAME
              valueFrom:
                secretKeyRef:
                  name: hasura-db-creds
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: hasura-db-creds
                  key: password
            - name: POSTGRES_DATABASE
              value: hasura
            - name: POSTGRES_HOST
              value: hasura-pooler.postgres.svc.cluster.local
            - name: POSTGRES_PORT
              value: "5432"
            - name: HASURA_GRAPHQL_DATABASE_URL
              value: postgres:https://$(POSTGRES_USERNAME):$(POSTGRES_PASSWORD)@$(POSTGRES_HOST):$(POSTGRES_PORT)/$(POSTGRES_DATABASE)
              ## enable the console served by server - need to run `hasura console` to access instead
            - name: HASURA_GRAPHQL_ENABLE_CONSOLE
              value: "true"
              ## enable debugging mode. It is recommended to disable this in production
            - name: HASURA_GRAPHQL_DEV_MODE
              value: "false"
            - name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
              value: startup, http-log, webhook-log, websocket-log, query-log
          resources:
            requests:
              cpu: 100m
              memory: 2Gi
            limits:
              cpu: 1000m
              memory: 4Gi

[dprotaso]

/triage accepted

Confirmed this is an issue.

I used a simplified knative service and used a test issuer from : https://github.com/knative/serving/blob/main/test/config/externaldomaintls/certmanager/caissuer/issuer.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hasura
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
        autoscaling.knative.dev/max-scale: "1"
        autoscaling.knative.dev/metric: "rps"
        autoscaling.knative.dev/target: "25"
    spec:
      containers:
        - image: ghcr.io/knative/helloworld-go:latest 
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 2Gi
            limits:
              cpu: 1000m
              memory: 4Gi

[dprotaso]

git bisect points to the regression being introduced in #14610

[dprotaso]

cherry pick is out - #15243

next point release should pick it up (so next tuesday) - thanks for the report @dicolasi