Ingress example returns 404

[shishkin]

What happened?

I followed the steps in Kubernetes guide installing and configuring ingress controller with the example verify app. All the status events seem ok (e.g. saying config updated) but the verify application is not accessible outside the cluster. Envoy proxy returns 404 Not Found.

What did you expect to happen?

Application exposed via ingress is reachable.

How'd it happen?

https://www.pomerium.com/docs/deploying/k8s/quickstart

What's your environment like?

• Pomerium version (retrieve with pomerium --version): Image pomerium/ingress-controller:sha-cdc389c
• Server Operating System/Architecture/Cloud: MacOS, Docker, K3s via K3d.

I have no other ingress controllers besides Pomerium installed. I've set Pomerium ingress controller class as default.

I also changed service ports to 8443 and 8080 as those are the ports my K3s LB is configured.

What's your config.yaml?

kubectl describe -n pomerium pomerium/global:

Spec:
  Authenticate:
    URL:  https://login.cluster.local:8443
  Certificates:
    pomerium/tls
  Identity Provider:
    Provider:  oidc
    Secret:    pomerium/pomerium-idp-client
    URL:       https://dex.cluster.local:8443
  Secrets:     pomerium/bootstrap
Status:
  Ingress:
    default/verify:
      Observed At:          2022-12-13T19:01:45Z
      Observed Generation:  1
      Reconciled:           true
    dex/dex:
      Observed At:          2022-12-13T19:01:45Z
      Observed Generation:  1
      Reconciled:           true
  Settings Status:
    Observed At:          2022-12-13T19:01:45Z
    Observed Generation:  2
    Reconciled:           true
    Warnings:
      storage: please specify a persistent storage backend, please see https://www.pomerium.com/docs/topics/data-storage#persistence
Events:
  Type     Reason      Age                From                                     Message
  ----     ------      ----               ----                                     -------
  Normal   Updated     73m                bootstrap pod/pomerium-845fc44d77-krk8b  config updated
  Warning  Validation  73m                pomerium-crd                             storage: please specify a persistent storage backend, please see https://www.pomerium.com/docs/topics/data-storage#persistence
  Normal   Updated     73m                pomerium-crd                             config updated
  Normal   Updated     73m (x2 over 73m)  pomerium-ingress                         default/verify: config updated
  Normal   Updated     73m (x3 over 73m)  pomerium-ingress                         dex/dex: config updated

I did set the ingress public assuming it will bypass any auth. Ingress kubectl get -n default ingress/verify -o yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.pomerium.io/allow_public_unauthenticated_access: "true"
    ingress.pomerium.io/pass_identity_headers: "true"
  creationTimestamp: "2022-12-13T18:31:19Z"
  generation: 1
  name: verify
  namespace: default
  resourceVersion: "31717"
  uid: b2cad25b-6d07-4df5-93b2-0a315cb411de
spec:
  ingressClassName: pomerium
  rules:
  - host: web.cluster.local
    http:
      paths:
      - backend:
          service:
            name: verify
            port:
              number: 8000
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 172.25.0.3
    - ip: 172.25.0.4

What did you see in the logs?

{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"web.cluster.local:8443","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36","referer":"","forwarded-for":"10.42.0.1","request-id":"3b1d1d98-5469-41b6-8e18-ada5274d3b95","duration":0.393458,"size":0,"response-code":404,"response-code-details":"route_not_found","time":"2022-12-13T20:09:47Z","message":"http-request"}

Additional context

I have Dex setup as IdP and exposed as via another ingress on another host (all hosts points to 127.0.0.1 via my local /etc/hosts). That ingress also gives 404 although it worked previously when I used it with Traefik without Pomerium. I assume IdP setup should not influence basic ingress controller functionality.

[calebdoxsey]

I think the issue here may be related to the non standard ingress port. The host rule web.cluster.local probably doesn't match web.cluster.local:8443, and the ingress spec does not allow port numbers in the host rule (the docs state that 80 and 443 are implied). I'm not sure if this is our issue or a limitation of ingress definitions.

[shishkin]

I'm not sure if this is our issue or a limitation of ingress definitions.

Traefik ingress controller works with such configuration using default ingress resource kind.

Also note that https://web.cluster.local:8443 is handled by the LB and routed to the proxy correctly otherwise I wouldn't see 404 response from envoy. From that point on, proxy should not bother with ports if it received a request on its https endpoint.

[calebdoxsey]

From that point on, proxy should not bother with ports if it received a request on its https endpoint.

The port is part of the authority: "authority":"web.cluster.local:8443". I disagree that the proxy should not care. Pomerium route definitions should contain the non-standard port as part of the host name.

Maybe we can write something to modify the host field in the ingress controller so its converted into a host with the non-standard port.

[shishkin]

I see. Let me try to use ports 80 and 443 and see if it works.

[shishkin]

Thanks @calebdoxsey, changing to default ports did solve the issue for me! I didn't realize that I could run standard ports locally.

Still supporting non-standard ports would be useful. Maybe Pomerium could support it through an extra annotation.