Implement client-side mTLS support settings

[andsens]

See #696

This is far from done. I tested it and quickly ran into issues with cert-manager:

{"level":"error","time":"2023-07-19T13:12:34Z","msg":"looking up info for HTTP challenge","service":"autocert","host":"test.example.com","error":"no information found to solve challenge for identifier: test.example.com"}
{"level":"error","error":"a valid client certificate is required to access this page","status":495,"status-text":"a valid client certificate is required to access this page","request-id":"c28412c5-e6c5-429e-a062-47ab3ad0585d","time":"2023-07-19T13:12:34Z","message":"httputil: error"}
{"level":"info","service":"authorize","request-id":"c28412c5-e6c5-429e-a062-47ab3ad0585d","check-request-id":"c28412c5-e6c5-429e-a062-47ab3ad0585d","method":"GET","path":"/.well-known/acme-challenge/i2K0F783PCrFnPkXh2jk2njfSn4GASGPhBOPnX0wvhk","host":"test.example.com","query":"","ip":"100.103.219.130","allow":true,"allow-why-true":["accept"],"deny":true,"deny-why-true":["invalid-client-certificate"],"user":"","email":"","time":"2023-07-19T13:12:34Z","message":"authorize check"}
{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"test.example.com","path":"/.well-known/acme-challenge/i2K0F783PCrFnPkXh2jk2njfSn4GASGPhBOPnX0wvhk","user-agent":"cert-manager-challenges/v1.9.1 (linux/amd64) cert-manager/4486c01f726f17d2790a8a563ae6bc6e98465505","referer":"http:https://test.example.com/.well-known/acme-challenge/i2K0F783PCrFnPkXh2jk2njfSn4GASGPhBOPnX0wvhk","forwarded-for":"100.103.219.130","request-id":"c28412c5-e6c5-429e-a062-47ab3ad0585d","duration":4.265011,"size":1560,"response-code":495,"response-code-details":"ext_authz_denied","time":"2023-07-19T13:12:34Z","message":"http-request"}

The autocert thing is a bit concering, I thought that was disabled in Kubernetes?
I also tried setting ingress.pomerium.io/allow_any_authenticated_user: 'true' on the acme solver ingress, but that gave the same errors. It would seem that the client cert requirement kicks in before the authentication check is performed?

[CLAassistant]

All committers have signed the CLA.

[wasaga]

Hello,

The overall direction of your PR is correct.

The issue with cert-manager is:

• when you enable client CA, naturally, all inbound requests would require to present a client TLS cert during TLS handshake process.
• That would interfere with HTTP01 challenge solver; when LetsEncrypt would try to access the challenge URL, a client cert would be requested that matches your CA, and LetsEncrypt would be unable to present one.

unfortunately there's no way to disable client cert request just for the HTTP01 challenges as this is configured per port (443) and as such, HTTP01 challenges are fundamentally incompatible with mTLS. You have to use a different cert-manager Issuer challenge type, such as DNS.

Please disregard autocert message, it's coming from OCSP code that's historically running inside autocert module. There's no autocert running in Kubernetes. This runaway message is tracked in pomerium/pomerium#4245

[andsens]

Thank you for the super quick answer!

unfortunately there's no way to disable client cert request just for the HTTP01 challenges as this is configured per port (443) and as such, HTTP01 challenges are fundamentally incompatible with mTLS. You have to use a different cert-manager Issuer challenge type, such as DNS.

Wouldn't it be possible to disable the https redirect on port 80 for the ACME challenge routes? That way we circumvent the problem entirely.

EDIT: Background info: Let's Encrypt verifies the challenges using plaintext requests but follows redirects.

[wasaga]

currently everything is configured on port 443, port 80 is only doing redirects.
I tracked that request in a separate ticket #698

For the time being I recommend to switch to DNS challenges.

[andsens]

You know, having thought a bit about this I think the entire approach might be wrong. mTLS is such a different beast from how Pomerium otherwise does things, that fully supporting it might end up ruining everything else. I'm thinking of just hosting node-oidc-provider and then connecting it to Pomerium.
That way you can extract all kinds of nifty things from the certificate subject and embed it in the OIDC response in order to do policy on top of that using Pomerium. You know, using the right tools for what they're good at :-)
Is there a way of doing that? I don't see any support for a generic OIDC in the docs...

[kenjenkins]

Hi @andsens, I believe you can set provider to oidc to use a generic OIDC provider, see https://www.pomerium.com/docs/deploy/k8s/reference#identityprovider.

I should also mention that we're currently in the middle of rethinking Pomerium's mTLS support, so if you have any specific suggestions or feature requests please feel free to share them. I can't make any promises, but if you have any thoughts I'd be interested to hear them.

[andsens]

if you have any specific suggestions or feature requests please feel free to share them.

I haven't been able to find a discussion or issue related to this, so I guess I'll just post my comments here :-)

The advantage of implementing mTLS directly in Pomerium is that API integration for anything Pomerium is proxying is super easy. Just setup e.g. an openssl config, reference it with curl, and you're done. This gets quite a bit more advanced if there is a separate mTLS endpoint that needs to be authenticated against first.

The disadvantage is that mixing mTLS into all of this can become super complicated. The requirements for it and configuration of it is just completely different from all the other authentication flows, which might mean having to account for it in every single feature that you implement (or worse, you forget to account for it).

I see a solution that could circumvent both issues, but it's a little more complicated: A built-in, but separate, mTLS server that acts as an OIDC provider and can authenticate browser clients, and an authentication flow that supports JWK/the x5c field in JWT tokens (rfc).
This way programmatic access doesn't require multiple requests or special openssl configuration and can be implemented using the dozens of libraries already available. At the same time you can still support seamless user access through browsers with installed client certs.

The advantage of using a single OIDC endpoint for browser mTLS is that you can avoid presenting the user with PIN prompts (when e.g. using a HSM) for every pomerium protected site they visit.

EDIT: Oh, and another advantage to this approach is that you don't have to figure out how to get mTLS working with HTTP3/QUIC

[kenjenkins]

Hi @andsens, delegating mTLS to a separate identity provider is definitely an interesting idea. I think one could even prototype this today without further changes to Pomerium itself: if this OIDC provider returned a custom claim in its ID token, you could write a Pomerium policy with a claim/ criterion against it. (This claim could be as simple as "validated: true" if all you care about is certificate validity, or something more complicated like "certificate_email: <email address>" if you wanted to, say, extract a SAN email address from the certificate.)

[andsens]

@kenjenkins That's actually exactly what I have done now! There are still some deps that require a little bit of documentation before publishing it. Basically the config for my "mTLS to OIDC bridge" looks like this:

---
issuerUrl: https://mtls-oidc-idp.mtls-oidc-idp.svc.cluster.local:8444/
claimMap:
  sub: x500UniqueIdentifier
  email:
    email: emailAddress
  profile:
    name: CN
    preferred_username: x500UniqueIdentifier
oidcConfig:
  clients:
  - client_id: pomerium
    client_secret: [...]
    redirect_uris: [https://pomerium.local/oauth2/callback]

So now I just configure pomerium with:

  jwtClaimHeaders:
    X-Forwarded-User: email

and annotate ingresses with e.g.:

  annotations:
    cert-manager.io/cluster-issuer: development-ca
    ingress.pomerium.io/pass_identity_headers: "true"
    ingress.pomerium.io/policy: |
      allow:
        and:
        - user:
            is: andsens

Next up I might start adding custom claims so that you can e.g. grant access to an entire department by mapping the certificate "OU".