UDP amplification?

[riastradh]

When I run kinit with an existing principal but the wrong password in a realm I use, it sends a 175-byte UDP packet to the KDC, and the KDC replies with a 313- or 367-byte packet. (kinit and KDC are both Heimdal.)

I haven't looked at the details of the messages here but evidently authentication is not required to provoke this ~2x-sized reply from the KDC, so it appears to serve as a DoS amplification vector. Perhaps not as big as a DNS ANY or DNSSEC query reply, but still ~2x.

Does Heimdal do anything to avoid or mitigate this (preferably out of the box), other than disabling UDP in the KDC?

[nicowilliams]

Hmm, w/o FAST armoring I'm observing a 160 byte response to a 289 byte request. However, PKINIT can probably be made to produce some amplification.

I believe Heimdal does nothing about this. In the open Internet it'd be best to disable UDP or filter it out. If this is a problem we could implement a throttle that sends back reduced KRB-ERROR responses, or no responses when requests fail. Because the Heimdal KDC is multi-processed we'd have to keep a per-process throttle, an exponentially decaying running average of failing request rate, and trigger the mitigation on some level.

[riastradh]

Can UDP access be limited to using a TGT to get a service ticket? And if that path fails because the TGT is bad, does it amplify?

If this works, initial login with kinit requires TCP and can't be a UDP amplifier, but there's no TCP round-trip overhead for getting service tickets once you've logged in.

[nicowilliams]

Can UDP access be limited to using a TGT to get a service ticket? And if that path fails because the TGT is bad, does it amplify?

In principle the AS and TGS can be separate services. In practice the discovery mechanisms don't support that... but the AS could respond with a bare-bones (all OPTIONAL fields left out) KRB-ERROR w/ KRB_ERR_FIELD_TOOLONG. And, of course, the AS could do that with failures only.

[elric1]

You don't have to entirely disable UDP, necessarily, you could always send KRB5KRB_ERR_RESPONSE_TOO_BIG if certain amplification criteria are met, that is: if the response packet is more than a certain percentage larger than the input packet.
I don't think that it makes sense to directly distinguish between AS and TGS, because that's not the determinant of whether it's an amplification: the size of the response packet is. IIRC, there is already logic in the code to deal with responses that are too large, so it shouldn't be terribly hard to implement.

[nicowilliams]

Oops, yes, I meant that error, not the other.

[nicowilliams]

I'd be inclined to do this for all AS requests, really, because the first one always fails as its purpose is to discover the pre-authentication options available. As long as it interops, I'd be fine with that.

[elric1]

Might be an idea to have the client prefer TCP for the AS in all cases if we do this.
It might be possible to get larger amplifications using the contents of the PAC. Certainly with a Windows KDC, an AS_REQ can potentially return a very large TGT for some users. Granted, IIRC, Windows only uses TCP, we may support large PACs either now or in a potential future release.
This can be a potential problem for the TGS as well as, I think, that ticket renewal, which uses the TGS, may want to re-evaluate group membership to construct a new PAC, or, perhaps, x-realm TGS TGT requests might also want to reconstruct PACs. I haven't looked into this, but it might be worth considering.
Do we do anything with PACs at the moment?

[nicowilliams]

Oh, and remember that the KDC protocol is stateless, so replays within the skew window are a way to generate amplification in both the AS and TGS cases. So I think maybe we do need an adaptive mechanism, and it shouldn't only be driven by failures because the fact that the request is authenticated doesn't prove it's not a replay.

[nicowilliams]

Do we do anything with PACs at the moment?

Yes, but very limited (no group membership list, so our PACs are always small).

[nicowilliams]

Maybe we want to return KRB5KRB_ERR_RESPONSE_TOO_BIG whenever the reply is larger than the request, or even always, and maybe we want the client to try TCP first. I'd be happy to go this route.

[elric1]

I'd be clear that I only think that we should go with TCP first on the relatively infrequent AS_REQs and not for the more common lighter weight TGS_REQs. And yes, the KRB5_ERR_RESPONSE_TOO_BIG makes sense to me in the case that the response is larger than the request.
One last question: do we think that this should be configurable? On the one hand, amplification attacks are a potential issue on the open internet, however, many KDC implementations are behind firewalls and may not need this protection.

[nicowilliams]

There is the [kdc] max-kdc-datagram-reply-length parameter, which defaults to 1400 (reply PDU size in bytes), which causes Heimdal to send KRB5_ERR_RESPONSE_TOO_BIG when reply PDU sizes exceed that. So there is a configuration parameter that is available now that can be tuned way down to force the use of TCP.

[riastradh]

One last question: do we think that this should be configurable? On the one hand, amplification attacks are a potential issue on the open internet, however, many KDC implementations are behind firewalls and may not need this protection.

Default should always be safe and responsible to run on the open internet out of the box.

---

What's the smallest AS-REQ or TGS-REQ packet that Heimdal will reply to?

How large is a KRB5_ERR_RESPONSE_TOO_BIG KRB-ERROR packet?

It looks like there is logic to apply a statically configured limit to AS-REP packets over UDP, but I can't find it for TGS-REP packets:

heimdal/kdc/kerberos5.c

Lines 2743 to 2750 in cb9a130

	/*
	* Check if message is too large
	*/
	if (r->datagram_reply && r->reply->length > config->max_datagram_reply_length) {
	krb5_data_free(r->reply);
	ret = KRB5KRB_ERR_RESPONSE_TOO_BIG;
	_kdc_set_e_text(r, "Reply packet too large");
	}

Perhaps this should just be adjusted to do the same if the reply length exceeds the request length, which I think is also available at this point in the logic. Or perhaps it should be applied more consistently elsewhere, like here:

heimdal/kdc/connect.c

Lines 443 to 446 in cb9a130

	if(reply.length){
	send_reply(context, config, prependlength, d, &reply);
	krb5_data_free(&reply);
	}

[nicowilliams]

It looks like there is logic to apply a statically configured limit to AS-REP packets over UDP, but I can't find it for TGS-REP packets:

: ; git grep max_datagram
ChangeLog.2006: * kdc/kdc.h (krb5_kdc_config): Add max_datagram_reply_length.
kdc/default_config.c:    c->max_datagram_reply_length =
kdc/kdc_locl.h:    size_t max_datagram_reply_length;
kdc/kerberos5.c:    if (r->datagram_reply && r->reply->length > config->max_datagram_reply_length) {
kdc/krb5tgs.c:    if (datagram_reply && data->length > config->max_datagram_reply_length) {

On the TGS side this is handled in kdc/krb5tgs.c:_kdc_tgs_rep().

Perhaps this should just be adjusted to do the same if the reply length exceeds the request length, which I think is also available at this point in the logic. Or perhaps it should be applied more consistently elsewhere, like here:

No, kdc/connect.c's do_request()/send_reply() don't know how to construct a KRB5_ERR_RESPONSE_TOO_BIG error.

[nicowilliams]

What's the smallest AS-REQ or TGS-REQ packet that Heimdal will reply to?

It depends on the principal and realm names, but ~200 bytes.

If one uses PKINIT it's larger then the reply can easily be larger than the 1400 byte default UDP PDU limit. With ECC though it's possible that PKINIT messages will fit in 1400 bytes. I'm seeing almost 3x reply size with anonymous client PKINIT.

How large is a KRB5_ERR_RESPONSE_TOO_BIG KRB-ERROR packet?

Less than 200 bytes w/o FAST, around 600 bytes w/ FAST. When using FAST error responses are generally, if not always smaller than the request.

[nicowilliams]

Note that the [kdc] max-kdc-datagram-reply-length parameter does not apply to the KRB5_ERR_RESPONSE_TOO_BIG error PDU itself! Therefore we can easily and safely (as far as interop goes) tune it down to zero.

[nicowilliams]

It isn't currently possible to disable listening on UDP, but that's easily handled with IP filtering anyways.

[riastradh]

What's the smallest AS-REQ or TGS-REQ packet that Heimdal will reply to?

It depends on the principal and realm names, but ~200 bytes.

My experiment with [email protected] was only 175 bytes. Can we write down an exact lower bound to make a proof of non-amplification easier?

If smallest request that can provoke a reply is n bytes, and if a KRB5_ERR_RESPONSE_TOO_BIG reply is at most m bytes, then all we have to prove is m <= n to be done with it once Heimdal replaces any longer-than-request replies by KRB5_ERR_RESPONSE_TOO_BIG.

[nicowilliams]

My experiment with [email protected] was only 175 bytes. Can we write down an exact lower bound to make a proof of non-amplification easier?

One would have to either write [a small amount of] code or set up a 1 character realm with a 1 character principal, or calculate it by hand from the ASN.1 types and knowledge of DER. If I'd written a JSON->DER converter to go with the DER->JSON converter that lib/asn1 currently has then it'd be easier :/ (this is something I eventually want to do, but there's a huge pile of other work to do on lots of things). The amount of code to write may be less trivial than I was thinking.

You could grab PDUs from your traces and then dump them with lib/asn1/asn1_print to get an idea for how much space the various fields have, then referencing lib/asn1/krb5.asn1 you could see which OPTIONAL fields might be possible to drop (though also with reference to RFC 4120, because in some contexts an OPTIONAL field can still be required to be present).

[nicowilliams]

If smallest request that can provoke a reply is n bytes, and if a KRB5_ERR_RESPONSE_TOO_BIG reply is at most m bytes, then all we have to prove is m <= n to be done with it once Heimdal replaces any longer-than-request replies by KRB5_ERR_RESPONSE_TOO_BIG.

My take is that we can solve this with a) advice to sites w/ KDCs in the open to not advertise DNS SRV RRs for UDP (so legit clients don't try UDP), c), advice to sites to use IP filtering blackhole (or reject) UDP requests to port 88 on their KDCs in the open, c) us setting the max-kdc-datagram-reply-length default to zero. Sure, (c) doesn't prevent the attack if the smallest AS-REQ could be much smaller than the smallest KRB-ERROR, though I don't think that's likely.

Also, we should make sure that Heimdal's KRB5_ERR_RESPONSE_TOO_BIG replies are the smallest possible by dropping all non-essential OPTIONAL fields from the reply. This won't help that much in the FAST case, but in the FAST case the error is practically guaranteed to be smaller than the request anyways.

[nicowilliams]

Although RFC 6113 doesn't say, KRB5_ERR_RESPONSE_TOO_BIG effectively has to always be sent as a non-FAST KRB-ERROR, even when the client used FAST. That's what MIT Kerberos' KDC does, and it's what Heimdal's client expects. I don't know what AD and Windows do.

[riastradh]

Should this have a CVE assigned and mitigation instructions for Kerberos administrators? (Am I the first person to have thought of this? Surely not?)

[nicowilliams]

Should this have a CVE assigned and mitigation instructions for Kerberos administrators? (Am I the first person to have thought of this? Surely not?)

IMO it's of low urgency because there are much better amplification attacks out there. But I could be convinced otherwise. A bigger problem is that this wouldn't be specific to Heimdal -- this is a problem with an Internet protocol, so the next step would be to seek to publish an update to RFC 4120, or a BCP (Best Current Practice) perhaps, but who will do it? We'd also need to reach out to @greghudson (MIT) and @SteveSyfuhs (MSFT) to get their take on this.

[riastradh]

Should this have a CVE assigned and mitigation instructions for Kerberos administrators? (Am I the first person to have thought of this? Surely not?)

IMO it's of low urgency because there are much better amplification attacks out there. But I could be convinced otherwise.

Yes, I agree it's low urgency. Seems that the DNS world isn't as far as I thought in reducing UDP amplifiers, and that's much more widely deployed and a bigger amplifier (e.g., 48-byte ANY query to a.root-servers.net. for the root zone gives a 1232-byte reply).

But I've already been in contact with at least one admin who was confused about how to mitigate it and hoping for clearer instructions. Even if it takes some time to fix it properly, by clamping all responses to be no larger than the requests, it would be nice if there were clear suggestions for how to reliably avoid running a KDC as a UDP amplifier on the open internet.

Here's an attempt:

1. If you have _kerberos._tcp.EXAMPLE.COM SRV records, and/or if clients haven't been explicitly configured to use only UDP, you can set [kdc] ports = kerberos/tcp in your KDC config file (first of /var/heimdal/kdc.conf or /etc/krb5.conf by default). Then Heimdal will never reply to UDP queries, but unmodified clients will continue to work with TCP.
2. If you have _kerberos._udp.EXAMPLE.COM SRV records, you can delete them. Then clients won't try unnecessary UDP requests which won't work anyway.

Maybe it is better to suggest [kdc] max-kdc-datagram-reply-length = N, but it's not documented at the moment and I'm not clear enough on the tradeoffs between (a) making this small enough to prevent much amplification, and (b) making this large enough that it doesn't just add more round-trip times to the whole protocol, to suggest a specific value of N.

A bigger problem is that this wouldn't be specific to Heimdal -- this is a problem with an Internet protocol, so the next step would be to seek to publish an update to RFC 4120, or a BCP (Best Current Practice) perhaps, but who will do it? We'd also need to reach out to @greghudson (MIT) and @SteveSyfuhs (MSFT) to get their take on this.

Yes, this appears to apply more widely than to just Heimdal. I tested with what I presume is an mit-krb5 KDC and got a 289-byte packet in reply to a 197-byte request from (Heimdal) kinit. With another presumably mit-krb5 KDC, I got a 506-byte packet in reply to a 195-byte request.

I filed here because I figured that it must be a known issue and that Heimdal probably has some underdocumented way to mitigate it.

[SteveSyfuhs]

We (Windows) considered this kind of attack a low severity issue. Amplification attacks in corporate networks just aren't a thing, and on top of that the amount of requests required to saturate a standard Gbit LAN is pretty high. Network ops folks would catch that relatively quickly. It's just not the same environment as the open internet.

…

________________________________ From: riastradh ***@***.***> Sent: Monday, January 15, 2024 8:17:24 AM To: heimdal/heimdal ***@***.***> Cc: Mention ***@***.***> Subject: Re: [heimdal/heimdal] UDP amplification? (Issue #1216) Should this have a CVE assigned and mitigation instructions for Kerberos administrators? (Am I the first person to have thought of this? Surely not?) IMO it's of low urgency because there are much better amplification attacks out there. But I could be convinced otherwise. Yes, I agree it's low urgency. Seems that the DNS world isn't as far as I thought in reducing UDP amplifiers, and that's much more widely deployed and a bigger amplifier (e.g., 48-byte ANY query to a.root-servers.net. for the root zone gives a 1232-byte reply). But I've already been in contact with at least one admin who was confused about how to mitigate it and hoping for clearer instructions. Even if it takes some time to fix it properly, by clamping all responses to be no larger than the requests, it would be nice if there were clear suggestions for how to reliably avoid running a KDC as a UDP amplifier on the open internet. Here's an attempt: 1. If you have _kerberos._tcp.EXAMPLE.COM SRV records, and/or if clients haven't been explicitly configured to use only UDP, you can set [kdc] ports = kerberos/tcp in your KDC config file (first of /var/heimdal/kdc.conf or /etc/krb5.conf by default). Then Heimdal will never reply to UDP queries, but unmodified clients will continue to work with TCP. 2. If you have _kerberos._udp.EXAMPLE.COM SRV records, you can delete them. Then clients won't try unnecessary UDP requests which won't work anyway. Maybe it is better to suggest [kdc] max-kdc-datagram-reply-length = N, but it's not documented at the moment and I'm not clear enough on the tradeoffs between (a) making this small enough to prevent much amplification, and (b) making this large enough that it doesn't just add more round-trip times to the whole protocol, to suggest a specific value of N. A bigger problem is that this wouldn't be specific to Heimdal -- this is a problem with an Internet protocol, so the next step would be to seek to publish an update to RFC 4120, or a BCP (Best Current Practice) perhaps, but who will do it? We'd also need to reach out to @greghudson<https://github.com/greghudson> (MIT) and @SteveSyfuhs<https://github.com/SteveSyfuhs> (MSFT) to get their take on this. Yes, this appears to apply more widely than to just Heimdal. I tested with what I presume is an mit-krb5 KDC and got a 289-byte packet in reply to a 197-byte request from (Heimdal) kinit. With another presumably mit-krb5 KDC, I got a 506-byte packet in reply to a 195-byte request. I filed here because I figured that it must be a known issue and that Heimdal probably has some underdocumented way to mitigate it. — Reply to this email directly, view it on GitHub<#1216 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAJHTYPYIR7THQVA2DEI3XTYOVJBJBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFUYZTCOJTGMZYFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDANZTGUZTIMRTGOTXI4TJM5TWK4VGMNZGKYLUMU>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[greghudson]

It looks like MIT krb5 mitigated an amplification attack via kpasswd in 2013 with an issued CVE (CVE-2002-2443). I didn't find any discussion of amplification via the KDC.

I'd be inclined to treat this as low-priority and best addressed through documentation.

[nicowilliams]

Perhaps the best fix is to a) not advertise UDP in DNS (nor local krb5.confs) and b) not serve Kerberos on UDP. I.e., this is just a docs fix.

[riastradh]

Perhaps the best fix is to a) not advertise UDP in DNS (nor local krb5.confs)

Well, what clients try first is just a matter of usability and performance, and, if TCP has been disabled, compatibility. Changing normal client behaviour won't affect the real issue, which is how the KDC responds when provoked.

and b) not serve Kerberos on UDP. I.e., this is just a docs fix.

Sure, that's reasonable. I think it would be best if the KDC had some simple reliable default-enabled mechanism to avoid returning UDP replies that are longer than the requests that provoked them. But at least clear suggestions about how to prevent UDP replies would be good (which was the line of thought that prompted me to file #1223).

[nicowilliams]

Perhaps the best fix is to a) not advertise UDP in DNS (nor local krb5.confs)

Well, what clients try first is just a matter of usability and performance, and, if TCP has been disabled, compatibility. Changing normal client behaviour won't affect the real issue, which is how the KDC responds when provoked.

Not really. Clients generally try UDP first when it's advertised, thus not advertising it is a way to get clients not to try UDP. Naturally that's not enough IF the KDC nonetheless would respond to requests sent via UDP, thus the and that followed this. To disable UDP on the KDC but continue advertising it would only serve to cause annoying timeouts for clients.

and b) not serve Kerberos on UDP. I.e., this is just a docs fix.

Sure, that's reasonable. I think it would be best if the KDC had some simple reliable default-enabled mechanism to avoid returning UDP replies that are longer than the requests that provoked them. But at least clear suggestions about how to prevent UDP replies would be good (which was the line of thought that prompted me to file #1223).

I've written such a patch, but the best fix is no code, because adding code risks adding bugs and adds to the future maintenance burden. That said, it will be a lot easier to get this "problem" (if it really is one) to go away if we patch it than if we tell operators how to configure it away. It's just not clear to me that this really is a serious problem.

[nicowilliams]

@SteveSyfuhs does the Windows client accept non-FAST KRB-ERROR w/ error-code set to KRB_ERR_RESPONSE_TOO_BIG when the request sent had been a FAST request?

And does the AD KDC respond with non-FAST KRB-ERROR w/ error-code set to KRB_ERR_RESPONSE_TOO_BIG when the response would be too big, or does it send the KRB_ERR_RESPONSE_TOO_BIG in an inner KRB-ERROR?

[SteveSyfuhs]

I think we just set the standard error with TOO-BIG. I don't think we can guarantee a FAST error will fit in a UDP message.

…

________________________________ From: Nico Williams ***@***.***> Sent: Tuesday, January 16, 2024 8:08:11 AM To: heimdal/heimdal ***@***.***> Cc: Mention ***@***.***>; Comment ***@***.***> Subject: Re: [heimdal/heimdal] UDP amplification? (Issue #1216) @SteveSyfuhs<https://github.com/SteveSyfuhs> does the Windows client accept non-FAST KRB-ERROR w/ error-code set to KRB_ERR_RESPONSE_TOO_BIG when the request sent had been a FAST request? And does the AD KDC respond with non-FAST KRB-ERROR w/ error-code set to KRB_ERR_RESPONSE_TOO_BIG when the response would be too big, or does it send the KRB_ERR_RESPONSE_TOO_BIG in an inner KRB-ERROR? — Reply to this email directly, view it on GitHub<#1216 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAJHTYNZP2UZRGOPL6NFYF3YO2QWZBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFUYZTCOJTGMZYFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDANZTGUZTIMRTGOTXI4TJM5TWK4VGMNZGKYLUMU>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nicowilliams]

I think we just set the standard error with TOO-BIG. I don't think we can guarantee a FAST error will fit in a UDP message.

Thanks. I assume that means in the outer KRB-ERROR (and no inner one). RFC 6113 kinda says to do just that too, and MIT does that. So I'll make sure Heimdal does too. MIT and Heimdal only correctly process the TOO-BIG error when it's on the outer KRB-ERROR, FYI.