Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: require Linux 3.17 kernel for Go 1.24 #67001

Open
rsc opened this issue Apr 23, 2024 · 25 comments
Open

all: require Linux 3.17 kernel for Go 1.24 #67001

rsc opened this issue Apr 23, 2024 · 25 comments

Comments

@rsc
Copy link
Contributor

rsc commented Apr 23, 2024

In 2021 in #45964 we moved the minimum Linux kernel version to 2.6.32.

There is a good summary from @baryluk about what we use from newer kernels at #60792 (comment).

@mdempsky points out in #60792 (comment) that Rust and glibc decided on a minimum kernel version of 3.2 last year.

Adopting 3.2 as Go's minimum seems reasonable given the rest of the Linux ecosystem doing the same.

I propose we announce this in Go 1.23 and make the change in Go 1.24. That will still be a year or so behind glibc and Rust, and the announcement will create time for people to speak up if they are affected and cannot upgrade.

Really everyone should be on newer kernels for security updates anyway.

@robpike
Copy link
Contributor

robpike commented Apr 23, 2024

I'm ambivalent about this trend in general. For Go, it also pushes a bit against the compatibility promise.

I've been burned multiple times by manufacturers dropping support for old hardware, requiring expensive upgrade purchases that provide no return benefit to me. I still run a 32-bit x86 Mac in the lab because without it I cannot control an important piece of kit that can only be interfaced to an old mac OS version. I can't even connect it to the network for fear some update will render my equipment non-working.

While I appreciate the implied - if uncertain - value of "newer is safer, newer is better", what is the actual cost to the project of letting Go binaries run on old OSes? That seems a better way to evaluate this proposal.

@rsc
Copy link
Contributor Author

rsc commented Apr 23, 2024

The cost to the project in general is continued maintenance and testing of code paths working around the lack of features from the (relatively) newer kernels. We don't do a great job of testing Go on these old kernels as it is. It probably only kind of works.

I should have mentioned that Linux 3.2 was released 12 years ago. It is not bleeding edge by any means. According to https://en.wikipedia.org/wiki/Linux_kernel_version_history, the final 2.6.x kernel was 2.6.32.71 in 2016.

We are talking about dropping support for kernels that are very old.

@rsc
Copy link
Contributor Author

rsc commented Apr 24, 2024

Talking to @FiloSottile about #66821, it sounds like moving to 3.17 (~2014) as the minimum would let us avoid needing to open /dev/urandom ever, because 3.17 added getrandom(2). That would be a better choice than 3.2.

@rsc rsc changed the title proposal: all: require Linux 3.2 kernel for Go 1.24 proposal: all: require Linux 3.17 kernel for Go 1.24 Apr 24, 2024
@rsc
Copy link
Contributor Author

rsc commented Apr 24, 2024

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@tianon
Copy link
Contributor

tianon commented Apr 24, 2024

In the way of additional data points, a few popular and older distribution versions and their associated kernel versions (all of which IMO point in favor of this proposal at 3.2 and even at 3.17 👍):

  • Debian 10 (Buster): 4.19
  • Ubuntu 20.04 (focal): 5.4 (you have to go all the way back to 14.04 to get 3.13 / incompatible)
  • CentOS 7: 3.10 (but, 7 is fully EOL on 2024-06-30 in two short months)
  • CentOS 8: 4.18 (EOL 2021-12-31)
  • Fedora 38: 6.2
  • OpenWrt 22.03: 5.10 (EOL 2024-04-11)

It's a bit hard to browse, but even more aggregated data at https://repology.org/project/linux/versions

(I dug up this information to convince myself this was sane and figured it was worth sharing 👍 ❤️)

Edit: and see #60792 (comment) for a much longer form of similar data with many more details! 😄 ❤️

@ulikunitz
Copy link
Contributor

ulikunitz commented Apr 24, 2024

Red Hat Enterprise Linux 7 is on kernel 3.10 and it's maintenance ends 30 June 2024. Red Hat offers Extended Lifecycle Support until 30 June 2028. Gcloud CLI supports RHEL7 and it includes Golang binaries. You may want to clarify how long Gcloud CLI plans to support RHEL7.

@empire
Copy link
Contributor

empire commented Apr 24, 2024

The RHEL 7.9 OS has a kernel version 3.10.0 and is supported until 2028-06. Changing the minimum version to 3.17 may impact services running on this OS.

https://distrowatch.com/table.php?distribution=redhat&pkglist=true&version=rhel-7.9#pkglist

@AGWA
Copy link

AGWA commented Apr 24, 2024

The RHEL 7 kernel had getrandom backported in 2017, per https://access.redhat.com/errata/RHSA-2017:1842:

BZ - 1330000 - kernel: Backport getrandom system call

So in practice Go programs would continue to work on RHEL 7 if the only 3.10+ feature used is getrandom.

@rsc
Copy link
Contributor Author

rsc commented Apr 25, 2024

Thanks for the added details. I think we can say 3.10+getrandom is fine too.

@rsc
Copy link
Contributor Author

rsc commented May 8, 2024

Have all remaining concerns about this proposal been addressed?

The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.

@baryluk
Copy link

baryluk commented May 12, 2024

I think 3.17 (or 3.10+getrandom) is doable and reasonable.

I had a minor concern with Debian 8 codename Jessie, released April 2015, which is still in extended long term support by freexian (paid support, until mid 2025, possibly mid 2026 at most if customers pay for it), because it had 3.16.7.

But even during normal LTS there were backports available with kernel 4.9, and now freexian recommends their backports (due to security updates mostly) maintained by them of kernel 4.19 and 5.10. And anybody still running so old system (initial release 2015-04-26, official EOL 2020-07; and now we are at Debian 12, so many many major releases behind), can just stick to old code, or continue using older Go to maintain any software they run.

So even Debian 8 with ELTS would be fine with this 3.17+ requirement in practice. (Debian 9 codename Stretch, was released with 4.9.30, on 2017-06-18, already out of normal LTS, and ELTS from freexian until 2027).

So, good to go with 3.17

@rsc
Copy link
Contributor Author

rsc commented May 15, 2024

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.

@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

The proposal is to require Linux 3.17 starting in Go 1.24,
with the exception that Linux 3.10 patched with an added getrandom(2) system call is also permitted.

@rsc rsc changed the title proposal: all: require Linux 3.17 kernel for Go 1.24 all: require Linux 3.17 kernel for Go 1.24 May 23, 2024
@rsc rsc modified the milestones: Proposal, Backlog May 23, 2024
@dmitshur dmitshur modified the milestones: Backlog, Go1.24 May 23, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/588216 mentions this issue: internal/poll,syscall: drop the fallback for accept4 on linux/arm

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/589015 mentions this issue: doc/next: preannounce a new Linux kernel version requirement

gopherbot pushed a commit that referenced this issue May 30, 2024
This change adds a preannouncement to the release notes. Go 1.24
will require the Linux kernel to be at version 3.17 or later,
with an exception that systems running 3.10 or later will
continue to be supported if the kernel has been patched to
support the getrandom system call.

For #65614
For #67001

Change-Id: I61a3838b9ce4f48eb9f94830aa03372d589afdcd
Reviewed-on: https://go-review.googlesource.com/c/go/+/589015
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
Reviewed-by: Dmitri Shuralyov <[email protected]>
@FiloSottile
Copy link
Contributor

I still support this, but in the interest of being aware of expected breakage, Synology apparently ships Linux 3.10 without getrandom() in recent (current?) firmwares. The "good" news is that apparently even glibc gave up on that, so a C application invoking g_random_int will fatal out, like we plan to do in crypto/rand (#66821). See photoprism/photoprism#4339 (comment) where a Go application was crashing out due to a cgo glibc invocation. They worked around it, but they won't be supported by Go 1.24.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/602495 mentions this issue: crypto/rand: remove /dev/urandom fallback and simplify package structure

@ianlancetaylor
Copy link
Contributor

@bradfitz See the comment #67001 (comment). I seem to recall that you've mentioned Synology before.

@cagedmantis cagedmantis self-assigned this Aug 6, 2024
gopherbot pushed a commit that referenced this issue Aug 7, 2024
For #67001

Change-Id: Ic0b91cd393046c53e41908e8dbbe2ab5fb6b8a74
Reviewed-on: https://go-review.googlesource.com/c/go/+/588216
Reviewed-by: Ian Lance Taylor <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: David Chase <[email protected]>
Reviewed-by: Tobias Klauser <[email protected]>
Auto-Submit: Tobias Klauser <[email protected]>
@bradfitz
Copy link
Contributor

bradfitz commented Aug 7, 2024

We have 10,329 7-day active users using Synology devices with kernels older than 3.17:

  • Linux 3.10.108 (7027 active users)
  • Linux 3.2.101 (1952 active users)
  • Linux 2.6.32.12 (484 active users)
  • Linux 3.2.40 (435 active users)
  • Linux 3.10.102 (431 active users)

And then we also have 6,026 other active Linux users NOT using Synology running kernels older than 3.17.

So 16,355 active users who would be impacted by this change.

@FiloSottile
Copy link
Contributor

FiloSottile commented Aug 9, 2024 via email

@rsc
Copy link
Contributor Author

rsc commented Aug 9, 2024

If we need to drop back to 3.2 to match Rust and glibc, with the understanding that it will mean keeping some crypto code that opens /dev/urandom for a while longer (and crashes if that fails), then I'm okay with that if others are. It is a very tiny amount of code, and getrandom appears to be the only thing we were hoping for after 3.2.

I don't see too much point to dropping behind what Rust and glibc now require, though.

@bradfitz
Copy link
Contributor

bradfitz commented Aug 9, 2024

And then we also have 6,026 other active Linux users NOT using Synology running kernels older than 3.17.

How many of those are RHEL 7 and derivatives (which backported getrandom)?

Looks like ~4770 active users are RHEL/Centos with some variant of Linux 3.10.0 ...

Screenshot 2024-08-09 at 11 32 23 AM

@meling
Copy link

meling commented Aug 14, 2024

I wonder how many of these older versions are actually using recent versions of Go? Seems to me that if people don’t or can’t upgrade to newer kernels, maybe they aren’t keeping up to date with Go releases as well… I’m guessing the telemetry data won’t help here?

@tianon
Copy link
Contributor

tianon commented Aug 14, 2024

The challenge is that it isn't just a constraint of the system running Go, but on the systems running the Go programs, so the telemetry won't capture that sample set.

(I still think the change is fine, to be clear, just trying to help explain why it's Complicated and doesn't necessarily relate to systems using the Go toolchain directly very much.)

@ianlancetaylor
Copy link
Contributor

@meling For a case like Synology, this is users who want to run a new Go program on their Synology system, but it may not be straightforward for them to update that system to a newer kernel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Accepted
Development

No branches or pull requests