HostProcess containers for macOS

[adamrehn]

The upcoming HostProcess container type for Windows stretches the traditional definition of what constitutes a "container" in many people's minds, conceptually boiling down to running non-sandboxed processes in a filesystem volume whose contents are sourced from an OCI container image, along with access to the host filesystem to facilitate administrative tasks. In addition to the administrative use cases discussed in KEP-1981, HostProcess containers will also provide an avenue for easily building Windows container images inside Kubernetes clusters once support for Windows containers lands in BuildKit, since access to the host network and filesystem facilitates direct access to the RPC interfaces for the Windows Host Compute Service (HCS) and Host Compute Networking (HCN) Service used by vmcompute.dll/computestorage.dll/computenetwork.dll and hcsshim atop them. The bottom line is this: even without other sandboxing features, filesystem-only containers still deliver value to users and to the container tooling ecosystem.

With that context in mind, I posit that there would be value in supporting HostProcess-style containers on macOS which are built atop chroot jails. Although true container support for macOS is currently precluded by the lack of necessary isolation primitives in the XNU kernel, there is a clear demand for some level of container support amongst macOS and iOS developers, as evidenced by the existence of open source projects such as Docker-OSX and proprietary products such as MacStadium Orka. These existing solutions typically rely on wrapping a macOS virtual machine in a Linux container in order to achieve full sandboxing, but I believe that a lightweight alternative based purely on filesystem isolation would be extremely attractive to developers who are simply looking to orchestrate CI/CD workloads on systems under their exclusive control and who are not concerned about strict security boundaries between containers and the host environment. This would also facilitate denser bin packing of workloads by sidestepping the need for VMs and the accompanying limitations around VM instances in the macOS EULA.

I can see at least two key components that would need to be implemented in order to support chroot jail containers under macOS:

• A shim implementing the containerd V2 Runtime API which unpacks OCI images into chroot jails and starts processes within them; and
• A snapshotter implementation for macOS volumes within containerd itself.

I'm by no means an expert in the architecture of either containerd or macOS itself, and so I'm seeking feedback from the community to validate this idea. Does anyone agree that this idea merits investigation and implementation effort? Would anyone be interested in assisting with said effort? Are there any details that I've overlooked, in either the premise or suggested starting points for implementation?

[TBBle]

Definitely an interesting idea.

Poking around, macOS used to support union mounts, but maybe does not now. It's possible that firmlinks can have the same effect (I'm not sure how write-through works there, but I suspect it's not quite the same as we'd want anyway), and it's also possible that these are only for OS usage.

There's also a possibility of using fused-based overlayfs or unionfs implementations via macFuse or similar.

For contrast, the Windows snapshotter works with distinct "read-only" and "read-write" layer formats on disk, where the read-only layers are (ignoring registry hives etc) just the on-disk filesystem, using hardlinks to reference the parent layer, and then a read-write layer is a "differencing VHDX" using some read-only layer stack as its base, all managed via an installable filesystem minidriver (WCIFS).

The maximum-effort version would be to actually implement overlayfs (or the wcifs model but using DMG files, I guess?) on OS X as a real KEXT. I imagine that will require real investment and signing and licenses from Apple etc.

Although from that link

Note: OS X does not support stacking in its file-system design.

so maybe an overlayfs KEXT is a non-starter.

Without one of those working, the snapshotter ends up being rather expensive in disk and time, as it would need to be extracting a copy of the source layers into the chroot, and then again in order to diff against them when committing the snapshot.

[TBBle]

Yeah, that could work. It would still (reading only that help text...) require fully extracting each layer of a base image into a single image file, and then mounting it with a shadow file for read-write; to then commit the new layer, I don't immediately think of a more-efficient method than a naive walking-diff across both images, and then generating a new image including the new layer.

If there was an option to 'pack down' a shadow file into the image, then that would make the 'commit' more efficient.

Disk consumption grows because each lower layer is repeated in every image; this is the issue WCIFS avoids using hard-links between the layers on disk, so that only the read-write image is in a layer, i.e. it's the -shadow option here, but against a directory on disk instead of an image.

[adamrehn]

It does look like shadow files can be merged with their base images to create new image files, but I agree that we'd probably want to unpack them to filesystem directories if possible rather than using disk images for every layer.

Unfortunately it looks like APFS does not support directory hard links, as per this FAQ page:

Does Apple File System support directory hard links?

Directory hard links are not supported by Apple File System. All directory hard links are converted to symbolic links or aliases when you convert from HFS+ to APFS volume formats on macOS.

APFS does apparently support a copy-on-write variant of a hardlink called a clone, although presumably that's limited to individual files rather than directories. EDIT: reading through the comments on that blog post confirms that APFS clones do not support directories.

[adamrehn]

Looking into additional alternatives to bind-mounting has yielded some options:

• Use a local NFS mount to mount directories. Probably not ideal from a performance standpoint, but requires no additional software.
• The mount_nullfs tool, which needs both SIP and AMFI to be disabled in order to work but would presumably be more performant than an NFS mount.

[adamrehn]

Update: thanks to the excellent work by @thehajime in #5935, we now have a mounting and snapshotting implementation backed by sparse bundle disk images that we can use! With this fundamental roadblock addressed, I can start working on a prototype runtime shim implementation that runs child processes on the host system, and then add chroot jail functionality later once we've determined how we're going to handle the whole base image situation.

[christian-korneck]

MacOS seems to have a new API for user mode file systems:
https://eclecticlight.co/2024/06/26/how-file-systems-can-change-in-sequoia-with-fskit/

[webbertakken]

Thanks for the great writeup @adamrehn.

These existing solutions typically rely on wrapping a macOS virtual machine in a Linux container in order to achieve full sandboxing, but I believe that a lightweight alternative based purely on filesystem isolation would be extremely attractive to developers who are simply looking to orchestrate CI/CD workloads on systems under their exclusive control and who are not concerned about strict security boundaries between containers and the host environment.

As GameCI we can confirm this would be extremely useful to have, and I would say that this statement is spot on. We also found a question about a docker image running MacOS on StackOverflow that seems to have stagnated by the lack of solutions.

I would agree that offering an initial solution, even without mentioned sandboxing features, would be a very good first step. Ideally it would end up creating some momentum.

[awakecoding]

Please implement this! Even without complete isolation, such macOS containers would be a zillion times better than having no macOS containers at all. I currently use Windows and Linux containers for all of my CI builds, but have to resort to updating and installing build tools directly on Mac Mini builds hosts. What you suggest would at least solve this problem by moving most if not all of the build tooling inside macOS containers for repeatable build environments.

[TBBle]

An issue that just occurred to me when thinking about base layers is licensing. How much of macOS needs to actually be duplicated inside the container for the functionality we need? I haven't played around with chroots on macOS, but if we need to copy any of the base OS into the container, then such a layer would be non-redistributable. I imagine locally there wouldn't be a license problem (or chroots would suffer the same problem) but you wouldn't be able to push such layers to Docker Hub, for example.

MS has this problem, which is why they provide the base layers themselves, and we have the 'foreign' layer system in OCI to support this, but unless Apple were providing the base layers, each user's locally-generated base layer is likely to have a different hash, and hence the foreign layer system doesn't help that much.

Depending on what OS components are needed, perhaps they could be sourced from Darwin, but in my mind this would be like trying to create a Windows base container layer from-scratch containing ReactOS, and is likely to land somewhere between difficult and impossible.

[adamrehn]

This was one of the first concerns I had about attempting to implement macOS containers without Apple's assistance, and it is likely to be a source of awkwardness until such time that community adoption of macOS containers convinces Apple that there is value in supporting container technology. I can see a few potential options:

• We don't use base layers at all, but instead mount the core OS components from the host system at runtime. This would almost certainly have compatibility implications across different versions of macOS, but then again we won't know until we've run some experiments whether the OS version needs to be tied to the kernel version anyway (in much the same manner as Windows containers today.) In this approach, we might need to create base layers for each supported version of macOS which contain nothing other than a version string as a file or metadata (to ensure unique hashes) and then use those as a marker to constrain whether a given image can run on a given host system (i.e. the image specified the host's macOS version in its initial FROM Dockerfile directive.)

• Users generate their own base layers (potentially through the use of an automated tool or script that the community provides) and host them privately for use within their own organisations. My reading of the macOS Big Sur EULA is that this is prohibited by Section 2J ("Other Use Restrictions") unless organisations have purchased a volume license for macOS as set out in Section 2C ("Volume or Maintenance License"). Organisations who do have a volume license in place should be able to make use of this option, but it's unclear whether they would need a macOS license for every individual container.

• We attempt to make use of the provisions set out in the Apple Security Bounty program terms to work around the distribution restrictions of the macOS EULA, which is how the Docker-OSX project justifies its legality and the public availability of its container images on Docker Hub that contain macOS VM images. I don't know whether this option could be pursued in good faith though, given that most users of macOS containers are unlikely to be participants in the Apple Security Bounty program.

[christian-korneck]

• Users generate their own base layers (potentially through the use of an automated tool or script that the community provides) and host them privately for use within their own organisations. My reading of the macOS Big Sur EULA is that this is prohibited by Section 2J ("Other Use Restrictions")

iiuc 2J disallows use on non-Apple branded computers, use by multiple users at a time and over a network. Would building the base image on the computer that it is going to be used on be an option? (i.e. you would need to run that community provided tool once on every Mac where you want to use the base image). Think typical local developer workstation? (original Mac, one user at a time, no sharing of the base image over the network).

[adamrehn]

@christian-korneck that would get around it, but at that point is there any real benefit over simply sourcing the base files automatically from the host system when each container starts?

[christian-korneck]

@adamrehn but at that point is there any real benefit over simply sourcing the base files automatically from the host system when each container starts?

I think users could have benefits from both variants (and I personally would love to see both implemented).

• sourcing base files from the host: This would give the user a vanilla/clean environment of the OS version that you're running on the host. This would be very lightweight in terms of additional disk usage. It would also allow you to push images to a public registry and they work immediately if you run a compatible OS version on the host.
• creating base images with a community provided tool: This would allow the user to create multiple base images for various macOS versions. This allows to build/test software in all supported MacOS versions (including betas). This is already possible (and allowed for up to 2 instances per host by the EULA afaik) with VMs. But these VMs come with a massive memory consumption overhead, slow startup times, etc. To support the creation of base images for multiple OS versions the tool could either download/extract the macOS installer - or if that turns out to be impossible the user could still run the tool in temporary VMs if a base image of a specific macOS version is needed.

[slonopotamus]

WRT licensing:

1. There is an important difference from Docker-OSX is that we're not talking here about running macOS on non-Apple hardware.
2. Cirrus Labs publicly distributes macOS base images via GHCR and you can download them via Tart. I asked them what do they think about legality of such action.

[awakecoding]

Regarding the macOS base images and the issue of redistribution: in the case of Windows containers, if you look carefully at the image manifest you'll notice it points to a Microsoft CDN URL even on Docker Hub. The base layer itself is not redistributable but it is relatively easy to point to it, such that even if you pull from a foreign registry you still get it from Microsoft. At least that's how I understand it.

Because containers work better with immutable layers, I suggest the following: find a way to create the base layers with community tooling in a way that makes the process repeatable with an identical output

The idea would be to provide base container images as pure manifests for which the blobs are never distributed publicly. Unlike the Windows containers, we wouldn't be able to upload the blobs somewhere public. This being said, it would be relatively easy to re-generate the corresponding base image with exactly the same blob hash, and push it in a private container registry, or just in the local cache.

Docker-OSX and other projects have a relatively good way to bootstrap the macOS installation using the network recovery installer. I think it could be used as a good starting point, but I don't think the installation has ever been automated (Apple being Apple, there is no such thing as an unattended installation apparently). This would probably be the biggest blocker, but I guess it might be possible to make a program to OCR the screen contents and simulate a manual installation from script.

[adamrehn]

Regarding the macOS base images and the issue of redistribution: in the case of Windows containers, if you look carefully at the image manifest you'll notice it points to a Microsoft CDN URL even on Docker Hub. The base layer itself is not redistributable but it is relatively easy to point to it, such that even if you pull from a foreign registry you still get it from Microsoft. At least that's how I understand it.

Yep, that's totally correct. The mechanism they use for this is to mark the base layers as "foreign layers", which instructs container registries like Docker Hub to never cache those layers and clients to always pull them from the original source. (Although this behaviour can be overridden for air-gapped machines.)

Because containers work better with immutable layers, I suggest the following: find a way to create the base layers with community tooling in a way that makes the process repeatable with an identical output

This would certainly be ideal if possible, since it would most closely replicate the way that base images work for other platforms and also how macOS base images will hopefully work if Apple eventually decides to introduce official support for containers down the track. The main concern with this approach is actually a legal one rather than a technical one, since as I mentioned in my other comment, Section 2J of the macOS EULA prohibits making the software available over a network except in specific circumstances. I suppose users could always export the base layer as a tarball (e.g. using the docker save command) and then copy it between machines using physical storage devices such as USB sticks if they want to avoid running the generation step on every machine.

Docker-OSX and other projects have a relatively good way to bootstrap the macOS installation using the network recovery installer. I think it could be used as a good starting point, but I don't think the installation has ever been automated (Apple being Apple, there is no such thing as an unattended installation apparently). This would probably be the biggest blocker, but I guess it might be possible to make a program to OCR the screen contents and simulate a manual installation from script.

This would definitely be an interesting option to explore, since it would be far easier to produce identical results across different machines if the image generation tool uses an installer as its input rather than the loose files on the host filesystem. The ability to successfully automate the process would be a pretty critical factor though.

[adamrehn]

Thanks to this thread on Ask Different, I've discovered the sailor project by Emile Heitor. Evidently Emile had a similar idea back in 2015, albeit in the form of a standalone tool with support for NetBSD, macOS and RHEL. This should certainly prove a useful source of information for investigation!

[adamrehn]

The same thread also leads to this article by Karl Tarvas about the macOS sandbox-exec tool, which is another fascinating avenue for investigation.

[TBBle]

I realised just now that if we invert the question, kata-containers (containers in lightweight VMs, like Hyper-V container isolation on Windows) would probably work on macOS with some fiddling, but it seems it never got any traction: kata-containers/runtime#379 which called out layered storage and "netlink" (I assume for network setup) as cost/risk points. (That issue was closed because they are now in a new repo for "version 2", and there's no corresponding macOS support request there)

Google also gave me https://fosdem.org/2021/schedule/event/containers_darwin_containerd/ which was about running Linux containers on macOS using "Linux Kernel Library (LKL), a library version of Linux kernel". That in turns points a containerd PR #4526 and also https://github.com/ukontainer/runu, which is probably not actually relevant for this, but might give a sense of what else is going on in containers ecosystem for Darwin.

All that said, a comment notes: https://github.com/containerd/containerd/pull/4526/files#r604543921 about runu

currently the runtime support Linux/ELF and macOS/Mach-O (GOOS=darwin) binaries.

so maybe this is something already on the right track.

[adamrehn]

The thing that confuses me about runu is that its repository describes it as an "OCI runtime for frankenlibc unikernel", which is presumably referring to https://github.com/justincormack/frankenlibc. Since frankenlibc does not list Darwin as a supported platform, does that mean that runu instead relies on its FreeBSD support? (Or is frankenlibc only involved when running Linux binaries?) It's not entirely clear to me what kind of macOS binaries runu is actually capable of running, but I get the feeling that broad compatibility is rather unlikely.

[leavez]

After reading the comments on the PR, I'm confused too. But anyway, it's not seem to run a standard macOS app inside the runu.

[adamrehn]

I've created some social links that people can share if they would like to help boost the visibility of this initiative:

• https://macoscontainers.org/
• https://twitter.com/macoscontainers

[slonopotamus]

There is #4526 that adds some kind of darwin support to containerd, though I am not sure from its description whether it is what current discussion is about.

[TBBle]

That's what I was looking at above, but I didn't dive too far into it at the time.

[slonopotamus]

According to #4526 (comment), that thing is for running Linux containers.

[awakecoding]

Has there been any progress on macOS containers since the last reply in this thread? I was reading on Darling the other day, and was surprised to learn how it was using an overlay filesystem very similar to Docker. There's WSL support if one can recompile the kernel to enable a specific feature, so I suspect it probably cannot be run inside a Linux container.

My thinking is that we could still probably borrow parts from Darling to try and build a Linux container that can mount parts of the macOS host filesystem as read-only. Darling can run the XCode build tools from Linux, so if could manage to call native macOS build tools from a Linux container, it would already provide a lot of the desired isolation for containers, even it looks like some sort of weird hybrid.

[awakecoding]

Here's an update on my darling-based experiment: even if we were to make it work, it would provide limited compatibility as opposed to real macOS containers. Just like wine for Windows, darling has to re-implement APIs for all frameworks and dylibs normally provided by the system. I went quite far and managed to install Xcode 12.5.1 only to realize that it is missing a new dylib that wasn't stubbed and reimplemented yet. Older versions of Xcode were apparently known to work, but I'd rather be able to stick to the latest. I would still recommend taking a closer look at the darling project for inspiration since the architecture is Docker-like with an overlay fileystem adapted specifically for macOS needs. However, I would definitely consider the hybrid approach to be limited, and would encourage going forward only with a full native macOS container approach instead.

[adamrehn]

@awakecoding thanks for investigating the feasibility of this route! That's a shame that the hybrid approach is so limited, but as you noted, hopefully some of the learnings from the Darling project can be of use or inspiration.

[awakecoding]

@awakecoding thanks for investigating the feasibility of this route! That's a shame that the hybrid approach is so limited, but as you noted, hopefully some of the learnings from the Darling project can be of use or inspiration.

I was recently approached by someone from MacStadium who saw my tweets. He's in charge of their FOSS relation program, and can offer some resources for free. I honestly don't have time to spend on this, and I would really like to redirect him to someone from this project to help.

Is anyone interested? I'm trying to get MacStadium interested in macOS containers to complement their macOS VM product.

[adamrehn]

@awakecoding please send him my way! (Either [email protected] or [email protected] would work.) I'd been planning on reaching out to MacStadium about this once I had a minimal prototype functioning (since I figure working code speaks louder than anything else), but if they're willing to help out at this early stage then that would be awesome!

[awakecoding]

@awakecoding please send him my way! (Either [email protected] or [email protected] would work.) I'd been planning on reaching out to MacStadium about this once I had a minimal prototype functioning (since I figure working code speaks louder than anything else), but if they're willing to help out at this early stage then that would be awesome!

email sent :) I don't have much time to work on this myself except talk to people and get them interested. Let's do this!

[slonopotamus]

Current state. There is proof-of-concept snapshotter for Darwin, accompanied with a PR to containerd.

[slonopotamus]

Current status of rund: we have working DNS inside container, we have bind-mounts, we have images published to GHCR, we have proper tty/pty support, we have containerd fork with minor macOS-specific changes. I definitely cut several edges in shim implementation, but do not see currently any side effects from that.

Next steps are:

• Fix buildkit/Moby to build and run on macOS
• Attempt to install xcode inside the container (this isn't dependent on previous step, can be done directly in containerd)

[slonopotamus]

More fun! Buildkit builds its first macOS image via Dockerfile. Requires some lightweight patching: moby/buildkit#4059

[slonopotamus]

WRT Xcode. I'm trying to reuse this. The problem is that xcodes crashes. The first issue is that macOS was killing it. Solved by codesign --remove-signature. But it crashes again, now because of dereference of some null pointer inside. Don't have working debugger inside container yet. So, it looks like I'll have to put debug prints here and there, rebuild xcodes on host and try rerunning inside chroot.

The Dockerfile that is eventually supposed to work is something like:

FROM ghcr.io/macoscontainers/macos-jail/ventura:latest
RUN curl -L https://github.com/XcodesOrg/xcodes/releases/download/1.4.1/xcodes.zip -o /xcodes.zip && \
    unzip /xcodes.zip -d /usr/bin && \
    rm /xcodes.zip && \
    codesign --remove-signature /usr/bin/xcodes
# Keep download a separate step for now to allow quick iterations
RUN curl -L https://storage.googleapis.com/xcodes-cache/Xcode_14.3.1.xip -o /xcode.xip
RUN xcodes install 14.3.1 --experimental-unxip --path /xcode.xip && \
    xcodes select 14.3.1 && \
    xcodebuild -downloadAllPlatforms

[slonopotamus]

Well, we're almost there...

[slonopotamus]

I think we're ready for a v0.0.1-prealpha release.

[zimme]

So this thought is probably more geared towards running linux "containers" on macos than running macos containers.

Perhaps combining runq with qemu's Hypervisor.Framework support could result in a way to containers on macos. I guess it wouldn't be much different from the current way most solutions are doing it, as vms are still involved.

Another alternative could be to write a OCI runtime for Virtualizaion.Framework and I guess that could support running macos "containers". But I'm not sure that would play nice with the MacOS EULA VM restriction.

[cguentherTUChemnitz]

Is here any update to potentially see native linux container support on macos without an virtual machine handling the container runtime?

[kbaegis]

The current VM based approach is simply awful. Simple things like running a dhcp/pxe server are fundamentally broken due to networking limitations.

[maikschulze]

FYI, I just discovered Cirrus Labs Tart which aims for similar goals: https://github.com/cirruslabs/tart .

[AkihiroSuda]

tart is a VM, not so similar.

[tonidy]

Based on this comment #4526 (comment). The current blocker is this PR #5935 still not merged yet. I hope this PR will be merged soon 😄 . Also @AkihiroSuda already asked @mxpv to review that PR. Hopefully @mxpv or another containerd contributor will review it

[slonopotamus]

#5935 is not a blocker. containerd has native snapshotter that is used on macOS by default. While it is awfully ineffective (it just copies the whole base image), it does its job: container gets an isolated copy of base image that it can work in.

[slonopotamus]

Actually... native snapshotted is not THAT bad on macOS because macOS supports clonefile syscall that is used in containerd via continuity.

[Feyko]

What is the current state of this? Is there information as to what the next step is or what is blocking?
Would really love to see this get some traction 😄, CI/CD for mac is currently more than awful

[slonopotamus]

Containerd & buildkit are working (well, to the extent I've tested them). Moby is pain in the ass because it uses ancient BuikdKit. I will possibly wait until moby/moby#45966 is merged. There are too many code incompatibilities totally unrelated to macOS. Alternatively, will try rebasing my patches on top of older containerd/buildkit versions.

There is no user-friendly installer yet. Too early given than the whole containerd+buildkit+moby set is not functional yet.

See https://github.com/macOScontainers/rund#rund for instructions on how to play with current state.

[slonopotamus]

Okay, let's go.

Today, on 2023-09-25, I proudly announce initial 0.0.1 release of macOS containers.

Supported features:

• Prebuilt image for macOS Ventura (arm64+amd64)
• docker build with and without BuildKit
• docker run
• Host network
• DNS inside containers
• Isolation from host OS via chroot
• Installation via Homebrew
• Bind mounts

Bugreports are welcome and expected.