[Docker Desktop] Improve Mac File system performance

[nebuk89]

Update Feb 6, 2024 - Released as part of Docker Desktop 4.27 - https://www.docker.com/blog/announcing-synchronized-file-shares/

Update Nov 9, 2023 - As announced in June, Docker has acquired Mutagen IO, Inc.. We are hard at work integrating it into Docker Desktop and working to roll it out as part of a limited early access program.

Update: we are now looking at using GRPCFuse rather than mutagen as a simpler path for perf improvement.

Tell us about your request
Integrate the mutagen pluggin within Docker Desktop to provide users with a file caching option to improve performance on modern web frameworks like PHP Symphony

Which service(s) is this request for?
Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
File system performance is big issue on Mac,our goal improve web page refresh for web languages like PHP from 2.8 seconds to 0.2 seconds

Are you currently working around this issue?
N/A
Additional context
N/A
Attachments
docker/for-mac#77

[daveisfera]

These issue are still happening, so can they be added to this or new cards made to include them?
docker/for-mac#1592
docker/for-mac#1899
docker/for-mac#2417
docker/for-mac#3186

[leehambley]

I hope leaving a note here is not unwelcome to highlight a particular difficulty in booting Rails applications in Docker on macOS.

Between Rails hot code reloading, which has a semi-hard dependency on working filesystem events (the polling alternative can eat as much as 20% of system CPU) and Bootsnap, which is a caching loader which tries to speed up booting Rails by bypassing the parsing step (and caching compiled btyecode on disk) which means that booting a Rails app with bootsnap loaded writes as many file as it reads (sometimes), but other times (e,g warm start) has a very different read/write ratio.

None of the existing methods account for this very well, even the relatively new mount options to try and performance boost either reads or writes don't particularly help because of the relative symmetry under certain configurations.

Most places I know are still using NFS, with polling, and trying to tune the fsattr cache times to get a reasonable balance of developer experience (speed of code reloading) vs. performance (outright speed), but it's a tightrope.

[nebuk89]

hey @daveisfera we haven't got this quite out onto edge yet :) once we do we will move this over to developer/preview and update this ticket. Hopefully then people on those tickets should see some movement!

[mbrioski]

@nebuk89 could u clarify caching? Because at the moment the problem is the slow sync from host machine to docker containers

[Jean85]

@mauri-brioschi With PHP/Symfony apps sometimes the reverse happens: the app itself rewrites a bunch of files in its own var/cache dir, where it dumps a lot of stuff (container definitions, configuration...) and this slows a lot the machine. You could try to ignore that folder from the sync, but you would lose some IDE functionality that relies on that folder.

[mbrioski]

@Jean85 i know the problem and it happens more with "Frameworks" like Magento where there is "generated" code. Usually i exclude these directories so there's not need for sync.
Caching these directories really solve the problem? At the end if u flush the cache and u are synchronizing your volumes, your folder must be sync again.

[markshust]

Artifacts and caching directories like var/cache, generated, node_modules, vendor, etc. should basically persist on the container, however the issue that @Jean85 pointed out that you miss IDE functionality is definitely valid. As a developer you want assets like these to be generally available, and don't want to have to worry about this folder becoming "out of sync" with the host. I wrote a blog post about all of this at https://markshust.com/2018/12/30/docker-mac-filesystem-volume-mount-approach-performance/ which may be useful to others dealing with filesystem performance issues. Magento 2 seems to be an ideal use-case to test the folder sync issues.

Mutagen offers different modes https://mutagen.io/documentation/synchronization and ideally we should have access to all of these modes. I believe the "one way" is more performant than "two way" since it only needs to check the filesystem changes in one direction. We'd also need to control whether the host or container is alpha or beta. Consistency also probably isn't required for these artifacts directories, it just needs to be "generally there". Being able to control volume mounts on these different modes is crucial to achieving maximum performance.

[mbrioski]

What's about mounting NFS volumes? It is the only working solution i have found, i.e. for Magento2.
No configuration is required and you can still use delegated, cache ...

[markshust]

@mauri-brioschi osxfs is (much) more performant than nfs in my experience. i haven't had any issues with osxfs as long as i am strategic in the volumes i mount.

[strayer]

I'd like to add that in our case it is important to not sync folders like node_modules or Elixirs build and deps folders back to the host especially because of IDE support. Elixirs folders contain architecture-specific compiled files that will break IDE functionality because many plugins need to execute these files. While file system performance is one of our most prominent Docker dev issues, the requirement of being able to exclude folders from syncing is very important too.

The obvious workaround would be to simply mount another volume on top of the shared app volume for node_modules and such, but we encountered strange issues with that too. There are filed issues for those problems, but I can't find them right now. I think the issue was the the volumes for node_modules and so on suddenly didn't mount anymore.

[rfay]

Just a note that all the docker-sync approaches we've tried have failed, and I'd be very suspicious of mutagen. Although the performance was great, the reliability was zero.

[mbrioski]

@markshust i'm following this issue docker/for-mac#1592 since a while, and NFS is definitely not slower than Osxfs... we are using it in all Magento2 projects and performances are not comparable at all. We save minutes with NFS.

[cweagans]

@mauri-brioschi it depends on what you're using it for. For PHP projects that all have a zillion tiny files that need to be read every time you send a request, osxfs is definitely not as good. IIRC, for small numbers of files, it does pretty well.

[ifeltsweet]

We've tried mutagen on macOS and it worked well but we've experienced syncing issues with Windows. Eventually, we chose not to use mutagen in our team since the difference in setups between Windows and macOS slowed us down more (in terms of maintenance) than a less performant "delegated" mode.

Hopefully, a native integration will make this as simple as adding a flag to your volume.

[ajardin]

I'm very excited to have a native implementation of Mutagen within Docker Desktop, as my company and I have been using it for over a year!

We are working on both Magento and Symfony, so we are heavily impacted by the famous file system performance issues... We tried several approaches (Docker Machine NFS, docker-sync, exclude cache/vendors from synchronization, etc.), but the most successful by far is Mutagen, at least for us.

Although having to use a third party solution in addition to Docker is still an issue from my point of view.

[nebuk89]

We've tried mutagen on macOS and it worked well but we've experienced syncing issues with Windows. Eventually, we chose not to use mutagen in our team since the difference in setups between Windows and macOS slowed us down more (in terms of maintenance) than a less performant "delegated" mode.

Hopefully, a native integration will make this as simple as adding a flag to your volume.

This is exactly the experience we want! You should be able to check a volume in the Desktop settings and get it cached :) Initially this will just be released on Mac (as our first preview) due to the point some of you have highlighted that behaviour in Windows differs. Our end goal is to ideally hide this difference so it 'just works with a check box'.

[nebuk89]

This is now out on Docker Desktop Edge, we are after feedback on the UX, the functionality and the performance!

[jasonwilliams]

For those of you having permission issues since switching to Edge there's an issue here:
docker/for-mac#4593

[nebuk89]

@jasonwilliams acknowledged, Dave has looked at the issue and we are looking :)

[dgageot]

@TomasLudvik is it 4.24.1 or 4.14.1 you're talking about?

[TomasLudvik]

I am using 4.14.1 as I noticed here in the comments there were some problems introduced in 4.15.0. It is for me stable version working perfectly with Mutagen.

I have just tried to update to 4.25.0 and I am getting this error when I run mutagen-compose up -d in my project
unable to bring up Mutagen Compose sidecar service: Error response from daemon: Get "https://registry-1.docker.io/v2/": proxyconnect tcp: dial tcp: lookup http.docker.internal on 192.168.65.7:53: read udp 192.168.65.6:59325->192.168.65.7:53: read: connection refused. The same issue has been reported by one of my colleagues.

Second problem, that my colleagues have been encountering on Docker around 4.22 - 4.24 is that synchronization on Mutagen took around 10+ seconds, when after downgrading to 4.14.1 it is instantaneous.

[lorenrh]

Hi @TomasLudvik, would you mind giving this builds a try?

For Intel https://desktop-stage.docker.com/mac/main/amd64/128006/Docker.dmg
For Arm https://desktop-stage.docker.com/mac/main/arm64/128006/Docker.dmg

[KPiccoloDaimao]

@lorenrh @djs55 @dgageot providing the context you have requested. By the way, my entire company is running out of the 4.14.1 release, as that is the only stable performant release. We are now exploring alternative products, as we have been waiting for a solution nearly a year at this point. As you can see below:

@thatJeztah mentioned a fix implemented for a permission issue on 4.14.1 which is the source of all issues apparently:
#7 (comment)

First user @o1y reporting the performance issues after 4.14.1 on 4.15.0:
#7 (comment)

User @o1y benchmark 4 reactions:
#7 (comment)

@kalifg benchmark 7 reactions:
#7 (comment)

@paolomainardi benchmark 2 reactions:
#7 (comment)
#7 (comment)
#7 (comment)

@fredericdalleau providing context to the issue:
#7 (comment)

@KPiccoloDaimao benchmark 5 reactions:
#7 (comment)

@dkechag benchmark 4 reactions:
#7 (comment)

@fredericdalleau providing context and a new release in an attempt to improve it:
#7 (comment)

@KPiccoloDaimao benchmark after attempt to improve it by @fredericdalleau:
#7 (comment)

@dkechag benchmark 2 reactions:
#7 (comment)

@thaJeztah providing a possible solution that never rolled out:
#7 (comment)

@mj3c mentioning the impact of this performance degradation since 4.14.1 18 reactions:
#7 (comment)

[franck-grenier]

Hello,
do you have the download link for Docker Desktop Mac release 4.14.1 ?

[franck-grenier]

Hello,
we are experiencing very slow response times with Docker Desktop on our Mac:

• core i5
• 16Gb RAM
• MacOS 13.6
• Docker 24.0.6
• Docker Desktop 4.25.2
• docker-compose 2.23.0
• VirtioFS on

We have a large local Docker setup with several containers and bind mounts...

We tried Orbstack but no improvements.

The simplest solution I see now is downgrading to a previous version that would provide better perfs. The oldest available is 4.18.0.

How can I download previous releases ?

Or maybe a beta version with big performance improvements ?

[TomasLudvik]

Hi @TomasLudvik, would you mind giving this builds a try?

For Intel https://desktop-stage.docker.com/mac/main/amd64/128006/Docker.dmg For Arm https://desktop-stage.docker.com/mac/main/arm64/128006/Docker.dmg

I am sorry, but we are busy at work so I cannot test every release until it is proven to be better as it takes me at least an hour to switch to the new release, test it, and switch back.

Hello,
do you have the download link for Docker Desktop Mac release 4.14.1 ?

@franck-grenier https://desktop.docker.com/mac/main/arm64/91661/Docker.dmg it is the arm version.

[edmarriner]

Hi All - Wanted to update everyone here - as announced in June, Docker has acquired Mutagen IO, Inc.. We are hard at work integrating it into Docker Desktop and working to roll it out as part of a limited early access program. If you're interested in participating, please sign up at the link below and we'll reach out when ready.

https://www.docker.com/docker-desktop-preview-program/

Hi @smnovick

Just wondering if there was any further progress with the Mutagen integration work mentioned above?

We signed up for the developer preview back in mid-November to help test and provide feedback but we have not been accepted into the program yet.

[itechKaitoQueiroz]

I have been following this topic for more than 1 year. I recently replaced my MacBook Pro Intel Core i7 16GB with a Macbook Pro Apple M2 Max 32GB and for Docker, I allocated a CPU limit of 12 and a memory limit of 12GB. Now I have no problems with Docker performance anymore.

I just would like to share that with you.

[smnovick]

Hi @smnovick

Just wondering if there was any further progress with the Mutagen integration work mentioned above?

We signed up for the developer preview back in mid-November to help test and provide feedback but we have not been accepted into the program yet.

@edmarriner Thanks for your interest - we had a lot of interest in the preview and chose to limit the number of participants in order to optimize feedback. We are planning on releasing the functionality in the 4.27 release at the end of the month, and will share more information at that time.

[NiklasBr]

We are planning on releasing the functionality in the 4.27 release at the end of the month

Can you share where in 4.27 we can enable mutagen?

[grEvenX]

@NiklasBr https://docs.docker.com/desktop/synchronized-file-sharing/

[s1p2a3m]

As per https://docs.docker.com/desktop/synchronized-file-sharing/

Synchronized file shares is available for customers with a Docker Pro, Team, or Business subscription.

Nice, we have to pay for a basic thing to just work, the file system to not be slow in docker on mac.
Seems like a big middle finger to have it a paid thing.
Thanks but no thanks, I'll keep using mutagen as it was before and switching to other containerization when a better one pops up.

[henrik242]

Thanks but no thanks, I'll keep using mutagen as it was before and switching to other containerization when a better one pops up.

It already exists.

[franck-grenier]

Synchronized file shares is available for customers with a Docker Pro, Team, or Business subscription.

Could it be a mistake in the documentation ?

[NiklasBr]

Huh. This was not what I was expecting.

You can use a .syncignore file at the root of each file share, to exclude local files from your file share instance. It supports the same syntax as .dockerignore files and excludes, and/or re-includes, paths from synchronization. .syncignore files are ignored at any location other than the root of the file share.

Some example of things you might want to add to your .syncignore file are:

• Large dependency directories, for example node_modules and composer directories (unless you rely on accessing them via a bind mount)

From my perspective hose directories are very important to make available both in the container and host. In multiple containers (e.g. cron and frontend) for execution, and in the host so that the IDE can detect classes for code autocomplete and ilntellisense-style help.

It seems the new, fourth way, to share files in Docker is still not a good help for Composer-based PHP projects, or am I thinking abut this wrongly?

[xenoscopic]

Hi @NiklasBr, if those directories are critical to your workflow, then there's no need to ignore them. We only recommend excluding them if you don't need them and you want to save space within the VM (since they'll be duplicated there when synced). We actually hope this functionality will be particularly effective for PHP projects, so by all means feel free to include their dependencies in the sync!

[NiklasBr]

Because you recommend ignoring those directories, what use cases would they be ignored in? I can't find any description on how to deal with multiple projects side by side using compose.yaml files where can I find that documentation? E.g. We have well a twenty parallel full PHP projects, do every engineer manually have to configure all this?

[xenoscopic]

Hi again @NiklasBr, great question! It's typically useful to ignore files via .syncignore in cases where you want to create only a single Synchronized File Share for your whole project but you have large assets or dependencies that you don't want to sync into the VM. The other option is to create individual Synchronized File Syncs for each location within a project that you want to bind mount, but even in those cases it can occasionally be useful to exclude contents.

For example, you might have graphic assets (such as .psd files) that aren't used as part of your build but are stored in a mono repo. Or you might have a frontend component to your app that its built on the host system and not in a container, so there wouldn't be a need to sync the (potentially large) node_modules directory in that case.

Another case would be where you have dependencies that include native code (e.g. node_modules dependencies that compile in C code). In those cases, you might want separate copies of the dependencies on the host and in the VM, allowing you to run code using those dependencies in both places.

In general, however, if you need to synchronize the dependencies directories (be they from npm, Composer, Cargo, go mod vendor, etc.), then feel free to do so! The only additional cost is the cost of storing those items in the VM, but that's typically relatively minor.

I'm not sure exactly what you mean with regard to the side-by-side projects using compose.yaml. If you're asking if it's possible to enable this functionality from Compose, not at the moment - the only mechanism to enable it currently is via the settings panel. However, if all of your PHP projects exist in some directory (such as /Users/yourname/Projects/php), then you can create a single Synchronized File Share for that top-level directory and then any Compose projects that you start up from within that directory will use the functionality.

We're still collecting feedback on this functionality, so if there's a mechanism that would be easier for enabling this (such as Compose integration, CLI control, etc.), please feel free to provide your feedback!

[NiklasBr]

I'm not sure exactly what you mean with regard to the side-by-side projects using compose.yaml. If you're asking if it's possible to enable this functionality from Compose, not at the moment - the only mechanism to enable it currently is via the settings panel. However, if all of your PHP projects exist in some directory (such as /Users/yourname/Projects/php), then you can create a single Synchronized File Share for that top-level directory and then any Compose projects that you start up from within that directory will use the functionality.

We pretty much need to be able to do this automatically, manually is too fraught with issues and time-consuming.

Consider this (simplified version) of our project folder structure with three side-by-side projects:

~/workspace/project_foo/
    - docker-compose.yaml
    - php-backend/
        - src/
        - vendor/
    - frontend/
        - node_modules/
    - templates/
    - data/
    - dockerfiles/

~/workspace/project_bar/
    - docker-compose.yaml
    - php-backend/
        - src/
        - vendor/
    - frontend/
        - node_modules/
    - templates/
    - data/
    - dockerfiles/

~/workspace/project_baz/
    - docker-compose.yaml
    - php-backend/
        - src/
        - vendor/
    - frontend/
        - node_modules/
    - templates/
    - data/
    - dockerfiles/

...and so on

These have separate Composer and Node dependencies and are wholly separate customer projects with their own docker images and own repositories. One might work on project_foo one day and the next day project_baz and the next week project_bar.

Some projects may be ElasticSearch, some may not, others may need a frontend pre-renderer, other may not, some projects may use Vue, others React... While they are often built on the same basic tech stack, and have a standardised folder structure and standardised docker volumes in their compose files they may wildy differ in the versions of dependencies, they cannot in reality share any data such as the vendor and node_modules or sync in-between one another.

The need to set up shares quickly without manual steps either via configuration via docker-compose.yaml or at lease an installation script is doubly important for our colleagues in the support department who need more than I to quickly juggle between a lot of different projects to respond to customer needs.

[xenoscopic]

Thanks for the clarification. I think the best suggestion I can offer at the moment would be to create a Synchronized File Share for ~/workspace, which would cause bind mounts for anything under ~/workspace to be replaced. It might be necessary to increase the Docker Desktop disk size, depending on the size of ~/workspace.

With regard to future automation and/or Compose integration, I don't have anything I can share or promise at the moment, but this is definitely something we'll be looking into, so please do keep an eye on future releases.

[NiklasBr]

Thanks @xenoscopic I'll try it as soon as I can find a solution for this issue docker/for-mac#6956 (comment) right now I have to go back to 4.26.1

[piotrsmykaj]

macbook pro 16, i9 + 32GB Ram, amd radeon 5500m + 2 external monitors + Docker with VirtioFS. After upgrading Docker to one of the recent versions (I don't know exactly which one introduced the issue), my fans are going up to 5600 rpm all the time when containers are up. It's uncomfortable to work in a loud office and feel like on the airport. I downgraded Docker to recommended here 4.14.1 and it's way better. Now my fans go to 5600 rpm only when I use and put a load on my containers, but in the idle fans go down to 3500-4000 rpm.

[smnovick]

We have completed our integration of the Mutagen plugin with Docker Desktop and released it as Synchronized File Shares in Docker Desktop 4.27. You can read more here: https://www.docker.com/blog/announcing-synchronized-file-shares/

Given the original request here was to "Integrate the mutagen plugin with Docker Desktop", we will be closing this issue.