-
Notifications
You must be signed in to change notification settings - Fork 579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
shared network mode not working on Mac M1 #1259
Comments
DHCP daemon might not be working? ( |
Great lead, thank you so much! Running With the bootpd lead, I found this issue for a different software (canonical/multipass#2387), and the suggested workaround was to disable the firewall. It's not entirely ideal, but it also works for this case. I'm still trying to figure out how to make the allowlist work, but for now, it looks like the firewall needs to be off when starting the instance. Please feel free to close this issue, as I'm not sure if anything can be done about this on lima's side. |
Some details, in case helpful: With firewall off:
With firewall on:
With firewall on, I ran this same process on an Intel Mac, on macOS Ventura 13.1. |
Apple's built-in firewall.
Same here, no problems on my other machine that is an Intel macOS 13. |
FWIW, I do believe the problem lies in
I then tried
Which successfully gave me an IP on the interface. So while we do have the error like this one (which might be related), I believe the core issue is |
I ran multiple tests on my macOS (disable firewall, disable this, disable that, etc) and I believe |
I just tested same in my setup and agree, it looks like bootp is it fault. If I create a seperate DHCP server similar to what you did above with I was getting this in Colima which seem to be related to this issue discussed here:
The issue is that one network interface does not get a DHCP Address. I updated to latest Mac OS 13.3 and started getting this issue. Disabled ipv6, which temporary worked, but updated to 13.3.1 and back again not working. Which made me dig into it a bit more. Steps followed: Step 1: Start colima:
Step 2: Stop the following:
Step 3: Start isc-dhcp service for bridge100, I used "-d" flag to get more output.
Step 4: SSH into colima host an obtain address for col0
IP Address now gets assigned. I can also now create new colima environments and they work out of box while the Does anyone know how we can troubleshoot or figure out why |
Additional update: When I adjust the Mac network ipv6 settings (I am using a LAN connection - not wifi) - and set under TCP/IP the "Configure IPv6" to "Link-Local Only and reboot (in my case, had to reboot twice which I guess does not make sense) - but then it seem as if the bootpd process starts, previously I just got launchd (one process). Not sure if anyone knows how we can debug or make sure bootpd start properly for ipv4 DHCP? below we can see what is using dhcp port 67
I did try a few times with something like this to manually try below before the changes to ipv6, but it was as if it starts, but nothing happens. Used below command (did unload
even tried:
I know of a few people now that has updated to latest Mac OS 13.x and seeing the issues as discussed here. To add you can use this to see what happens in bootpd and when things work, you can see the DHCP requests, otherwise there is nothing...
some working messages when I start an environment that requests dhcp via socket_vmnet / bridge100
|
@aelsnz thank you for the debug commands, did you find any permanent resolution for this, your steps did help, I did set the IPV6 to link locally and it seems to work after a reboot. |
@AravindGopala unfortunately the issue seem that bootpd does not provide the DHCP address. As work-around in colima (latest 0.5.5) you can now pass in an environment variable COLIMA_IP and set a fixed IP in the 192.168.106.0/24 subnet - (ideally use above 200) - it is at least an option to get past this if the change on ipv6 does not work for you - I know for some this does not work, and the only option then is to use this environment variable to get a fixed IP. Hope this helps.
|
For the others who see the above command failing with
|
After a reboot, I run the following: sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd Please refer to |
Is this still an issue with macOS 14? |
Seems the answer is yes. mprimeaux@lima-default:/Users/mprimeaux/source$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.5.15 netmask 255.255.255.0 broadcast 192.168.5.255
inet6 fec0::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x40<site>
inet6 fe80::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x20<link>
ether 52:55:55:f9:d3:74 txqueuelen 1000 (Ethernet)
RX packets 3723886 bytes 4536204731 (4.5 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 730753 bytes 426101191 (426.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 947 bytes 82721 (82.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 947 bytes 82721 (82.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
nerdctl0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.4.0.1 netmask 255.255.255.0 broadcast 10.4.0.255
inet6 fe80::8865:18ff:fe2d:e4d5 prefixlen 64 scopeid 0x20<link>
ether 8a:65:18:2d:e4:d5 txqueuelen 1000 (Ethernet)
RX packets 19 bytes 1344 (1.3 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16 bytes 1680 (1.6 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
mprimeaux@lima-default:/Users/mprimeaux/source$ exit
logout
❯ ping 192.168.5.15
PING 192.168.5.15 (192.168.5.15): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- 192.168.5.15 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss This is on macOS 14.0 on an M2 Ultra. I see this open issue is on the M1 but the fact I'm on an M2 is unlikely a factor. Turning the macOS firewall off has no effect on ping absorption and stealth mode is not enabled. |
Thanks
This issue might be still specific to ARM Mac? |
This is invalid anyway. This IP is never reachable from the host. Please make sure you have “lima:shared” network in the YAML. If it is working properly, you’ll get an IP like 192.168.105.2 associated with “lima0” interface. |
I can't really comment on this issue since I don't really have a need to ping the NAT'd address. I just access the loopback on a specific port if needed.
Now as to this point, there most definitely are behavioral differences with Lima on Intel versus ARM. As an example, Lima freezes indefinitely at times when I build container images. The only way to recover is |
Ah. Right. Good catch. Let me try that now. |
I started with mprimeaux@lima-default:/Users/mprimeaux/source$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.5.15 netmask 255.255.255.0 broadcast 192.168.5.255
inet6 fec0::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x40<site>
inet6 fe80::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x20<link>
ether 52:55:55:f9:d3:74 txqueuelen 1000 (Ethernet)
RX packets 20613 bytes 25957347 (25.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3422 bytes 297873 (297.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lima0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.105.116 netmask 255.255.255.0 broadcast 192.168.105.255
inet6 fd05:f750:89b5:cf4b:5055:55ff:fe35:e8c3 prefixlen 64 scopeid 0x0<global>
inet6 fe80::5055:55ff:fe35:e8c3 prefixlen 64 scopeid 0x20<link>
ether 52:55:55:35:e8:c3 txqueuelen 1000 (Ethernet)
RX packets 84 bytes 12799 (12.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 92 bytes 8200 (8.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 236 bytes 22474 (22.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 236 bytes 22474 (22.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ...and I am able to ping it from my host machine... ❯ ping 192.168.105.116
PING 192.168.105.116 (192.168.105.116): 56 data bytes
64 bytes from 192.168.105.116: icmp_seq=0 ttl=64 time=0.282 ms
64 bytes from 192.168.105.116: icmp_seq=1 ttl=64 time=0.271 ms
64 bytes from 192.168.105.116: icmp_seq=2 ttl=64 time=0.247 ms
^C
--- 192.168.105.116 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.247/0.267/0.282/0.015 ms |
Thanks, so the issue seems resolved with the recent macOS? |
To test that hypothesis, I'll need to fully reboot and run a few tests. Let me try that next. I also no longer have a machine with an M1 anymore; only M2 machines and an Intel-based Mac in my home lab so I'm unsure how conclusive the test results will be but it will at least provide a data point. |
Actually, I do have an M1 machine. I'll test on both. |
Unfortunately, the 😄 minikube v1.31.2 on Darwin 14.0 (arm64)
✨ Using the qemu2 driver based on user configuration
🌐 Automatically selected the socket_vmnet network
👍 Starting control plane node minikube in cluster minikube
🔥 Creating qemu2 VM (CPUs=6, Memory=32768MB, Disk=81920MB) ...
🔑 Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Password:
🔄 Successfully unblocked bootpd process from firewall, retrying
🔥 Deleting "minikube" in qemu2 ...
🤦 StartHost failed, but will try again: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for a6:8:57:8b:65:a8
🔥 Creating qemu2 VM (CPUs=6, Memory=32768MB, Disk=81920MB) ...
📦 Preparing Kubernetes v1.27.4 on containerd 1.7.2 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
💡 metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.6.4
🌟 The 'metrics-server' addon is enabled
💡 registry is an addon maintained by minikube. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
▪ Using image gcr.io/k8s-minikube/kube-registry-proxy:0.0.5
▪ Using image docker.io/registry:2.8.1
🔎 Verifying registry addon...
🌟 The 'registry' addon is enabled Same failure on both the M1 and the M2 machines, which are running macOS |
This doesn't seem Lima though |
I was a bit overzealous :) My apologies. For lima, specifically, I started with ? Creating an instance "default" Proceed with the current configuration
INFO[0002] Starting socket_vmnet daemon for "shared" network
INFO[0002] QEMU binary "/opt/homebrew/bin/qemu-system-aarch64" seems properly signed with the "com.apple.security.hypervisor" entitlement
INFO[0002] Attempting to download the image arch=aarch64 digest="sha256:af62ca6ba307388f7e0a8ad1c46103e6aea0130a09122e818df8d711637bf998" location="https://cloud-images.ubuntu.com/releases/23.04/release-20230810/ubuntu-23.04-server-cloudimg-arm64.img"
INFO[0002] Using cache "/Users/mprimeaux/Library/Caches/lima/download/by-url-sha256/5a75d2d43280fdaaa39f811921fa8e5906da4945667788f75460b6fc69dbf90d/data"
INFO[0003] Attempting to download the nerdctl archive arch=aarch64 digest="sha256:32a2537e0a80e1493b5934ca56c3e237466606a1b720aef23b9c0a7fc3303bdb" location="https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-arm64.tar.gz"
INFO[0003] Using cache "/Users/mprimeaux/Library/Caches/lima/download/by-url-sha256/2e9505df478bbb8427823380c3ab4ef36836e7c2c9317f6c885d39be36546b19/data"
INFO[0004] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/mprimeaux/.lima/default/serial*.log")
INFO[0004] SSH Local Port: 60022
INFO[0004] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0025] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0025] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0025] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0025] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0034] [hostagent] The essential requirement 3 of 5 is satisfied
INFO[0034] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
INFO[0037] [hostagent] The essential requirement 4 of 5 is satisfied
INFO[0037] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0038] [hostagent] The essential requirement 5 of 5 is satisfied
INFO[0038] [hostagent] Mounting "/Users/mprimeaux" on "/Users/mprimeaux"
INFO[0038] [hostagent] Mounting "/tmp/lima" on "/tmp/lima"
INFO[0038] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available"
INFO[0038] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/mprimeaux/.lima/default/ga.sock" (host)
INFO[0038] [hostagent] The optional requirement 1 of 2 is satisfied
INFO[0038] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed"
INFO[0038] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0038] [hostagent] Not forwarding TCP 127.0.0.54:53
INFO[0038] [hostagent] Not forwarding TCP [::]:22
INFO[0044] [hostagent] The optional requirement 2 of 2 is satisfied
INFO[0044] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0053] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0053] READY. Run `lima` to open the shell. However, I don't see an address resembling mprimeaux@lima-default:/Users/mprimeaux$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.5.15 netmask 255.255.255.0 broadcast 192.168.5.255
inet6 fe80::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x20<link>
inet6 fec0::5055:55ff:fef9:d374 prefixlen 64 scopeid 0x40<site>
ether 52:55:55:f9:d3:74 txqueuelen 1000 (Ethernet)
RX packets 20888 bytes 25972780 (25.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3705 bytes 345293 (345.2 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lima0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fd05:f750:89b5:cf4b:5055:55ff:fe35:e8c3 prefixlen 64 scopeid 0x0<global>
inet6 fe80::5055:55ff:fe35:e8c3 prefixlen 64 scopeid 0x20<link>
ether 52:55:55:35:e8:c3 txqueuelen 1000 (Ethernet)
RX packets 46 bytes 6940 (6.9 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 53 bytes 7421 (7.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 180 bytes 16116 (16.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 180 bytes 16116 (16.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Same behavior on both the M1 and M2. |
...applying the mprimeaux@lima-default:/Users/mprimeaux/source/go-scriptures/platform$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:55:55:f9:d3:74 brd ff:ff:ff:ff:ff:ff
altname enp0s2
inet 192.168.5.15/24 metric 100 brd 192.168.5.255 scope global dynamic eth0
valid_lft 86357sec preferred_lft 86357sec
inet6 fec0::5055:55ff:fef9:d374/64 scope site dynamic mngtmpaddr noprefixroute
valid_lft 86359sec preferred_lft 14359sec
inet6 fe80::5055:55ff:fef9:d374/64 scope link
valid_lft forever preferred_lft forever
3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:55:55:35:e8:c3 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 192.168.105.116/24 metric 100 brd 192.168.105.255 scope global dynamic lima0
valid_lft 86357sec preferred_lft 86357sec
inet6 fd05:f750:89b5:cf4b:5055:55ff:fe35:e8c3/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591960sec preferred_lft 604760sec
inet6 fe80::5055:55ff:fe35:e8c3/64 scope link
valid_lft forever preferred_lft forever Same behavior on both the M1 and M2. Basically, upon reboot, the workaround seems to still be required. |
Thank you, looks like A weird thing is that the $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
The file path you specified does not exist
$ file /usr/libexec/bootpd
/usr/libexec/bootpd: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/libexec/bootpd (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/libexec/bootpd (for architecture arm64e): Mach-O 64-bit executable arm64e |
A rumor is that
|
The gist is interesting and caused me to run the same file /usr/libexec/ApplicationFirewall/socketfilterfw
/usr/libexec/ApplicationFirewall/socketfilterfw: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/libexec/ApplicationFirewall/socketfilterfw (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/libexec/ApplicationFirewall/socketfilterfw (for architecture arm64e): Mach-O 64-bit executable arm64e
file /usr/libexec/bootpd
/usr/libexec/bootpd: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/libexec/bootpd (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/libexec/bootpd (for architecture arm64e): Mach-O 64-bit executable arm64e I had the pleasure of living in Tokyo for a few years and still keep in contact with my friends there; I'm sure they will be interested in the gist. Thanks for sharing! Of course, the big question is "why does |
It would be interesting to know if |
|
Ah. I thought the exception "The file path you specified does not exist" was from |
I have the same problem on Intel mac. Using macOS 14.5 and lima 0.22.0. Neither lima:shared or vzNat will get on the second interface an ip. Which is the service expected to provide dhcpd and maybe there must be some one time setup before creating vms to do? I'm posting this comment like 1h after I tried Lima for the first time. Except not getting an ip, following all in the docs, everything else worked as expected. Inspecting the network after getting started with Lima, I noticed a new bridge100 interface with 2 member interfaces vmenet0 and vmenet2. The bridge has 192.168.105.1 setup, but if Lima was supposed to configure some dhcpd service on the host or instruct virtualization framework for doing it, it didn't worked. |
Description
Environment
macOS Monterey version 12.6.2
Apple M1 Pro chip
limactl version 0.14.1
What I expected
Guest IP address of VM created with
limactl start --name=default template:https://vmnet
should be accessible (i.e. can be pinged) from host.What actually happened
No accessible IP address.
Some notes
/etc/sudoers.d/lima
is properly set up.ping 192.168.5.15
times out.ping 192.168.105.2
times out.The text was updated successfully, but these errors were encountered: