Mounting volume device using the --volumes flag not working

[mdundek]

Hello Weaveworks team,

I am trying to start a VM that uses the --volume flag to mount a disk device inside a VM, but all attempts to make this work are futile. Here is what I do:

# Create loop device
dd if=/dev/zero of=/test_vol/diskimage bs=1M count=1024
# Make it a ext4 FS
mkfs.ext4 /test_vol/diskimage
# Set up the loop device
sudo losetup /dev/loop17 /test_vol/diskimage
# Start Ignite with the --volume flag (the loop device is not mounted locally, my understanding is that Ignite / Firecracker will mount it inside the VM)
sudo ignite \
        run weaveworks/ignite-kubeadm:latest \
        --network-plugin cni \
        --cpus 2 \
        --memory 2GB \
        --ssh \
        --volumes /dev/loop17:/dev/loop1 \
        --name master-1

Here is the output of the command:

INFO[0000] Created VM with ID "e2c782438e20f3f3" and name "master-1" 
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "e2c782438e20f3f3" in a container with ID "ignite-e2c782438e20f3f3" 
INFO[0002] Waiting for the ssh daemon within the VM to start... 
INFO[0013] Waiting for the ssh daemon within the VM to start... 
INFO[0023] Waiting for the ssh daemon within the VM to start... 
FATA[0033] dial tcp 10.50.59.6:22: connect: connection refused

There is no documentation on how to use the --volumes flag, I did try all sorts of variations to make this work, but nothing helps. Can someone help me out on this please?

BR

[darkowlzz]

Hi, thanks for reporting this. I tried this and I can confirm that this works with docker as the runtime but fails with containerd (default). If you pass --runtime docker and --network-plugin docker bridge to the run command, it works.

$ sudo ignite run weaveworks/ignite-ubuntu --runtime docker --network-plugin docker-bridge --volumes /dev/loop0:/myblock --ssh --name my-vm
INFO[0000] Created VM with ID "804ba2d7cc185524" and name "my-vm"
INFO[0001] Networking is handled by "docker-bridge"
INFO[0001] Started Firecracker VM "804ba2d7cc185524" in a container with ID "f93b90a1002119fc3fb4c46fa655526e4e4397907dc00480847a43e691f47d68"
INFO[0001] Waiting for the ssh daemon within the VM to start...

$ sudo ./bin/ignite ssh my-vm

root@804ba2d7cc185524:~# ls /myblock/
lost+found

root@804ba2d7cc185524:~# df
Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/root        4119616 316444   3577100   9% /
devtmpfs          246292      0    246292   0% /dev
tmpfs             249260      0    249260   0% /dev/shm
tmpfs              49856    184     49672   1% /run
tmpfs               5120      0      5120   0% /run/lock
tmpfs             249260      0    249260   0% /sys/fs/cgroup
/dev/vdb          388462   2318    361568   1% /myblock
tmpfs              49852      0     49852   0% /run/user/0

Doing the same with defaults:

$ sudo ignite run weaveworks/ignite-ubuntu --volumes /dev/loop0:/myblock --ssh --name my-vm
INFO[0000] Created VM with ID "d0d480a8d925b8f0" and name "my-vm"
INFO[0001] Networking is handled by "cni"
INFO[0001] Started Firecracker VM "d0d480a8d925b8f0" in a container with ID "ignite-d0d480a8d925b8f0"
INFO[0002] Waiting for the ssh daemon within the VM to start...
INFO[0014] Waiting for the ssh daemon within the VM to start...
INFO[0024] Waiting for the ssh daemon within the VM to start...
FATA[0034] dial tcp 10.61.0.3:22: connect: connection refused

Although ssh fails, the VM does start and is usable with ignite attach. Need to investigate more about what causes this failure. (UPDATE: reason in the next comment)

We don't have any test coverage for this feature at the moment, so we weren't aware of the issue.

If you would like to contribute documentation with example of how to use it, it would be great 🙂

[darkowlzz]

Some updates after a debugging session for this issue in the weekly developer meeting:

The issue seems to be because of the device that's attempted to be mounted is not available in the VM container, maybe due to some missing containerd option. ignite-spawn skips the volume:

WARN[0000] Skipping nonexistent volume: "volume0"
INFO[0000] Starting DHCP server for interface "br_eth0" (10.61.0.5)
INFO[0000] Called startVMM(), setting up a VMM on /var/lib/firecracker/vm/032511832f331d2d/firecracker.sock
...

Due to this, when the VM starts up, it tries to load the device but since the device isn't available, it waits for 1min 30s

[ ***  ] A start job is running for /dev/dis…8a70-1ae0c9ba1866 (29s / 1min 30s)

The VM startup continues after the timeout:

[  OK  ] Finished Load/Save Random Seed.
[ TIME ] Timed out waiting for device 66-4cec-8a70-1ae0c9ba1866.
[DEPEND] Dependency failed for /myblock.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for File…9-0f66-4cec-8a70-1ae0c9ba1866.

This causes the ssh daemon to start later than usual.

Since the ssh connection is checked right after creating the VM, and times out before 1min 30s, it fails. Trying to ssh into the VM later works. But the device that's requested to be mounted isn't available.

[stealthybox]

@mdundek Thank you for your bug report!
We fixed this in #653

You can build from master or wait for the next release :)