Baggageclaim btrfs driver "leaks" loopback device attachments

[cirocosta]

Co-authored-by: @topherbullock

Bug Report

When Baggageclaim gets initialized with btrfs set as its device driver, sets up a disk image, a loopback device (finding or creating)

Steps to Reproduce

Here's what we did to repro this:

• Deployed the Helm Chart multiple times to our K8s cluster, with image tags to select a specific K8s node.
• Ran the following docker command against the K8s cluster (using the node the worker has been scheduled on)
  docker run -it --rm --privileged --pid=host justincormack/nsenter1

Expected Results

• We expected that there wouldn't be any loopback devices left behind on the host by worker containers which have been evicted from that node

Actual Results

• From the container, we verified that we could see ~250 loopback devices (loop1, loop2, etc..) losetup --list
• We also observed that these were definitely loopback devices set up for baggageclaim, as they were all (except the one in use) backed by a missing volumes.img

Additional Context

We started looking at this to address concerns associated with this issue in the Helm charts repo helm/charts#9668

Version Info

• Concourse version: 4.2.2
• Deployment type (BOSH/Docker/binary): K8s Helm Chart
• Infrastructure/IaaS: GKE
• Did this used to work? Probably not

[vito]

I think this is one reason to avoid the btrfs driver in Docker/K8s. Loopbacks are a global system resource that are tricky to reliably release.

[william-tran]

Yes! In that you've confirmed something we've dealt with for a while and have spent a lot of effort to work around. I haven't tried overlay recently but maybe worth another shot; or worth the effort to find a kops-k8s image that supports it.

[analytically]

FYI https://lkml.iu.edu/hypermail/linux/kernel/1903.0/02184.html

Kernel 5.1: "btrfs: introduce new ioctl to unregister a btrfs device" - this might help with this issue?

[elgohr]

We ended up deleting them regularly.
If it's still used losetup -d will not delete it.

for loopback in $(/sbin/losetup -a | cut -d: -f1); do /sbin/losetup -d $loopback; done