podman build not invalidating layers after a change in a --mount=type=bind directory

[AndrewLipscomb]

Issue Description

When building an image using podman build and a build mount of type --mount=type=bind - a change in content within that mount directory does not cause the layer of the build to be invalidated and run again.

docker build on the other hand does do this - I would expect the same behaviour between both programs

Steps to reproduce the issue

I have a demo of the problem at https://github.com/AndrewLipscomb/podman-build-cache-issue-demo/tree/main

But in short

• Make content in a folder in directory
• Mount that folder as part of an image build
• Build the image
• Change the content in the directory and rebuild the image
• The image will come out the same due to that layer not invalidating

For some reproduction instructions in full

Tested with podman 5.0.3 and podman 5.1.2 - both did the same thing
Docker version for comparison was

Docker version 26.1.3, build b72abbb6f0

Describe the results you received

The layer is not invalidating on a mount directory content change

Describe the results you expected

The layer should invalidate on a mount directory content change

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 97.07
    systemPercent: 0.7
    userPercent: 2.24
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2046
  hostname: andrew.swarmfarm.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.9.2-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 1502134272
  memTotal: 32409886720
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.11.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.11.0
    package: /usr/lib/podman/netavark is owned by netavark 1.11.0-2
    path: /usr/lib/podman/netavark
    version: netavark 1.11.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.15-1
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: /usr/bin/pasta is owned by passt 2024_06_24.1ee2eca-1
    version: |
      pasta 2024_06_24.1ee2eca
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 16142110720
  swapTotal: 17064521728
  uptime: 151h 15m 9.00s (Approximately 6.29 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/andrew.lipscomb/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrew.lipscomb/.local/share/containers/storage
  graphRootAllocated: 485565878272
  graphRootUsed: 427259781120
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 138
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrew.lipscomb/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720733172
  BuiltTime: Fri Jul 12 07:26:12 2024
  GitCommit: 94a24974ab345324db1a1489c924af4b89d2d0e9
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.2

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

_No response_

### Additional information

_No response_

[flouthoc]

I think this can be easily solved by adding hash of directory to the history, i'd like to propose a solution if @nalind agrees then I can create a patch.

As of now buildah adds history to images for --mount like.

Current

 {
            "created": "2024-08-13T15:07:01.335991879Z",
            "created_by": "/bin/sh -c --mount=type=bind,source=/host/path/,dst=/test,Z ls /test",
            "comment": "FROM docker.io/library/alpine:latest"
        },

Proposed

 {
            "created": "2024-08-13T15:07:01.335991879Z",
            "created_by": "/bin/sh -c --mount=type=bind,source=/home/ar/work/buildah/bin/,dst=/test,Z:<hash-of-the-mounted-directory-or-file> ls /test",
            "comment": "FROM docker.io/library/alpine:latest"
        },

[nalind]

When the source is in the build context or a previous stage, I guess that'll work. When it's in an image, it's going to be cheaper to use the image's digest.

[flouthoc]

I agree, in case of image it can be digest. I can take a stab at it.