Issue: hashFiles function attempts to hash directories and errors with io.Copy error

[leighmcculloch]

System information

• Operating System: macOS
• Architecture: arm64 (64-bit)
• Apple M1: yes
• Docker version: 20.10.12
• Docker image used in act: ghcr.io/catthehacker/ubuntu@act-latest
• act version: v0.2.26-0.20220215200300-ff13844b8643

Expected behaviour

Running a workflow that includes this string interpolated works on GitHub actions:

${{ github.workflow }}-${{ github.job }}-${{ matrix.go }}-${{ runner.os }}-go-all-${{ github.ref }}-${{ hashFiles('**', '!.git') }}

It results in a string like so:

Go-check-1.17-Linux-go-all-refs/pull/3727/merge-5ee8a89e3d82367e7e6cc247d85e042a1952b32fd4331db7d74cdd08b5bb80ad

Example: https://github.com/stellar/go/runs/5309071935?check_suite_focus=true

Actual behaviour

Running the action with the string interperolated with act displays this error:

ERRO[0037] Unable to interpolate expression 'format('{0}-{1}-{2}-{3}-go-all-{4}-{5}', github.workflow, github.job, matrix.go, runner.os, github.ref, hashFiles('**', '!.git'))': Unable to io.Copy: read /Users/leighmcculloch/Code/stellar--go/.circleci: is a directory 

It appears that act is interpreting the hashFiles('**', '!.git') incorrectly and is attempting to hash directories, not just files.

Workflow and/or repository

https://github.com/stellar/go/blob/189950c43f89ea908699953d298b120f1a81b371/.github/workflows/go.yml

Steps to reproduce

git clone https://github.com/stellar/go
cd go
git checkout 189950c43f89ea908699953d298b120f1a81b371
act -j check

act output

Log

[Go/check] 🧪  Matrix: map[go:1.17 os:ubuntu-latest]
[Go/check] 🚀  Start image=ghcr.io/catthehacker/ubuntu:act-latest
[Go/check]   🐳  docker pull image=ghcr.io/catthehacker/ubuntu:act-latest platform= username= forcePull=false
[Go/check]   🐳  docker create image=ghcr.io/catthehacker/ubuntu:act-latest platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Go/check]   🐳  docker run image=ghcr.io/catthehacker/ubuntu:act-latest platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Go/check]   🐳  docker exec cmd=[mkdir -m 0777 -p /var/run/act] user=root workdir=
[Go/check]   🐳  docker cp src=/Users/leighmcculloch/Code/stellar--go/. dst=/Users/leighmcculloch/Code/stellar--go
[Go/check]   🐳  docker exec cmd=[mkdir -p /Users/leighmcculloch/Code/stellar--go] user= workdir=
[Go/check] ⭐  Run actions/checkout@v2
[Go/check]   ✅  Success - actions/checkout@v2
[Go/check] ⭐  Run Set up Go
[Go/check]   ☁  git clone 'https://github.com/actions/setup-go' # ref=v2
[Go/check]   🐳  docker cp src=/Users/leighmcculloch/.cache/act/actions-setup-go@v2/ dst=/var/run/act/actions/actions-setup-go@v2/
[Go/check]   🐳  docker exec cmd=[mkdir -p /var/run/act/actions/actions-setup-go@v2/] user= workdir=
[Go/check]   🐳  docker exec cmd=[node /var/run/act/actions/actions-setup-go@v2/dist/index.js] user= workdir=
| Setup go stable version spec 1.17
[Go/check]   💬  ::debug::isExplicit: 
[Go/check]   💬  ::debug::explicit? false
[Go/check]   💬  ::debug::evaluating 0 versions
[Go/check]   💬  ::debug::match not found
| Attempting to download 1.17...
| matching 1.17...
[Go/check]   💬  ::debug::check 1.17.7 satisfies 1.17
[Go/check]   💬  ::debug::x64===x64 && darwin===linux
[Go/check]   💬  ::debug::x64===x64 && linux===linux
[Go/check]   💬  ::debug::matched 1.17.7
| Acquiring 1.17.7 from https://github.com/actions/go-versions/releases/download/1.17.7-1828071684/go-1.17.7-linux-x64.tar.gz
[Go/check]   💬  ::debug::Downloading https://github.com/actions/go-versions/releases/download/1.17.7-1828071684/go-1.17.7-linux-x64.tar.gz
[Go/check]   💬  ::debug::Destination /tmp/4af9b0ae-da4a-4d2c-83de-044a43a88ee3
[Go/check]   💬  ::debug::download complete
| Extracting Go...
[Go/check]   💬  ::debug::Checking tar --version
[Go/check]   💬  ::debug::tar (GNU tar) 1.30%0ACopyright (C) 2017 Free Software Foundation, Inc.%0ALicense GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.%0AThis is free software: you are free to change and redistribute it.%0AThere is NO WARRANTY, to the extent permitted by law.%0A%0AWritten by John Gilmore and Jay Fenlason.
| [command]/usr/bin/tar xz --warning=no-unknown-keyword -C /tmp/57e481b6-1fc9-4bd1-86ea-23b386b4bfa6 -f /tmp/4af9b0ae-da4a-4d2c-83de-044a43a88ee3
| Successfully extracted go to /tmp/57e481b6-1fc9-4bd1-86ea-23b386b4bfa6
| Adding to the cache ...
[Go/check]   💬  ::debug::Caching tool go 1.17.7 x64
[Go/check]   💬  ::debug::source dir: /tmp/57e481b6-1fc9-4bd1-86ea-23b386b4bfa6
[Go/check]   💬  ::debug::destination /opt/hostedtoolcache/go/1.17.7/x64
[Go/check]   💬  ::debug::finished caching tool
| Successfully cached go to /opt/hostedtoolcache/go/1.17.7/x64
| Added go to the path
[Go/check]   💬  ::debug::which go :/opt/hostedtoolcache/go/1.17.7/x64/bin/go:
[Go/check]   💬  ::debug::go env GOPATH :/root/go:
[Go/check]   💬  ::debug::creating /root/go
[Go/check]   💬  ::debug::creating /root/go/bin
[Go/check]   💬  ::debug::add bin true
| Successfully setup go version 1.17
[Go/check]   ❓  ##[add-matcher]/run/act/actions/actions-setup-go@v2/matchers.json
| go version go1.17.7 linux/amd64
| 
[Go/check]   ❓  ::group::go env
| GO111MODULE=""
| GOARCH="amd64"
| GOBIN=""
| GOCACHE="/root/.cache/go-build"
| GOENV="/root/.config/go/env"
| GOEXE=""
| GOEXPERIMENT=""
| GOFLAGS=""
| GOHOSTARCH="amd64"
| GOHOSTOS="linux"
| GOINSECURE=""
| GOMODCACHE="/root/go/pkg/mod"
| GONOPROXY=""
| GONOSUMDB=""
| GOOS="linux"
| GOPATH="/root/go"
| GOPRIVATE=""
| GOPROXY="https://proxy.golang.org,direct"
| GOROOT="/opt/hostedtoolcache/go/1.17.7/x64"
| GOSUMDB="sum.golang.org"
| GOTMPDIR=""
| GOTOOLDIR="/opt/hostedtoolcache/go/1.17.7/x64/pkg/tool/linux_amd64"
| GOVCS=""
| GOVERSION="go1.17.7"
| GCCGO="gccgo"
| AR="ar"
| CC="gcc"
| CXX="g++"
| CGO_ENABLED="1"
| GOMOD="/Users/leighmcculloch/Code/stellar--go/go.mod"
| CGO_CFLAGS="-g -O2"
| CGO_CPPFLAGS=""
| CGO_CXXFLAGS="-g -O2"
| CGO_FFLAGS="-g -O2"
| CGO_LDFLAGS="-g -O2"
| PKG_CONFIG="pkg-config"
| GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1532238468=/tmp/go-build -gno-record-gcc-switches"
| 
[Go/check]   ❓  ::endgroup::
[Go/check]   ✅  Success - Set up Go
[Go/check] ⭐  Run actions/cache@v2
[Go/check]   ☁  git clone 'https://github.com/actions/cache' # ref=v2
[Go/check]   🐳  docker cp src=/Users/leighmcculloch/.cache/act/actions-cache@v2/ dst=/var/run/act/actions/actions-cache@v2/
[Go/check]   🐳  docker exec cmd=[mkdir -p /var/run/act/actions/actions-cache@v2/] user= workdir=
[Go/check]   🐳  docker exec cmd=[node /var/run/act/actions/actions-cache@v2/dist/restore/index.js] user= workdir=
[Go/check]   ❓  ::save-state name=CACHE_KEY::Go--1.17-Linux-go-mod-
[Go/check]   💬  ::debug::Resolved Keys:
[Go/check]   💬  ::debug::["Go--1.17-Linux-go-mod-","Go--1.17-Linux-go-mod"]
[Go/check]   💬  ::debug::Checking zstd --version
[Go/check]   💬  ::debug::*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
[Go/check]   💬  ::debug::getCacheEntry - Attempt 1 of 2 failed with error: Cache Service Url not found, unable to restore cache.
[Go/check]   💬  ::debug::getCacheEntry - Attempt 2 of 2 failed with error: Cache Service Url not found, unable to restore cache.
| [warning]getCacheEntry failed: Cache Service Url not found, unable to restore cache.
| 
[Go/check]   ⚙  ::set-output:: cache-hit=false
[Go/check]   ✅  Success - actions/cache@v2
ERRO[0037] Unable to interpolate expression 'format('{0}-{1}-{2}-{3}-go-all-{4}-{5}', github.workflow, github.job, matrix.go, runner.os, github.ref, hashFiles('**', '!.git'))': Unable to io.Copy: read /Users/leighmcculloch/Code/stellar--go/.circleci: is a directory 
[Go/check] ⭐  Run actions/cache@v2
[Go/check]   ☁  git clone 'https://github.com/actions/cache' # ref=v2
[Go/check]   🐳  docker cp src=/Users/leighmcculloch/.cache/act/actions-cache@v2/ dst=/var/run/act/actions/actions-cache@v2/
[Go/check]   🐳  docker exec cmd=[mkdir -p /var/run/act/actions/actions-cache@v2/] user= workdir=
[Go/check]   🐳  docker exec cmd=[node /var/run/act/actions/actions-cache@v2/dist/restore/index.js] user= workdir=
[Go/check]   ❗  ::error::Input required and not supplied: key
[Go/check]   ❌  Failure - actions/cache@v2
[Go/check] exit with `FAILURE`: 1
Error: Job 'check' failed

[leighmcculloch]

I think the problem is here:

act/pkg/exprparser/functions.go

Lines 208 to 216 in ff13844

	for _, file := range files {
	f, err := os.Open(file)
	if err != nil {
	return "", fmt.Errorf("Unable to os.Open: %v", err)
	}
	if _, err := io.Copy(hasher, f); err != nil {
	return "", fmt.Errorf("Unable to io.Copy: %v", err)
	}

Glob will return directories and files, and so after opening the file, act should check if the file is a directory and skip it if so. Something like:

diff --git a/pkg/exprparser/functions.go b/pkg/exprparser/functions.go
index 64103e0..a1c99fa 100644
--- a/pkg/exprparser/functions.go
+++ b/pkg/exprparser/functions.go
@@ -211,6 +211,14 @@ func (impl *interperterImpl) hashFiles(paths ...reflect.Value) (string, error) {
                        return "", fmt.Errorf("Unable to os.Open: %v", err)
                }
 
+               fi, err := f.Stat()
+               if err != nil {
+                       return "", fmt.Errorf("Unable to File.Stat: %v", err)
+               }
+               if fi.IsDir() {
+                       continue
+               }
+
                if _, err := io.Copy(hasher, f); err != nil {
                        return "", fmt.Errorf("Unable to io.Copy: %v", err)
                }

[ZauberNerd]

Ok, it seems like the standard golang path/filepath Glob/Match implementation is incompatible with the GitHub syntax.

• It doesn't support ** to match any directory in any depth.
• And it doesn't support ! to exclude matches.

Issue about double star: golang/go#11862 and some alternatives: https://www.client9.com/golang-globs-and-the-double-star-glob-operator/
Unfortunately most of the mentioned alternatives don't support the ! negation pattern.
The glob implementation that is used inside godo seems to be the most compatible and maybe we can use it as a dependency (https://pkg.go.dev/gopkg.in/godo.v2/glob) although it doesn't have a go.mod file and the glob/glob.go file is part of the overall main godo package. Also, it seems like the last commit has been 7 years ago: https://github.com/go-godo/godo/tree/v2/glob.

Here's some more information on the hashFiles function: https://docs.github.com/en/actions/learn-github-actions/expressions#hashfiles and on the filter patterns: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet

Edit: Here's the GitHub glob functionality implemented: https://github.com/actions/toolkit/tree/main/packages/glob and the hashFiles implementation is here: https://github.com/actions/runner/tree/main/src/Misc/expressionFunc/hashFiles

[catthehacker]

these patterns are gitignore syntax, so you could just apply the reversed gitignore and walk through files

act/pkg/container/docker_run.go

Lines 579 to 592 in c802064

	var ignorer gitignore.Matcher
	if useGitIgnore {
	ps, err := gitignore.ReadPatterns(polyfill.New(osfs.New(srcPath)), nil)
	if err != nil {
	log.Debugf("Error loading .gitignore: %v", err)
	}
	ignorer = gitignore.NewMatcher(ps)
	}
	err = filepath.Walk(srcPath, func(file string, fi os.FileInfo, err error) error {
	if err != nil {
	return err
	}

[github-actions]

Issue is stale and will be closed in 14 days unless there is new activity