Restore kustomize windows build disabled for v4.2.0

[monopole]

The windows build was failing in goreleaser for unknown reasons during the attempt to release kustomize/v4.2.0.

In the interests of making the kubectl integration deadline with a new release, the windows build was disabled and a canned version for windows isn't offered for v4.2.0. It can appear in a patch release.

Possibly related to #4001

[KnVerey]

We should consider doing a goreleaser build (https://goreleaser.com/quick-start/#dry-run) in CI to help discover problems like this ahead of release time in the future.

[juergenzimmermann]

@monopole @KnVerey Unfortunately, the latest Windows release is 4.1.3 while the latest linux release is 4.3.

[juergenzimmermann]

@monopole @KnVerey @natasha41575 Is there any chance to get 4.4 for Windows, too? The latest Windows release is still 4.1.3.

[natasha41575]

It might take some time because none of us have windows machines, but we are working on a fix.

[olljanat]

It should enough if you can make cross build working with command like:
GOOS=windows GOARCH=amd64 go build

Some of us Windows users can help with testing after that.

[natasha41575]

/assign

[natasha41575]

/unassign

If there is anyone with a windows machine that would like to dig deeper into why the windows build fails, we would greatly appreciate that.

[olljanat]

@natasha41575 I can try help but can you share how to build this locally on Linux dev machine? I cannot see anything on readme and command like make build looks to be not supported.

[natasha41575]

From inside the kustomize/kustomize directory, run go install .

EDIT: The make target is make $(go env GOPATH)/bin/kustomize

[olljanat]

@natasha41575 hmm so what is the problem here? As far I see Windows build works just fine. Here cross build example done inside of Linux container

# Start build environment inside of container
git clone https://github.com/kubernetes-sigs/kustomize
docker run -it --rm -v $(pwd)/kustomize:/go/src/github.com/kubernetes-sigs/kustomize -w /go/src/github.com/kubernetes-sigs/kustomize golang

# Install prerequirements and build Linux binary first (as it also install pluginator which is need to build Windows part)
go get golang.org/x/tools/cmd/goimports
make $(go env GOPATH)/bin/kustomize

# Build Windows binary
export GOOS=windows
make $(go env GOPATH)/bin/kustomize

as a result /go/bin/windows_amd64/kustomize.exe is available inside of container and can be copied to Windows machines.

[natasha41575]

@olljanat I'm hoping someone can do @KnVerey's suggestion:

We should consider doing a goreleaser build (https://goreleaser.com/quick-start/#dry-run) in CI to help discover problems like this ahead of release time in the future.

All we know is that the goreleaser failed to build the windows executable. Perhaps if a goreleaser dry-run build passes in CI we can reenable windows builds and try again the next release. If you are able to build the Windows build without a problem then it is possible that the error was a fluke and it won't happen again.

[patricknelson]

Found it! I'm not a Go developer, but I'm on Windows, so I took a stab at this.

It turns out it wasn't the windows build failing. The real issue was actually in the configuration of goreleaser itself. It was failing from gobinary: "go1.16.5" (which I believe was a regression) introduced by #4026 but got immediately rolled back in the next PR #4027 the same day (resulting in no more windows builds but also fixing the real source the error).

Here's what I think happened...

---

See PR #4262 for a revised localbuild.sh and the associated docker command to easily reproduce these results locally with goreleaser.

---

Things were made worse by the build parallelism introducing a race condition which was combined with the fact that goreleaser was generating a vague error message. The result ended up with Windows winning the lottery in that particular build and thus being blamed. A fluke indeed, @natasha41575! 😄 Basically, it kicks off the builds in parallel and whichever happens to error first is the one you see in the final output (which will always be the one that runs first since all of them will fail). It's important to note that the CLI output of building binary=... is also randomized completely separately from the actual build execution (see below). So if you ran this build 100 times, you should get a random distribution of platforms failing, a distribution which would also be unrelated to the order of the platforms you see in the CLI output.

So, I think that might also explain @monopole's commit message suggesting that it was an issue with Windows. Sometimes the CLI output order happens to align in some way with the error generated, e.g.:

   • building binaries
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_windows_amd64/kustomize.exe
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_amd64/kustomize
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_arm64/kustomize
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_darwin_amd64/kustomize
   ⨯ build failed after 0.40s error=failed to build for windows_amd64:

However, when you drop in a --debug flag, you'll see the real order that the commands are actually executed in. While the error output is still a bit vague, it becomes quite a bit easier to reason about:

   • building binaries
      • building                  build={kustomize [linux darwin windows] [amd64 arm64] [6] [hardfloat] [linux_amd64 linux_arm64 darwin_amd64 windows_amd64] [] . . [-s -X sigs.k8s.io/kustomize/api/provenance.version={{.Version}} -X sigs.k8s.io/kustomize/api/provenance.gitCommit={{.Commit}} -X sigs.k8s.io/kustomize/api/provenance.buildDate={{.Date}}
] [] [] kustomize {[] []} [] go [] []  false go1.16.5 false}
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_windows_amd64/kustomize.exe
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_arm64/kustomize
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_amd64/kustomize
      • building                  binary=/go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_darwin_amd64/kustomize
      • running                   cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_amd64/kustomize .] env=[GOPATH=/go CGO_ENABLED=0 GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOOS=linux GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=]
      • failed                    cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_amd64/kustomize .] env=[GOPATH=/go CGO_ENABLED=0 GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOOS=linux GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=] error=exec: "go1.16.5": executable file not found in $PATH
      • running                   cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_darwin_amd64/kustomize .] env=[OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOPATH=/go CGO_ENABLED=0 GO111MODULE=on GOOS=darwin GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=]
      • failed                    cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_darwin_amd64/kustomize .] env=[OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOPATH=/go CGO_ENABLED=0 GO111MODULE=on GOOS=darwin GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=] error=exec: "go1.16.5": executable file not found in $PATH
      • running                   cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_windows_amd64/kustomize.exe .] env=[GOPATH=/go CGO_ENABLED=0 GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOOS=windows GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=]
      • failed                    cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_windows_amd64/kustomize.exe .] env=[GOPATH=/go CGO_ENABLED=0 GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOLANG_VERSION=1.16.5 TERM=xterm SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOOS=windows GOARCH=amd64 GOARM= GOMIPS= GOMIPS64=] error=exec: "go1.16.5": executable file not found in $PATH
      • running                   cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_arm64/kustomize .] env=[SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOLANG_VERSION=1.16.5 TERM=xterm GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOPATH=/go CGO_ENABLED=0 GOOS=linux GOARCH=arm64 GOARM= GOMIPS= GOMIPS64=]
      • failed                    cmd=[go1.16.5 build -ldflags=-s -X sigs.k8s.io/kustomize/api/provenance.version=kustomize/v4.4.0 -X sigs.k8s.io/kustomize/api/provenance.gitCommit=984a2dab3d34563c4c7b92b94c606e1f764e7937 -X sigs.k8s.io/kustomize/api/provenance.buildDate=2021-10-29T02:42:00Z
 -o /go/src/github.com/kubernetes-sigs/kustomize/kustomize/dist/kustomize_linux_arm64/kustomize .] env=[SHLVL=2 OLDPWD=/go/src/github.com/kubernetes-sigs/kustomize _=/usr/local/bin/goreleaser HOSTNAME=7030ef210186 HOME=/root GOLANG_VERSION=1.16.5 TERM=xterm GO111MODULE=on PWD=/go/src/github.com/kubernetes-sigs/kustomize/kustomize PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin GOPATH=/go CGO_ENABLED=0 GOOS=linux GOARCH=arm64 GOARM= GOMIPS= GOMIPS64=] error=exec: "go1.16.5": executable file not found in $PATH
   ⨯ build failed after 0.36s error=failed to build for linux_amd64:

I suggest we do the following:

1. As @KnVerey suggested: Have the ability to easily run goreleaser build locally without actually releasing it (decoupling it from Cloud KMS itself and etc).
2. Have a build script that initializes goreleaser consistently, regardless of the actual build environment, this should make local builds much easier to reproduce (decoupling it from Cloud Builder)

My implementation was to rename localbuild.sh to cloudbuild-local.sh (since it was tightly coupled still to Cloud Build) and simply copy cloudbuild.sh to a new local-build.sh and change goreleaser release to goreleaser build. Then I added --debug to both so the errors would be easier to understand moving forward.

I'll edit this comment once my PR is issued. 😃

EDIT: See PR at #4262.

[natasha41575]

Windows build is reenabled in 4.4.1.

[juergenzimmermann]

@natasha41575 Thank you! It works fine.