[Question] Kustomize URL formats

[thatsmydoing]

kustomize's support for URL formats is not well-specified and provides shortcuts for sources like github that do not extend to other repositories and can be cause for confusion. For kustomization v1, I think it would be a good idea to formalize it since it will be a breaking change for other fields as well.

The problems with kustomize's url formats that I see are:

git url extraction

Typically, kustomize manifests live in subdirectories of a repository so a typical url looks like

https://github.com/kubernetes-sigs/kustomize/examples/helloWorld

where the repository to clone is https://github.com/kubernetes-sigs/kustomize and the subdirectory to build is examples/helloWorld. And kustomize just assumes that the repository is always 2 subdirectories deep. This leads to problems when this isn't the case and more special cases were added that say look one after the _git directory and look for the directory that ends in .git. So we support cloning the following repositories correctly:

• https://itfs.mycompany.com/collection/project/_git/somerepos
• https://bitbucket.example.com/scm/project/repository.git

but https://bitbucket.example.com/scm/project/repository will be treated as clone https://bitbucket.example.com/scm/project and build the subdirectory repository.

I think the better answer here is to just use // to delimit the repository url and the directory (this is already supported but is still subjected to the above). In that case, there is no need for special cases as we're not guessing anymore. So that https://bitbucket.example.com/scm/project/repository will mean just clone the repository repository and build the root. While https://bitbucket.example.com/scm/project//repository will mean clone the project repository and build repository

go-getter legacy and url magic

kustomize states that it supports "go-getter" format and previously actually did use go-getter (but not always for git cloning). For specifying a repository, all of the following are currently valid for HTTP clones

• github.com/kubernetes-sigs/kustomize/examples/helloWorld
• gh:github.com/kubernetes-sigs/kustomize/examples/helloWorld
• https://github.com/kubernetes-sigs/kustomize/examples/helloWorld
• git::https://github.com/kubernetes-sigs/kustomize/examples/helloWorld
• github.com:kubernetes-sigs/kustomize/examples/helloWorld

and the following are valid for SSH clones

• [email protected]:kubernetes-sigs/kustomize/examples/helloWorld
• [email protected]/kubernetes-sigs/kustomize/examples/helloWorld
• git::[email protected]:kubernetes-sigs/kustomize/examples/helloWorld
• ssh:https://[email protected]:kubernetes-sigs/kustomize/examples/helloWorld
• ssh:https://[email protected]/kubernetes-sigs/kustomize/examples/helloWorld

and probably a lot more. The logic lives in https://github.com/kubernetes-sigs/kustomize/blob/master/api/internal/git/repospec.go which greatly needs an overhaul.

What's more, most of these only apply to github.com urls, so you can say github.com/kubernetes-sigs/kustomize/examples/helloWorld but bitbucket.com/kubernetes-sigs/kustomize/examples/helloWorld does not. There's also some weird logic with .git being appended to the repository but not if it's an AWS or Azure code host.

I think we should simply restrict the urls to one's accepted by git clone. Many of the above examples are not valid or wrong. Obviously gh: and git:: prefixes are not clonable.

• git clone github.com/kubernetes-sigs/kustomize tries to clone the local directory github.com/kubernetes-sigs/kustomize
• git clone github.com:kubernetes-sigs/kustomize will do an SSH clone as your current user but kustomize does an HTTPS one
• git clone [email protected]/kubernetes-sigs/kustomize is not valid as you must use : to delimit host and directory
• git clone ssh:https://[email protected]:kubernetes-sigs/kustomize is not valid as you must use / to delimit the host and directory

remote resource overlap

kustomize currently tries to load a resource as a "file" first and then as a base. kustomize also supports loading remote file resources via HTTPS so this means it's possible a resource that has https:// can be interpreted as either a file or a base and you'd have to try both to know which is which. This does mean for bases that begin with https://, kustomize will try to load it as a file first, then fail, then clone the repository and succeed. This is fine for local files and bases but has some performance penalty for remote files and bases.

I'm not really sure what can be done here so long as the formats can overlap. We could say that all remote bases must have a //. So urls without // are treated as files but those with // are treated as bases. This might be acceptable since I don't believe it's common to have kustomization.yaml at the root of a repository and for those cases we can just say add // or //. to the end. This is a completely breaking change though and can't be eased into like the other suggestions.

[k8s-ci-robot]

@thatsmydoing: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thatsmydoing]

In #4453, it was suggested that remote git bases should just be prefixed with a git:: to disambiguate between remote file resources.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thatsmydoing]

/remove-lifecycle stale

[thatsmydoing]

It seems tests have been added in #4615

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[thatsmydoing]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thatsmydoing]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[thatsmydoing]

/remove-lifecycle stale

[thatsmydoing]

It looks like gh: and git:: are now deprecated as of #4954

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted