• Release Signoff Checklist
• Summary
• Motivation
  • Goals
• Proposal
  • Manage vendor folders using go modules
  • Publish staging component modules to individual repositories
  • Select a versioning strategy for published modules
  • Remove Godeps
• Design Details
  • Test Plan
  • Graduation Criteria
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Implementation History
• Alternatives
  • Alternatives to vendoring using go modules
  • Alternatives to publishing staging component modules
  • Alternative versioning strategies
• Reference

ACTION REQUIRED: In order to merge code into a release, there must be an issue in kubernetes/enhancements referencing this KEP and targeting a release milestone before Enhancement Freeze of the targeted release.

These checklist items must be updated for the enhancement to be released.

• kubernetes/enhancements issue in release milestone, which links to KEP: #917
• KEP approvers have set the KEP status to implementable
• Design details are appropriately documented
• Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• Graduation criteria is in place
• "Implementation History" section is up-to-date for milestone
• Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
• User-facing documentation has been created

Manage the vendor folder in kubernetes/kubernetes using go modules, and define go.mod module files for published components like k8s.io/client-go and k8s.io/api.

Since its inception, Kubernetes has used Godep to manage vendored dependencies, to ensure reproducible builds and auditable source.

As the go ecosystem matured, vendoring became a first-class concept, Godep became unmaintained, Kubernetes started using a custom version of Godep, other vendoring tools (like glide and dep) became available, and dependency management was ultimately added directly to go in the form of go modules.

The plan of record is for go1.13 to enable go modules by default and deprecate GOPATH mode. To be ready for that, Kubernetes components must make any required adjustments in the Kubernetes 1.15 release.

In addition to simply keeping up with the go ecosystem, go modules provide many benefits:

• rebuilding vendor with go modules provided a 10x speed increase over Godep in preliminary tests
• go modules can reproduce a consistent vendor directory on any OS
• if semantic import versioning is adopted, consumers of Kubernetes modules can use two distinct versions simultaneously (if required by diamond dependencies)

• Remove the kubernetes/kubernetes dependency on godep (including the need for a customized version of godep)
• Increase speed and reliability of building and verifying the vendor directory in kubernetes/kubernetes
• Provide accurate go.mod module descriptors for kubernetes/kubernetes and published staging components
• Enable use of published kubernetes components by go module aware consumers
• Allow continued use of published kubernetes components by go module unaware consumers until go modules are enabled by default
• Allow consumers of go modules to understand the versions of kubernetes libraries they are using

1. Generate go.mod files for k8s.io/kubernetes and each staging component (e.g. k8s.io/client-go and k8s.io/api) as a distinct go module

   • Add require and replace directives in all the go.mod files to register a preference for the same dependency versions currently listed in Godeps.json

     require (
       bitbucket.org/bertimus9/systemstat v0.0.0-20180207000608-0eeff89b0690
       ...
     )
     replace bitbucket.org/bertimus9/systemstat => bitbucket.org/bertimus9/systemstat v0.0.0-20180207000608-0eeff89b0690
     ...
     

   • Add require and replace directives in the k8s.io/kubernetes go.mod file to point to the staging component directories:

     require (
       k8s.io/api v0.0.0
       ...
     )
     replace k8s.io/api => ./staging/src/k8s.io/api
     ...
     

   • Add require and replace directives in the staging component go.mod files to point to the peer component directories:

     require (
       k8s.io/api v0.0.0
       ...
     )
     replace k8s.io/api => ../api
     ...
     

2. Change vendor creation and verification scripts in kubernetes/kubernetes to use go module commands to:

   • Sync dependency versions between k8s.io/kubernetes and in staging component go.mod files
   • Build the vendor directory (go mod vendor)
   • Generate the vendored LICENSES file

See the alternatives section for other vendoring tools considered.

Update the staging component publishing bot:

1. Rewrite the staging component require directives in go.mod files to require specific published versions (the same thing done today for Godeps.json files)
2. Remove the staging component replace directives in go.mod files (relative path references don't make sense for independent repositories)
3. Stop including vendor content in published modules. Vendor folders are ignored in non-top-level modules, and published go.mod files inform go get of desired versions of transitive dependencies.
4. Generate synthetic Godeps.json files containing the SHA or git tag of module dependencies, for consumption by downstream consumers using dependency management tools like glide. Continue publishing these at least until our minimum supported version of go defaults to enabling module support. Update (June 2021): Support for publishing these files was removed since Kubernetes now uses go 1.16.4 and module-aware mode is enabled by default.

See the alternatives section for other publishing methods considered.

As part of transitioning to go modules, we can select a versioning strategy.

State prior to adoption of go modules:

• client-go tagged a major version on every kubernetes release
• all components tagged a (non-semver) kubernetes-1.x.y version on each release
• no import rewriting occurred
• consumers could only make use of a single version of each component in a build

State after adoption of go modules:

• all components tagged a (non-semver) kubernetes-1.x.y version on each release
• no import rewriting occurred
• consumers could only make use of a single version of each component in a build

This proposes publishing components with the following tags:

• Non-semver tags of kubernetes-1.x.y (corresponding to kubernetes v1.x.y)
• Semver tags of v0.x.y (corresponding to kubernetes v1.x.y)

v0.x.y accurately convey the current guarantees around the go APIs release-to-release. The semver tags are preserved in the go.mod files of consuming components, allowing them to see what versions of kubernetes libraries they are using. Without semver tags, downstream components see "pseudo-versions" like v0.0.0-20181208010431-42b417875d0f in their go.mod files, making it extremely difficult to see if there are version mismatches between the kubernetes libraries they are using.

This results in the following usage patterns:

Consumers:

• GOPATH consumers
  • import k8s.io/apimachinery/... (as they do today)
  • go get behavior (e.g. go get client-go):
    • uses latest commits of transitive k8s.io/... dependencies (likely to break, as today)
    • uses latest commits of transitive non-module-based dependencies (likely to break, as today)
• module-based consumers using a specific version
  • reference published module versions as v0.x.y or kubernetes-1.x.y
  • import k8s.io/apimachinery/... (as they do today)
  • go get behavior (e.g. GO111MODULE=on go get k8s.io/[email protected] or GO111MODULE=on go get k8s.io/[email protected]):
    • uses go.mod-referenced versions of transitive k8s.io/... dependencies (unless overridden by top-level module, or conflicting peers referencing later versions)
    • uses go.mod-referenced versions of transitive non-module-based dependencies (unless overridden by top-level module, or conflicting peers referencing later versions)
• consumers are limited to a single copy of kubernetes libraries among all dependencies (as they are today)

Kubernetes tooling:

• minimal changes required

Compatibility implications:

• breaking go changes in each release impact consumers that have not pinned to particular tags/shas (as they do today)
• conflicting version requirements (direct or transitive) can result in impossible-to-build or impossible-to-update dependencies (as they do today)

This would not limit future changes to our versioning strategy:

• modules published and tagged this way could transition to semantic import versioning in the future, if desired
• modules published and tagged this way could transition to v1.x.y semver tagging in the future, if desired (this would require enforcement of go API compatibility in perpetuity, and prevent removal of any go API element, so we are unlikely to pursue this approach, but adding v0.x.y tags now does not remove the option)

See the alternatives section for other possible versioning strategies considered for the initial move to modules.

• Move aggregated Godeps/LICENSES file to vendor/LICENSES (and ensure it is packaged correctly in build artifacts)
• Remove Godeps.json files from kubernetes/kubernetes and staging component directories. With the change to go modules, the only use for these is as a hint to non-module-based downstream consumers of the published staging components.
• Remove the custom Godeps fork from kubernetes/kubernetes.
• Remove all other Godeps references in scripts, comments, and configurations files.
• Update (June 2021): Remove the Godeps directory for all staging component directories. This completely removes Godeps from kubernetes/kubernetes.

• CI scripts to verify vendor contents are recreatable and match referenced versions
• CI scripts to verify vendor licenses are up to date
• CI scripts to verify staging component dependencies are correct
• CI scripts to verify staging component publishing is successful
• CI scripts to verify examples using k8s.io/client-go can be consumed and build automatically and successfully by GOPATH and module-based consumers

• k8s.io/kubernetes vendor management uses go modules
  • CI verifies vendor management scripts succeed with GOPATH unset, in a directory structure not shaped like $GOPATH
• there are documented processes for:
  • adding/pinning a new dependency
  • updating the pinned version of an existing dependency
  • removing an unnecessary dependency
• published staging components can be successfully consumed by:
  • go module consumers using go get
  • GOPATH-based consumers using go get

Not applicable

Not applicable

• 2019-03-19: Created
• 2019-03-26: Completed proposal
• 2019-03-26: Marked implementable
• 2019-04-03: Implemented go module support
• 2019-11-01: Added proposal for tagging published modules with v0.x.y

• Continue using godep for vendor management. This is not viable for several reasons:

  • The tool is unmaintained (the project readme states "Please use dep or another tool instead."), and we have had to make our own fork to work around some edge cases.
  • There are significant performance problems (the pull-kubernetes-godeps CI job takes ~30 minutes to verify vendoring)
  • There are significant functional problems (the tool cannot be run in some environments, gets confused by diamond dependencies, etc)

• Use an alternate dependency mechanism (e.g. dep, glide). This is not preferred for several reasons:

  • Some of the other dependency tools (like glide) are also unmaintained
  • go modules are supported by the go tool, and have stronger support statements than independent vendoring tools
  • Defining go.mod files for published kubernetes components is desired for interoperability with the go ecosystem

• Move away from vendoring in k8s.io/kubernetes as part of the initial move to go modules

  • To ensure reproducible builds in hermetic build environments based solely on the published repositories, vendoring is still necessary
  • Moving away from vendoring is orthogonal to moving to go modules and could be investigated/pursued in the future if warranted.
  • In go1.12.x, vendor-based builds are still the default when building a component located in the GOPATH, so producing components that work when built with go modules or with GOPATH+vendor maximizes interoperability

Since require directives allow locating modules within other modules, it is theoretically possible to stop publishing staging component repositories and require consumers to clone k8s.io/kubernetes and reference the staging component modules within that clone.

Pros:

• Removes the need to publish separate staging component repositories

Cons:

• Git SHAs from before 1.15 would not be available as go modules, and would need to continue being accessed via the separately published repositories, and vanity import meta tags do not allow splitting the location go get looks at based on version

• Pointing to a relative location for a module like k8s.io/api works well for references from within kubernetes/kubernetes, but it's unclear how a consumer outside k8s.io/kubernetes would use a replace directive to let go build automatically locate the nested module (vanity import meta tags do not allow anything other than a repository root when pointing at a VCS, so we could not indicate "k8s.io/api is located at k8s.io/kubernetes//staging/src/k8s.io/api")

• The k8s.io/kubernetes repository is extremely large, and forcing a clone of it to pick up k8s.io/api, for example, is unpleasant

• switch to tagging major versions on every kubernetes/kubernetes release (similar to what client-go does), and use semantic import versioning. This remains a possibility in the future, but requires more tooling and consumer changes to accommodate rewritten imports, and doesn't fully allow multiple versions of kubernetes components to coexist as long as there are transitive non-module-based dependencies that change incompatibly over time.

  • consumers
    • GOPATH consumers
      • import k8s.io/apimachinery/... (as they do today)
      • are limited to a single copy of kubernetes libraries among all dependencies (as they are today)
      • go get behavior (e.g. go get client-go):
        • uses latest commits of transitive k8s.io/... dependencies (likely to break, as today)
        • uses latest commits of transitive non-module-based dependencies (likely to break, as today)
    • module-based consumers
      • reference published modules versions as a consistent vX.y.z version (e.g. v15.0.0)
      • import k8s.io/apimachinery/v15/... (have to rewrite kubernetes component imports on every major version bump)
      • can have multiple copies of kubernetes libraries (though non-semantic-import-version transitive dependencies could still conflict)
      • go get behavior (e.g. GO111MODULE=on go get [email protected]):
        • uses transitive k8s.io/{component}/v15/... dependencies
        • uses go.mod-referenced versions of transitive non-module-based dependencies (unless overridden by top-level module, or conflicting peers referencing later versions)
  • kubernetes tooling
    • requires rewriting all k8s.io/{component}/... imports to k8s.io/{component}/vX/... at the start of each release
    • requires updating code generation scripts to generate versioned imports for k8s.io/{api,apimachinery,client-go}, etc
  • compatibility implications
    • allows breaking go changes in each kubernetes "minor" release
    • no breaking go changes are allowed in a kubernetes patch releases (need tooling to enforce this)
  • allowed versioning changes
    • modules published this way would have to continue using semantic import versioning
    • modules published this way could switch to incrementing major/minor versions at a difference cadence as needed

• tag major/minor versions as needed when incompatible changes are made, and use semantic import versioning. This remains a possibility in the future, but requires more tooling and consumer changes to accommodate rewritten imports, and doesn't fully allow multiple versions of kubernetes components to coexist as long as there are transitive non-module-based dependencies that change incompatibly over time.

  • consumers
    • GOPATH consumers
      • import k8s.io/apimachinery/... (as they do today)
      • are limited to a single copy of kubernetes libraries among all dependencies (as they are today)
      • go get behavior (e.g. go get client-go):
        • uses latest commits of transitive k8s.io/... dependencies (likely to break, as today)
        • uses latest commits of transitive non-module-based dependencies (likely to break, as today)
    • module-based consumers
      • reference published modules versions as a variety of vX.y.z versions (e.g. k8s.io/[email protected], k8s.io/[email protected], k8s.io/[email protected])
      • import k8s.io/apimachinery/v15/... (have to rewrite kubernetes component imports on every major version bump)
      • can have multiple copies of kubernetes libraries (though non-semantic-import-version transitive dependencies could still conflict)
      • go get behavior (e.g. GO111MODULE=on go get [email protected]):
        • uses transitive k8s.io/{component}/vX/... dependencies
        • uses go.mod-referenced versions of transitive non-module-based dependencies (unless overridden by top-level module, or conflicting peers referencing later versions)
  • kubernetes tooling
    • requires rewriting all k8s.io/{component}/vX/... imports when a major version bump occurs
    • requires updating code generation scripts to generate versioned imports for k8s.io/{api,apimachinery,client-go}, etc
    • requires tooling to detect when a breaking go change has occurred in a particular component relative to all tagged releases for the current major version
    • requires tooling to manage versions per component (instead of homogenous versions for staging components)
  • allowed versioning changes
    • modules published this way would have to continue using semantic import versioning
    • modules published this way could switch to incrementing major/minor versions at a difference cadence as needed

• @rsc description of options for kubernetes versioning
• go help modules
• https://github.com/golang/go/wiki/Modules, especially:
  • https://github.com/golang/go/wiki/Modules#semantic-import-versioning
  • https://github.com/golang/go/wiki/Modules#how-to-prepare-for-a-release
• https://golang.org/cmd/go/#hdr-The_go_mod_file
• https://golang.org/cmd/go/#hdr-Maintaining_module_requirements
• https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning
• discussion of tagging with v0.x.y