cmd/go: no easy way to use modules fully privately outside of GOPATH

[marystern]

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

What did you do?

I'm trying out the new finalized modules feature in 1.14 and doing everything outside of GOPATH.
I want all my code to never be publicly visible (i.e. it's a fully private codebase and always will be).

I had an existing codebase with a main package and a separate "utility" package.
Under ~/go/src, everything works fine. But, to try out modules locally, I'm putting my code elsewhere (eg ~/src for the sake of example).

However, to migrate to using modules I'm now having to add a compulsory preceding domain to use the utility package from the main package, so eg, before I had this in my main program:

import "util/log"

which has to change to:

import "x.x/util/log"

"x.x" was the minimum domain name that I could think of that:
a) Would not accidentally resolve to a real domain
b) satisfies the "dot required" restriction in the tools.

My Issues:

1. Inside the utility package itself, I cannot use relative imports, so I have a package util/email that uses util/log, and has now to be changed to import from x.x/util/log to allow "go test" to work (even though it's inside the same package!)
2. I had set GOPRIVATE, but had changed the package name, so go was now trying to resolve and access domain x.x (I realize this is "user error", but is easy to get wrong accidentally, and leaks info very easily (there's another issue somewhere that mentions this).
3. x.x is ugly, other ideas such as example.com, etc are all fine, but have the problem of looking like real places, and hence the leaking issue is a real security concern.
4. Code is not portable from within GOPATH to outside (ie without adding doamin prefixes inside the codebase). (It's possible that this is "by design"?)
5. The docs don't really help users to understand how to work in this very simple (fully local and private) case! My guess is that you would recommend using ~/go (and hence GOPATH), but I think I read somewhere that this is no longer the case?

Proposals

Two thoughts comes to mind (apologies for using this issue also for a prooposal):

1. It might be fairly easy to just say "if the first part of an import does not have a dot in it, assume it's local and private and hence do none of the public module stuff at all". This has the advantage of being simple; not having to remember to set GOPRIVATE and keeping the code GOPATH-compatible.

2. Another idea might be to use "_" as a fully private, local, never-publicly used domain name to indicate such usage in a Module World. So in this case, I could use

import "_/util/log"

Maybe there would be problems with these two ideas, but I'm throwing them out there, not as fully formed proposals, but rather to see if you can see the problem and come up with a good solution.

Anyway, I've been a golang user for many years, but Modules seem to be making my life 100 times harder for zero gain (in this particular case; of course I realize it's all brilliant for public code and dependency/version management, etc, so thanks, and keep up the good work :) )!

[DisposaBoy]

Not a comment on the proposal -- even though I agree 100% and am disappointed that they've highjackingreserved non-domain module names for std lib usage...

I just wanted to recommend the use of x.localhost instead of x.x.

As you've seen, using x.x will result in a name lookup and depending on what you've used instead of literal x.x, you might suddenly find yourself pulling in code from some random, possibly malicious, server on the internet. Think the recent-ish drama around Google now owning the .dev TLD. I've also seen legit requests coming into golag.org and other domains I've registered for security research purposes.

[marystern]

Hi again, sorry, but this is an addendum to the above.

I was under the impression that having a local module and doing a "go install" would build and install that module into the local module-cache, but it apparently does not: is that expected? It seems that I'm possibly supposed to use a "replace" directive in my go.mod so that the main module can find the util module. If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare! (Maybe I'm just missing something here?)

(Would it not be a good idea for "go install" to simply install the module into my local module cache so that it can be easily found by any other local modules that use the utils? Is this a bug?)

[marystern]

Not a comment on the proposal -- even though I agree 100% and am disappointed that they've highjackingreserved non-domain module names for std lib usage...

I just wanted to recommend the use of x.localhost instead of x.x.

As you've seen, using x.x will result in a name lookup and depending on what you've used instead of literal x.x, you might suddenly find yourself pulling in code from some random, possibly malicious, server on the internet. Think the recent-ish drama around Google now owning the .dev TLD. I've also seen legit requests coming into golag.org and other domains I've registered for security research purposes.

Thanks DisposaBoy, I'm not sure, but I thought that top-level domains had to be a minimum of 3-chars in length, so I chose "x" assuming that it could never be resolved to a real server (also to avoid import-namespace cruft), but it seems that the name still tries to get resolved anyway (apparently even when it's listed in GOPRIVATE which I find very strange) :(

[bcmills]

A module's path only has to have a dot in the first component if you want to be able to fetch it automatically (such as using go mod download or go get -u).

You can, today, replace a module with a dotless prefix, and in fact we do that a lot in the cmd/go tests themselves. (For example, see https://github.com/golang/go/blob/52fdd624a4087342f529cd7e52c92f44adf2e2bf/src/cmd/go/testdata/script/mod_indirect.txt.)

Note that we still reserve those names in general for the standard library (#32819), but you can use them at your own risk.

[DisposaBoy]

You can't tell people that it's fine to use names without a dot, and then immediately take it back by claiming them as reserved...

[bcmills]

@DisposaBoy, we don't want to limit the names available to the standard library, but it seems reasonable to reserve a specific path prefix (or set of path prefixes) for local, user-defined modules. Want to draft a proposal?

[bcmills]

@marystern, go install intentionally does not add anything to the module cache.

(Unlike GOPATH, the module cache is a cache: given a complete go.mod file, and assuming that the upstream sources still exist or are still mirrored in a GOPROXY, the module cache can be rebuilt from the network, starting from scratch, in a way that does not depend on any local state.)

See also #28835.

[bcmills]

@marystern

If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare!

Could you explain more about your use-case? (Why would 'replace' directives — or a single dotless root module — not scale, and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)

[marystern]

@marystern

If this is how it's supposed to work, it's not very scalable at all for my use-case, and would be a maintenance nightmare!

Could you explain more about your use-case? (Why would 'replace' directives — or a single dotless root module — not scale, and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)

@bcmills
The use case is this:
I have two main programs which use a shared, separate package at 2 different version levels. These are all held in 3 separate, private, local git repos. All code is private and will always be, and my Company doesn't want external network requests where they are not necessary (or open to abuse).

Also individual team members should be free to clone the 3 repos anywhere they like on their local filesystems, and work on any or all projects.

So, user Alice checks out

~/src/main1/..
~/src/main2/...
~/src/util/...

but Bob likes to use this arrangement:

~/main1/src/..
~/main2/src/...
~/shared_libs/special/util/src/...

So, from the above you can see that both Alice and Bob might need to edit the go.mod replace directives to accommodate their individual workspace preferences. This is what I was referring to as "un-scalable", i.e. across disparate team members or other users of the code (i.e. without mandating a scheme such as "put all your code in ~/go/src so that all relative go.mod replace directives don't need to be edited", i.e. they all have to be under a shared parent dir much like the old GOPATH approach).

re: and why would issuing a request against a domain that you own, or an invalid domain, be a security concern?)
if I use a domain that is currently invalid, it's possible for it to be made valid in the future, in which case that domain's server gets requests from our organization even though this wasn't our intention, so potentially leaks some info about our codebase accidentally (I'm not sure what would actually be sent, but you get the idea). Hope this helps!

[bcmills]

@marystern, if Alice and Bob were using GOPATH mode, they would presumably both have to check out the repositories into exactly:

GOPATH/src/main1/...
GOPATH/src/main2/...
GOPATH/src/util/...

In that case, you could use replace directives pointing to ../main1, ../main2, and ../util respectively, and those would be in the correct location in any GOPATH. So, at the very least, this does not seem like a significant feature-parity regression from GOPATH mode.

[bcmills]

if I use a domain that is currently invalid, it's possible for it to be made valid in the future, in which case that domain's server gets requests from our organization even though this wasn't our intention

What about one that you, or in this case your company, owns? It is possible that you will leak information about your own domain to your own DNS server on the local network — but is there a reason you can't trust your own DNS server?

Also note that you can set GOPROXY=off to disable remote fetches entirely, regardless of path.

[marystern]

So, at the very least, this does not seem like a significant feature-parity regression from GOPATH mode.

Agreed, but the point of this experiment was to check: "Can we successfully use MODULE-MODE locally and ignore GOPATH mode totally?". If the answer is don't even try, then I agree, GOPATH-Mode is the only sensible option (and was the conclusion that I ended up with myself). However, my understanding (which may well be incorrect: please feel free to clarify) was that everyone should be moving to Module Mode: maybe this is not the intention?

Note however, that using Module Mode would indeed be preferable to GOPATH Mode for this use case (because 2 projects require 2 different versions of a shared module), and with a couple of tweaks (eg as per my proposal) would be very nice indeed for private, local development as I've outlined in the Use Case.

My feeling is that this particular use-case is one that has slipped through the net and has not been given full consideration during the addition of Module support (again, I might well be wrong, and I am indeed aware of the amount and complexity that's been involved in properly supporting dependencies/modules, so please don't take this the wrong way). Hence, you may be fully justified as seeing this as a request for a further enhancement to Module support.

[marystern]

Also note that you can set GOPROXY=off to disable remote fetches entirely, regardless of path.

Yes, but we usually want some external code (eg from github). (Yes, we could use vendoring, but again, I thought this was one of the things that Module support was designed to avoid!)

[bcmills]

Yes, but we usually want some external code (eg from github).

If you want to selectively filter dependencies, you can already do that using a local GOPROXY: for example, it could return status 403 for any module that isn't explicitly approved for use.

Or, you can set GOPROXY=off for normal use and only switch it on when you are intentionally adding a new dependency.

Or, you can set GOPROXY=off and use GONOPROXY to unblock specific repositories, although you wouldn't then be able to fetch those from the default public proxy.

[bcmills]

my understanding … was that everyone should be moving to Module Mode

Yes: at this point we believe that module mode is ready for general use. We're continuing to improve the user experience, but at this point all of the known use-cases have at least some available option.

In this case, you have at least three existing options:

1. GOPROXY (either off or set to a privacy-enforcing local proxy)
2. GOPRIVATE and a domain under the control of your organization
3. replace directives with dotless import paths.

We should investigate those options fully before we consider adding yet another one.

[marystern]

my understanding … was that everyone should be moving to Module Mode

Yes: at this point we believe that module mode is ready for general use. We're continuing to improve the user experience, but at this point all of the known use-cases have at least some available option.

In this case, you have at least three existing options:

1. `GOPROXY` (either `off` or set to a privacy-enforcing local proxy)

The downside of this is that we would be forced to setup a proxy for this very simple use-case which seems like overkill.

2. `GOPRIVATE` and a domain under the control of your organization

The downside is that, in this case, there is no domain under our control that we want to (or can) use for this purpose.

3. `replace` directives with dotless import paths.

Can you explain what you mean by "dotless" import paths please (otherwise I think have explained the problems above)?

We should investigate those options fully before we consider adding yet another one.

[bcmills]

1. GOPROXY (either off or set to a privacy-enforcing local proxy)

The downside of this is that we would be forced to setup a proxy for this very simple use-case which seems like overkill.

go env -w GOPROXY=off seems very simple to me.

(A privacy-enforcing local proxy is admittedly less simple, but we're examining all of the options here.)

2. GOPRIVATE and a domain under the control of your organization

The downside is that, in this case, there is no domain under our control that we want to (or can) use for this purpose.

I hear that there are companies that will sell you a domain for $12/yr. or less.

Also note that you can use an IANA-reserved IP address as an import-path prefix — for free! — if you so desire:
https://play.golang.org/p/opatQQKGdlR

3. replace directives with dotless import paths.

Can you explain what you mean by "dotless" import paths please (otherwise I think have explained the problems above)?

Pick a name that you are confident will never be part of the Go standard library, and use that as your import prefix:
https://play.golang.org/p/eaNGtR2NqCA

[dmitshur]

The downside is that, in this case, there is no domain under our control that we want to (or can) use for this purpose.

@marystern If you don't mind, I want to ask for more information about this, in order to understand your situation better. That can help with finding a better solution to this issue.

Is the problem that you don't have a domain available for this, or that you don't want to maintain a domain for this purpose?

You mentioned earlier that:

Also individual team members should be free to clone the 3 repos anywhere they like on their local filesystems,

If it's something you can talk about, can you say if a domain is used for cloning the repositories? If so, would it be viable to reuse that domain for your module paths? Or are there concerns with doing so?

I agree it's unfortunate to have to purchase and maintain a new domain for the sole purpose of having a shorter import path and making it so the go command doesn't send requests to a domain that you don't control. That's why I want to learn if there is perhaps a domain you can reuse for this purpose that you haven't considered yet.

[marystern]

@dmitshur

Is the problem that you don't have a domain available for this, or that you don't want to maintain a domain for this purpose?

It's both: we don't have a domain and we don't want to have to maintain a domain purely to do private, local builds! Isn't that a reasonable thing?

If it's something you can talk about, can you say if a domain is used for cloning the repositories? If so, would it be viable to reuse that domain for your module paths? Or are there concerns with doing so?

No, it's very simple: think of it as all code on one machine (ie fully network disconnected). Simply try to build 2 main products, each using different versions of a shared module (I'm reducing the reality here for the sake of example so that you can see the issue clearly: consider the case of a file-system-local repo; a single developer).

This is almost the simplest scenario for using GOPATH, but if I want to build both products, I have to somehow switch the versions of the shared module each time I build (and make sure it's correct each time). Modules should be able to fix this Use Case (I had hoped! :) as they deal with the versioning and dependency issues quite nicely. However, this Issue is trying to highlight the complexity of implementing the solution: ie stick to manually switching code all the time; or use vendoring; or introduce a local, private proxy (even though the local module cache looks like it should be sufficient to me, maybe somehow integrated with the local build cache, but that's not detail that I'm fully clear about. Also there are the issues of naming etc as mentioned above.).

[marystern]

1. GOPROXY (either off or set to a privacy-enforcing local proxy)

The downside of this is that we would be forced to setup a proxy for this very simple use-case which seems like overkill.

go env -w GOPROXY=off seems very simple to me.

(A privacy-enforcing local proxy is admittedly less simple, but we're examining all of the options here.)

1. GOPRIVATE and a domain under the control of your organization

The downside is that, in this case, there is no domain under our control that we want to (or can) use for this purpose.

I hear that there are companies that will sell you a domain for $12/yr. or less.

Also note that you can use an IANA-reserved IP address as an import-path prefix — for free! — if you so desire:
https://play.golang.org/p/opatQQKGdlR

1. replace directives with dotless import paths.

Can you explain what you mean by "dotless" import paths please (otherwise I think have explained the problems above)?

Pick a name that you are confident will never be part of the Go standard library, and use that as your import prefix:
https://play.golang.org/p/eaNGtR2NqCA

@bcmills I agree that it's possible to do things today in a way that can work, it's just that I find the options very un Go-like (ie very complex for a pretty simple requirement)! (Dare I say it reminds me of python3...... ;)

[dmitshur]

Thanks for the answers @marystern.

It's both: we don't have a domain and we don't want to have to maintain a domain purely to do private, local builds! Isn't that a reasonable thing?

Before answering that, I think it's helpful to think about the reason using a domain, even if a fake one like project.localhost, is a recommended solution.

Go packages are identified by import paths, which are string values. Two distinct packages made by different groups need to find a way to create unique import paths. If two different packages have the same import path, using those packages together becomes difficult and confusing.

The Go project has made use of an existing system, the URL (and as part of that, DNS), as a mechanism for ensuring all import paths can be unique. If everyone picks an import path that has a domain name they own, import path collisions can be avoided.

Given the existing solution has good properties and is very simple (an import path is a URL), I think we should be careful not to attempt to build something on top of it.

The Go standard library does not have a domain prefix, which itself is a unique namespace reserved by the Go standard library. If the Go standard library used the "golang.org" domain for all packages, then an import like "fmt" would be "golang.org/fmt", "net/http" would be "golang.org/net/http". Doing that would allow users to use no-domain paths for their projects, but that would still be risky in case you want to combine two such codebases in the future.

As a result, I think it is reasonable to suggest using a localhost domain like project.localhost as a way to work within the current system and have all of its benefits, at the cost of more verbose import paths for your project.

There may still be room for improvement here, but I just wanted to share my understanding of the trade-offs in the current design of import paths.

[bcmills]

@marystern, the most basic development model in module mode is to have everyone work in the same module in the same repository. Then you don't need to worry about versioning at all; that is, IMO, the most “Go-like” option.

And I don't think the comparison to python3 is apt at all: this issue has absolutely nothing to do with syntactic compatibility or Unicode strings. (In fact, the vast majority of packages that worked with go get prior to modules continue to work in module mode today.)

[bcmills]

If you really need the complexity of multiple modules, then the cost of that is the complexity of telling the go command where to find those modules. Moreover, we have a strong expectation that a given version of a given module means the same thing to everyone, so we also expect that some server provides that consistent view of versions, via commits and tags in a single copy of the repository that serves as the shared source of truth.

That protocol is described in https://golang.org/cmd/go/#hdr-Remote_import_paths, and it identifies the server hosting a given module by using an HTTP or HTTPS connection to a server at a readily-identifiable domain.

The response from that HTTP server tells the go command where to find the repository containing the module. I believe that it should (but have not tested that it does) work with a file:https:// URL, and if it does not then that seems like a straightforward fix.

So one option is to run a local HTTP server, and either use the path prefix 127.0.0.1 or configure your machine to resolve a given domain name (such as pkg.localhost) to a localhost IP address: that is, to explicitly tell the go command to find the packages in specific places on your local machine.

[bcmills]

Another option (as described earlier) is to run a local GOPROXY that injects versions from known repo locations. There are a number of third-party GOPROXY implementations, some of which are open-source, and I would be surprised if none of them supports this sort of use-case in any fashion.

A GOPROXY is, in a very deep sense, another way to explicitly tell the go command how to resolve the locations of modules.

[marystern]

Hi all,

Thanks for the discussion and feedback.

I'd like to conclude with my own thoughts (and remember I'm looking at this as a simple coder with a simple use-case who doesn't really need all the bells and whistles of the "enterprise" module features):

1. Local "publishing" of modules

I would like to be able to simply work locally using modules so that I can manage dependencies in my private, local codebase. The part that seems to be missing is a mechanism to simply build and install a version of one module (with a specific module version number) locally so that it can be picked up by other local modules.

When looking at 1.14, I had expected that there be some way to do this without installing a module-proxy. I think I expected to be able to something like:

go install 

to build a dev-version (or perhaps check locally to see if the current code was tagged and hence at a specific VCS version), or possibly something like

go install -mod v2.0.0

to build and install as a specific named version (ie not rely on any VCS naming, or to override it if required).

So, this would build my module and install it into the local module cache, ready to be picked up by other local modules (without any need to consult anything outside my network/machine).

Question: is there any technical reason that this would not be possible (eg as an enhancement to go install)?

2. Naming of domain for local modules

I had expected to be able to use something simple (or indeed nothing, ie GOPATH-compatible import paths) to name my module so that it clearly would never be accessible remotely.

My top 2 favorites are
a) no domain specifier at all (ie GOPATH-compatible)
b) "_" as a "non-domain".

I still like option b) as it looks clean and clear and seems to be consistent with the other uses of "_" to mean something that isn't really there.

Question: is there any reason that "_" could not be used like this (I realize it might be a lot of work and special cases, etc, but as a user it seems to be a nice, clear solution).

If the above 2 things were already possible, I wouldn't have had to stop my progress with using modules locally, and I believe that this would enhance the user experience and ease the transition to using modules when they are required, and would mean that all new code could live in the new "module world", and GOPATH could fade away into history.

[bcmills]

this would build my module and install it into the local module cache, ready to be picked up by other local modules (without any need to consult anything outside my network/machine).

Question: is there any technical reason that this would not be possible (eg as an enhancement to go install)?

Nope. In fact, I proposed something very similar in #28835 (which is still open for consideration, but not a top priority at the moment).

[bcmills]

Question: is there any reason that "_" could not be used like this (I realize it might be a lot of work and special cases, etc, but as a user it seems to be a nice, clear solution).

That appears to already work today: https://play.golang.org/p/hl_tG2_ndkb

I can't see us ever using _ as a package name in the standard library, but I could also see it getting caught by future module-path validation. If you like, feel free to send a CL adding a regression test to src/cmd/go/testdata/script to lock in that behavior — that will at least prompt us to think about whether it's worth breaking if and when we tighten up path checks.

(CC @jayconrod @matloob)

[marystern]

So, to close off things from my end: I looked into running a private proxy, but the issue I had is how to build/publish the package (zip) for the proxy to use (probably not too hard, but I've spent too much time on this already, so didn't look into it). Decided instead to use:

"_" as a domain name (looks nice and clean and clear to me: please don't make this invalid in the future! From my perspective, it would even be nice if this was highlighted in intro docs as an alternative to a valid domain for anyone else who wants to work privately.)

..together with..

replace directives in my go.mod files.

eg a go.mod for main1 would include:

module _/main1
replace _/util => ../util

and in the main1, main2 code I use these import paths:

import "_/util/..."

This works as long as all the modules are placed next to each other (ie because I'm replacing with "../").

I can live with this for now, but it's not as flexible as I might wish (I seem to desire a "go put" opposite to "go get" which could be spelled as "go install" or "go mod pack" or whatever). You can close this if you like (sounds like #28835 covers the missing "go put" feature which is something I would like to have!), thanks for all the help and guidance! :)

[bcmills]

Thanks! Closing out this issue, and we'll keep this feedback in mind when considering future improvements.