add rfc about registry url in lock file #84

vernak2539 · 2017-09-21T10:42:04Z

No description provided.

bestander

Thanks, @vernak2539 that is a great start!
Just some cleanup and more concrete examples would be enough.

bestander · 2017-10-04T16:12:31Z

accepted/0000-registry-url-in-lock-file.md

+Let me know if this isn't enough, first time writing an RFC.
+
+
+As a dev that generated their lock file against REGISTRY_A


Here we need a more technical explanation with examples of yarn.lock and config parameter names and concrete URL examples.

@bestander updated the PR. Let me know if more is needed

bestander · 2017-10-04T16:14:00Z

accepted/0000-registry-url-in-lock-file.md

+
+# How We Teach This
+
+This would largely be in the internal workings, and wouldn't require teaching. npm 5 does this


We just need an entry in docs page, maybe here https://yarnpkg.com/en/docs/yarnrc.

A short blog post would be great too.

It is straight forward, so no need to mention anything else here.

bestander

Good start, @vernak2539.
I've left a few comments

bestander · 2017-10-24T06:22:00Z

accepted/0000-registry-url-in-lock-file.md

+
+# Motivation
+
+We currently have two internal registries/mirrors. We generate the lock file against one we have in


Add more details please.
Short examples how the yarn.lock files are different would be useful

bestander · 2017-10-24T06:22:24Z

accepted/0000-registry-url-in-lock-file.md

+our download to error as yarn cannot retrieve the packages.
+
+The expected output is for yarn to recognise the registry URL supplied (via either cli flag or
+`.yarnrc` or `.npmrc`) and download from that if the registry used in the lock file is not available.


no need to do it in .npmrc, just mention .yarnrc.

bestander · 2017-10-24T06:22:43Z

accepted/0000-registry-url-in-lock-file.md

+no longer have access to the original registry that was used to generate the lock file. This causes
+our download to error as yarn cannot retrieve the packages.
+
+The expected output is for yarn to recognise the registry URL supplied (via either cli flag or


please provide the name of the flag

bestander · 2017-10-24T06:27:36Z

accepted/0000-registry-url-in-lock-file.md

+I would like to designate an alternate registry URL in my `.yarnrc` or `.npmrc` (REGISTRY_B)
+And I would like yarn to download my dependencies from the alternate URL (REGISTRY_B)
+
+I feel like the simplest solution would be to add a "hash" or "commit-ish" field to entries.


That is a lot of text that does not add much value compared to the previous section.
Here we need to see concrete examples how Yarn will behave with the new feature.
I suggest something like this:

If yarn.lock file contains

abbrev@1: version "1.1.0" resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.0.tgz#d0554c2256636e2f56e7c2e5ad183f859428d81f"

And .yarnrc has setting override-registry={"registry.yarn.pkg.com": "registry.npm.org"}
Then Yarn will fetch abbrev-1.1.0.tgz from https://registry.npm.org/abbrev/-/abbrev-1.1.0.tgz#d0554c2256636e2f56e7c2e5ad183f859428d81f instead of the one specified in yarn.lock.

bestander · 2017-10-24T06:29:01Z

accepted/0000-registry-url-in-lock-file.md

+
+This, with the version already provided, could then be used to generate a URL on the fly or at the beginning of operations if an alternate registry is specified in the scenarios above.
+
+Or, the `resolved` field could change to not include the hostname/origin, which would then be inserted during operations. This would


Let's keep this proposal simple and not have many possible solutions in one place, you can list them below in the alternatives section

bestander · 2017-10-24T06:30:28Z

accepted/0000-registry-url-in-lock-file.md

+already, so it would be inherent.
+
+> If you generated your package lock against registry A, and you switch to registry B, npm will now try to install the packages from registry B, instead of A. If you want to use different registries for different packages, use scope-specific registries (npm config set @myscope:registry=https://myownregist.ry/packages/). Different registries for different unscoped packages are not supported anymore.
+


Then let's just copy what npm5 does.
Make a proposal to match npm5 behavior with similar examples and configs

If we want a proposal to do just like npm, nothing would need to change, just yarn would have to take into account the registry in the rc file, and use the registry in that no matter how the lock file was generated. I think that's where I was going with the original one, but that would be a fundamental change in behaviour, probably a breaking change

I've updated the PR as well as my comment

Can I contribute something here? Very eager to see this go through - it is a dealbreaker at the moment for us since we (like @vernak2539 ) use different registries in different build environments.

@csvan, everyone's contribution is welcome.

Hi,

our use case is sharing code between external (outside the company, seeing directly public npm repos) and internal developers (i.e. working inside a company, with proxy and an artifactory mirroring the public npm repos). moreover, there is a ci that also sees only internal repos.

actually, there are two issues with repos in yarn.lock:

while reading (i.e. when installing npm modules)

while writing (i.e. when new modules are added)

my remarks / desires (without regard whether they are fulfilled by the suggested solutions above):

for 1.: basically, what we need is to have a single yarn.lock that is valid and working in both environments, so it can reside in our git repo. even if we would be fine without them, yarn.lock must have embedded repo URLs for backward compatibility. hence we would like to have a single set of URLs, which should be: public URLs for public available npm packages and private (i.e. pointing to the artifactory repo) URLs for private packages (actually, the company has separate artifactory URLs for public and private npm packages). obviously, access to the private packages is only possible from inside the company, so this would break for external developers - which is OK.

for 2.: the desire to have a single set of URLs both valid inside and outside the company (obviously this is only possible for public packages) would mandate that public URLs are written into yarn.lock even when a dependency on a public npm package is added by an internal developer and thus served by the internal artifactory repo. note that this use case does typically not occur during a build process (e.g. in a ci/cd). of course, the package namager has no other means to decide whether a package is public than configuration and the final package URL (in our case, it would be possible to make this distinction, as private packages have a different internal URL, in other cases there might be a set of rules giving such a distinction - which in turn might be overshooting to support).

thus we would probably need means to override a certain public repo by a private one as a local configuration property (not going into the git repo - e.g. in .npmrc) and simultaneously using this override "backwards" when writing URLs into yarn.lock. note also that the override cannot only be only a host, as the URL path and/or the protocol under which npm modules are visible may differ between different repos.

any update on this rfc ?

bestander · 2017-10-30T21:09:03Z

Overall I like it.
Thanks for following this up, @vernak2539, very much appreciated.
Does anyone have anything to add?

If not then let's merge it and wait for a PR.

bestander

LGTM, waiting for more comments.
If none please ping me to merge this as is in 5 days.

BYK · 2017-11-01T16:05:51Z

I think this is a good approach but why not go one step further and remove the registry URLs from the lockfile completely?

bestander · 2017-11-02T01:44:04Z

but why not go one step further and remove the registry URLs from the lockfile completely?

backwards/forwards compatibility
easier for Yarn to recognize the pattern as registry sourced
smaller change required

imsnif · 2017-11-02T15:54:27Z

Hey, thanks to @csvan for pointing this out! We're feeling this pain when moving registries (yarnpkg/yarn#4817)... This might seem trivial, but I want to make sure (since I did not see this mentioned) - will the added flag also recreate the package checksum? (the checksum ultimately includes the repository url, so if a package was re-published to another registry we'll get a checksum mismatch).

bestander · 2017-11-02T16:18:20Z

Commented on the linked issue, I think this is a separate thing

BYK · 2017-11-02T17:31:27Z

@bestander

backwards/forwards compatibility
easier for Yarn to recognize the pattern as registry sourced

We can probably add a new field per resolution and signal this easily. That should be forward and backwards compatible. Only thing is new lock files wouldn't work with old Yarn versions but I think that was never a concern.

smaller change required

With my suggestion above, I'm not sure if it would be that big of a difference. The advantage here is avoiding extra processing per field.

bestander · 2017-11-06T17:45:08Z

@BYK, I am ready to discuss the change. :)
Here is my argument - making this change will be a safe and implementation could be fast.

Stripping npm/yarn domain needs way more than just one step forward, we need to think of the migration path.
I think that kind of change is worth a major version update even though coding it is easy.

bestander · 2017-11-13T08:00:49Z

To keep the ball rolling I'll merge this as is.
Very much welcome to enhance it in future iterations but I think we are good to go to get the minimum viable solution in.

@vernak2539 are you eager to send a PR?

arcanis · 2018-05-03T11:02:05Z

Regarding the implementation, I have two comments:

Not a fan of passing a JSON object as command line argument. Do we need this option? I feel like the most common use case will be to switch the default registry for everything at once (for example to use registry.npmjs.org instead of registry.yarnpkg.com), so I'd prefer if it was possible to do this without having to specify a full JSON object (which doesn't play well with shellscripts variables).
The yarnrc files use our lockfile file format, which supports dicts. It would be better to use this syntax instead of encoding the registry mapping with JSON.

arcanis · 2018-05-03T11:05:57Z

I also note the following comment quoting the npm behavior:

Different registries for different unscoped packages are not supported anymore.

Why do we need to support a registry=>registry mapping, then?

rally25rs · 2018-05-03T12:27:28Z

I think this is a good approach but why not go one step further and remove the registry URLs from the lockfile completely?

Since this could be slated for yarn v2, we could ignore backwards compatibility. IMO adding another setting to override the registry is just going to cause confusion between when to use registry= and override-registry=

kopax-polyconseil · 2023-09-21T08:28:25Z

The checksum of the yarn lock change if you edit it, that's one problem
The git repo not being up to date after editing the yarn lock, a second problem.
A workaround would be to allow to specifiy the lock file then create a 2nd one with the first one using the other repo, that's not possible
A setting stronger than --repository that could take precedence over the lock file, also not possible.

Any plan to implement a solution within yarn after all those years ?

Alex Vernacchia and others added 2 commits September 21, 2017 11:40

add rfc about registry url in lock file

459e860

Update 0000-registry-url-in-lock-file.md

02aad79

vernak2539 mentioned this pull request Sep 26, 2017

yarn.lock should not include base registry(https://registry.npmjs.org) yarnpkg/yarn#3330

Closed

BYK self-assigned this Sep 29, 2017

bestander suggested changes Oct 4, 2017

View reviewed changes

BYK removed their assignment Oct 6, 2017

update registry url in lock file 'detailed design'

88beb71

bestander suggested changes Oct 24, 2017

View reviewed changes

update download from different registry rfc

9639772

bestander approved these changes Oct 30, 2017

View reviewed changes

csvan mentioned this pull request Nov 2, 2017

Moving repositories without recreating yarn.lock? yarnpkg/yarn#4817

Closed

bestander merged commit 0461f38 into yarnpkg:master Nov 13, 2017

arcanis mentioned this pull request May 3, 2018

🦁 Yarn 2.0 Working Group yarnpkg/yarn#5773

Closed

5 tasks

Gudahtt mentioned this pull request Aug 19, 2018

Replace resolved field by hash #64

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add rfc about registry url in lock file #84

add rfc about registry url in lock file #84

vernak2539 commented Sep 21, 2017

bestander left a comment

bestander Oct 4, 2017

vernak2539 Oct 9, 2017

bestander Oct 4, 2017

bestander left a comment

bestander Oct 24, 2017

bestander Oct 24, 2017

bestander Oct 24, 2017

bestander Oct 24, 2017

bestander Oct 24, 2017

bestander Oct 24, 2017

vernak2539 Oct 25, 2017 •

edited

Loading

csvan Oct 26, 2017

bestander Oct 30, 2017

Remigius2011 Dec 12, 2017

kopax-polyconseil Sep 21, 2023

bestander commented Oct 30, 2017

bestander left a comment

BYK commented Nov 1, 2017

bestander commented Nov 2, 2017

imsnif commented Nov 2, 2017

bestander commented Nov 2, 2017

BYK commented Nov 2, 2017

bestander commented Nov 6, 2017

bestander commented Nov 13, 2017

arcanis commented May 3, 2018

arcanis commented May 3, 2018

rally25rs commented May 3, 2018

kopax-polyconseil commented Sep 21, 2023

		Let me know if this isn't enough, first time writing an RFC.


		As a dev that generated their lock file against REGISTRY_A


		# How We Teach This

		This would largely be in the internal workings, and wouldn't require teaching. npm 5 does this


		# Motivation

		We currently have two internal registries/mirrors. We generate the lock file against one we have in


		This, with the version already provided, could then be used to generate a URL on the fly or at the beginning of operations if an alternate registry is specified in the scenarios above.

		Or, the `resolved` field could change to not include the hostname/origin, which would then be inserted during operations. This would

		already, so it would be inherent.

		> If you generated your package lock against registry A, and you switch to registry B, npm will now try to install the packages from registry B, instead of A. If you want to use different registries for different packages, use scope-specific registries (npm config set @myscope:registry=https://myownregist.ry/packages/). Different registries for different unscoped packages are not supported anymore.

add rfc about registry url in lock file #84

add rfc about registry url in lock file #84

Conversation

vernak2539 commented Sep 21, 2017

bestander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bestander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vernak2539 Oct 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bestander commented Oct 30, 2017

bestander left a comment

Choose a reason for hiding this comment

BYK commented Nov 1, 2017

bestander commented Nov 2, 2017

imsnif commented Nov 2, 2017

bestander commented Nov 2, 2017

BYK commented Nov 2, 2017

bestander commented Nov 6, 2017

bestander commented Nov 13, 2017

arcanis commented May 3, 2018

arcanis commented May 3, 2018

rally25rs commented May 3, 2018

kopax-polyconseil commented Sep 21, 2023

vernak2539 Oct 25, 2017 •

edited

Loading