Deno should support gzipped and (or) git repo dependencies fetch

[CGQAQ]

FWIK, Deno currently fetch dependencies only support download ts/js file(s) individually, it's not a good idea for middle or even large project dependencies fetch(hundreds of or even thounds of dependencies).
So I think it's a good idea to support gzipped file through deno fetch https://foo.gz which contains the mod.ts file in the root and (or) git repo through deno fetch https://foo.git just like golang go get

[ry]

Gzip encoding is certainly something we should support. git repo fetch is much more questionable since it's not a browser standard.

[CGQAQ]

@ry But it's so convenient, git project does not need pack to gzip, or release anymore, just add a mod.ts in the git repo root directory. Deno just need to clone it and done just like golang.
And, plus we will be able to just using github.com/username/projectname/path/to/ts as import url, instead of some https://raw.githubusercontent.com/ urls
But long urls could be solved by import map, so nevermind :)

[IllusionPerdu]

for the gzip file it should be great we can call the file we want :
deno fetch https://foo.gz**@thefile.ext**
And the import of this file can read the other file in the gzip ;)

[CGQAQ]

for the gzip file it should be great we can call the file we want :
deno fetch https://foo.gz**@thefile.ext**
And the import of this file can read the other file in the gzip ;)

deno fetch will extract that gzipped mod file into cache folder, so you don't need special syntax like <url>**@filename.ext**, just use import '<url>/filename.ext' inside script will be ok

[IllusionPerdu]

The logic of deno is : don't make magic resolution. With the syntax i write before the gzip become a virtual folder and we use the file normaly we can call another file than mod.ts or for exemple we can gzip all the standard lib and call anything.

[CGQAQ]

The logic of deno is : don't make magic resolution

But deno is also web compatible, es modules do not like that special syntax. deno fetch domin/foo.gz should just straightforward exact and cache it into domin folder
deno fetch domin/foo.gz just do fetch and ungzip, and that's it. es modules just treat foo as folder.
e.g. import 'https://domin/foo/mod.ts'

for exemple we can gzip all the standard lib and call anything.

yes you can, if you have std.gz, just do deno fetch std.gz and you can use file in it import 'std/foo.ts'.
I'm not say you could only use mod.ts inside a gzipped mod file, I just say gzipped mod should have a mod.ts in root directary to describing meta data of the mod

And if you direct import a file inside a gzipped fileimport 'https://domin/foo/mod.ts' without fetch first, there are two ways to parse this:

1. just throw url content does not exsit error
2. import http(s):https://domin/foo/mod.ts, auto fetch http(s):https://domin/foo.gz and ungzip, then import mod.ts inside it

[IllusionPerdu]

I think it's easier with my solution to specify which file to use when importing it avoids making unnecessary requests that make errors.

Deno will actually upload the gzip file to the cache but with the syntax https://[email protected] we could use any file in the gzip. And we could write a script like:
main.ts

import {anything} from https://example.com/[email protected]
console.log (anything ());

See even put several libraries in the same gzip and access to each of them with yours script

[CGQAQ]

I think it's easier with my solution to specify which file to use when importing it avoids making unnecessary requests that make errors.

Deno will actually upload the gzip file to the cache but with the syntax https://[email protected] we could use any file in the gzip. And we could write a script like:
main.ts

import {anything} from https://example.com/[email protected]
console.log (anything ());

See even put several libraries in the same gzip and access to each of them with yours script

yours and mine solution are both need to cache whole gzip uncompressed content, the only diffrence is mine is standard es modules syntaximport 'http(s):https://example.com/mod_name/mod_file.ext', yours are es modules with a special syntax import 'http(s):https://example.com/mod_name.gzip@mod_file.ext'

[IllusionPerdu]

The two solutions have a lot in common, however with your solution this requires either a special configuration of the server or making unnecessary requests. For the example of the std lib you would need an additional mod.ts to re-export all mod.ts from the subdirectories or make one gzip by subdirectory.

In addition my solution would allow the developer for example to provide a gzip of a library with which they are sure it works and make in their script a:
import {function} from "./lib/[email protected]"
...
and therefore it simplifies the distribution of their program.

[CGQAQ]

import {function} from "./lib/[email protected]"

@ is alreay indicates version in deno, so I think # is another good choice in your solution.
But I still think there should be a mod.ts in root to indicates it is whether or not a gzipped mod file, and mod.ts should also contains mod author(s) current version and others meta-data

if you are trying fetch a vanila(a gzipped file w/o any code, or even contains malwares) gzipped file, deno should deny to fetch(actually deny to cache it after fetch, decompress and parse) it

[IllusionPerdu]

Yes it can be another symbol ./file.gzip#script.ts

For security reasons it may be necessary to add a command to Deno: deno gzip ./script.ts which allows you to create a gzip from the cache to run the script.ts and which adds a description file to gzip with the SHA1 of each file.

[rsp]

@CGQAQ @IllusionPerdu
By "gzip file" do you mean a tarball? Because gzip can only compress a single file (stream), not multiple files. Also, the gzip encoding that I think @ry is talking about is about the HTTP Content-Encoding header, see:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

that could compress files transparently on the server that supports it - and GitHub does, see:

$ curl -sI https://raw.githubusercontent.com/rsp/deno-clipboard/master/mod.ts \
  -H 'Accept-Encoding: gzip' | egrep 'Encoding|Length'
Content-Encoding: gzip
Content-Length: 801
Vary: Authorization,Accept-Encoding

vs:

$ curl -sI https://raw.githubusercontent.com/rsp/deno-clipboard/master/mod.ts \
  -H 'Accept-Encoding: deflate' | egrep 'Encoding|Length'
Content-Length: 2117
Vary: Authorization,Accept-Encoding

GitHub seems to use gzip even when asked to use encoding with a name that only contains "gzip":

$ curl -sI https://raw.githubusercontent.com/rsp/deno-clipboard/master/mod.ts -H 'Accept-Encoding: NOTgzipButMatches' | egrep 'Encoding|Length'
Content-Encoding: gzip
Content-Length: 801
Vary: Authorization,Accept-Encoding

but it uses the gzip nonetheless, and it will send compressed files when deno requests them with Accept-Encoding: gzip header. Or maybe it already does? Let me check. Yes, it does. See:

$ nc -l 3345 & deno run <(echo 'import {} from "http:https://localhost:3345"')
[1] 67702
Download http:https://localhost:3345/
GET / HTTP/1.1
user-agent: Deno/0.25.0
accept: */*
accept-encoding: gzip
host: localhost:3345

So, since Deno already uses Accept-Encoding: gzip when requesting imported files, and since GitHub serves files with Content-Encoding: gzip when there is Accept-Encoding: gzip in the request, then it seems that Deno already uses gzip when downloading files. It doesn't extract files from tarballs but it supports gzip compression in transit (it could also use gzip for storing caches if it makes sense but probably it doesn't).

Now, if you want Deno to be able to use dependencies in compressed tarballs (.tar.gz, .tzg, .tar.bz2, .tbz etc.) then I'm not sure it would be worth it because you would need to download the entire project even to use a single file.

The same for git repos. Cloning a repo takes much more time than fetching a single or even multiple files, and you need to clone the entire repo even if you then want to use a single or few files.

Using HTTP and importing individual files gives you something like a file-level tree shaking, i.e. you download only the files that are actually needed. With a convention of using dep.ts that groups all external imports you could even go far with the optimization of what to download before you start (you could even include files that you know will be needed by the files you need directly (especially with lock files, see #200 and #3231) so that they are downloaded in one go using Connection: keep-alive or HTTP/1.1 pipelining or HTTP/2 multiplexing if (when) supported by Deno, done completely transparently to the imports in the program).

For all download performance optimizations that can be done transparently without changing the import syntax, see:

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview

Using files that are explicitly gzipped individually, something like local x.ts.gz files, may or may not be useful but it is not a browser standard and the utility would probably be questionable if the files can already be compressed in transit and they could easily be compressed in the Deno cache if that is something that proves to make sense as a size/performance tradeoff, but it can be done transparently to programs and modules, just like the HTTP 1.1/2/3 features.

Using git repos in imports is something that looks nice (cute?) but in practice there are few issues:

• not a browser standard
• implementation complexity
• not actual URLs
• performance of cloning a repo vs downloading files
• overkill of getting all files in the repo just to use a few
• caching of entire repos instead of single files
• multiple protocols - https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
• a precedent to support more version-control systems like SVN, Mercurial, Bazaar etc.

The only upside that I see right now is being able to use dependencies in private repos using SSH but this can already be done using submodules.

To sum it up. Deno already supports gzip downloads, cloning entire repos over SSH doesn't seem as the most efficient way to get individual files that are already available via HTTP, getting files from tarballs may be interesting for things like having compressed dependencies or all files bundled, like distributing two files:

• main.ts
• main.tar.gz

and having main.ts contain only:

import { main } from './main.tar.gz#main.ts';
main();

Or maybe even running it directly with deno run 'main.tar.gz#main.ts'

Seems quite nice for the distribution of applications, actually. Now, if it only treated main.ts magically and used it by default when running tarballs then it could be just deno run main.tgz but this "magically" is probably against the design goals of Deno, and also this would duplicate the functionality of Deno bundles (see #2357 and #2475) - for the bundles it might be nice to support gzip (not tar) compression since a bundle is already a single file and is likely to be large.

[CGQAQ]

By "gzip file" do you mean a tarball?

Yes, I did. I did mean package many files in one tarball

[IllusionPerdu]

Yes i think also the tarball (or zip file)

[kitsonk]

My opinion... When fetching remote modules, Deno should support both gzip and brotli encoding. They are the two most common forms of encoding.

As far as grouping multiple modules together, I think further enhancements to bundling are the best solution to that, where both the .js bundle and the .d.ts file are generated during deno bundle and then can be used to re-distribute "libraries" of Deno modules.

[fakoua]

Can deno bundle arg resolve this problem? Not the gzip issue but multiple files.

[kitsonk]

@fakoua I am not sure what you mean... deno bundle creates a single file module out of whatever modules that root module depends on.

[CGQAQ]

@fakoua I am not sure what you mean... deno bundle creates a single file module out of whatever modules that root module depends on.

He probably meant that deno should provide a bundle feature to create a gzipped bundle file that contains multiple ts/js modules

[kitsonk]

He specifically said "not the gzip" issue though... so the bundle feature already does that.

[CGQAQ]

He specifically said "not the gzip" issue though... so the bundle feature already does that.

Oh! Didn't see that, sorry :)

[solson]

Deno supports downloading with HTTP gzip and brotli compression thanks to #3597.

As noted previously, archiving multiple files together is orthogonal to the question of compression, so I think it might be clearest to close this issue and open a new one focused on downloading archives (or other alternatives to deno bundle) if anyone wants to continue that conversation.