build: allow disabling snapshots for dev

[nayeemrmn]

Closes #19399 (running without snapshots at all was suggested as an alternative solution).

Adds a __runtime_js_sources pseudo-private feature to load extension JS sources at runtime for faster development, instead of building and loading snapshots or embedding sources in the binary. Will only work in a development environment obviously.

Try running cargo test --features __runtime_js_sources integration::node_unit_tests::os_test. Then break some behaviour in ext/node/polyfills/os.ts e.g. make function cpus() {} return an empty array, and run it again. Fix and then run again. No more build time in between.

Blocked by #20043.

cc @bartlomieju

[bartlomieju]

@nayeemrmn fantastic! I think I managed to find an edge case here.

If I do:

$ cargo build --features __runtime_js_sources

And then:

$ ./target/debug/deno

I get:

./target/debug/deno

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.36.0
Args: ["./target/debug/deno"]

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotPresent', ext/node/lib.rs:136:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The panic doesn't appear if I do cargo run --features __runtime_js_sources. Any ideas?

EDIT: The same problem happens if I actually run a script:

// foo.js
import process from "node:process";

console.log("process.arch", process.arch);

$ cargo run --features __runtime_js_sources -- run foo.js
process.arch arm64
$ ./target/debug/deno run foo.js
============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.36.0
Args: ["./target/debug/deno", "run", "foo.js"]

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotPresent', ext/node/lib.rs:136:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[nayeemrmn]

Strange, I'm not getting that error. From a quick glance at

deno/ext/node/lib.rs

Lines 135 to 136 in 5311f69

	std::env::var("TARGET")
	.unwrap()

why is it reading TARGET at runtime instead of env!()?

Ah yeah makes sense of course that won't work given what this mode is.

[bartlomieju]

Strange, I'm not getting that error. From a quick glance at

deno/ext/node/lib.rs

Lines 135 to 136 in 5311f69

	std::env::var("TARGET")
	.unwrap()

why is it reading TARGET at runtime instead of env!()?

This is the diagnostic I get if I try to use env!:

environment variable `TARGET` not defined at compile time
Cargo sets build script variables at run time. Use `std::env::var("TARGET")` instead

[bartlomieju]

It's strange you don't get that error - this op is called in 3 separate places by ext/node so you should definitely be calling it if executing some node: built-in module or npm: package.

[nayeemrmn]

Okay it's because the TARGET is defined in runtime/build.rs,deno_node is build before that.

[bartlomieju]

Okay it's because the TARGET is defined in runtime/build.rs,deno_node is build before that.

We should probably move all of that to deno_core and have build info attached there to Deno.core.build. We are currently using these env vars in several places :/

[bartlomieju]

@nayeemrmn this is absolutely fantastic. I've been playing with it for half an hour now and it's a game changer in terms of working on the JS APIs. I love how few changes you had to do, I'll release deno_core tonight after Matt finishes some stuff so we can land it ASAP.

[marvinhagemeister]

Just wanted to say that this is fantastic feature! Really looking forward to seeing this sometime in Deno 👍

[nayeemrmn]

$ ./target/debug/deno

By the way I could reproduce the error with target/debug/deno, using cargo run seems to initialise the cargo:rustc-env values on startup (for me only, still strange). Anyway I added target-lexicon from bytecodealliance to get the target triple at compile time ext/node/build.rs to set the var.

Just wanted to say that this is fantastic feature! Really looking forward to seeing this sometime in Deno 👍

Thanks!

[sigmaSd]

does this have any effect on cross compiling, as in can I now crosscompile by disabling snapshotting ?

[nayeemrmn]

does this have any effect on cross compiling, as in can I now crosscompile by disabling snapshotting ?

No, this snapshot-less mode creates a binary that only works on the development machine it was built on. That would require an alternative snapshot-less mode which bundles all the sources in the binary.

[bartlomieju]

LGTM, fantastic work @nayeemrmn, this will make working on JS/TS polyfills way faster! Kudos 🚀