Deno v1.14: readFileSync returns an Uint8Array whose underlying buffer is null-terminated

[elisee]

Hi!

The underlying ArrayBuffer of the Uint8Array returned by Deno.readFileSync() is null-terminated in v1.14 but wasn't in v1.13.
Since this change just caused a bug in my project, I was wondering if this is an expected change (and I need to update my code) or whether it should be fixed in Deno?

const content = Deno.readFileSync("test.txt");
console.log(content.length, content.buffer.byteLength, content.length === content.buffer.byteLength);

test.txt can be just the letter a for instance.

In Deno v1.13, this prints: 1 1 true (buffer contains "a")
In Deno v1.14, it now prints: 1 2 false (buffer contains "a\0")

[autoxbc]

It looks like we have the same problem. The version where the bug first appeared was 1.14.1, and everything is normal in 1.14.0

When I tried to use @swc/wasm-web to create a transcoding server, an exception occurred when importing the wasm file

error: Uncaught (in promise) CompileError: WebAssembly.instantiateStreaming(): unexpected end of stream @+17187927

I was not sure whether it was a deno or swc bug before, now I’m sure it’s a deno problem, both Deno.readFile and Deno.readFileSync are affected.

[bnoordhuis]

#12057 is the change that's responsible but I wouldn't call it a bug. The only guarantee is that the Uint8Array is a sub-view of its backing ArrayBuffer, not that they're a 1-to-1 match.

edit: what are the odds of two people commenting at exactly same time after 5 days...

[autoxbc]

The only guarantee is that the Uint8Array is a sub-view of its backing ArrayBuffer, not that they're a 1-to-1 match.

const buffer = new ArrayBuffer(42);
const uint8 = new Uint8Array(buffer);
console.log( uint8.length , buffer.byteLength , uint8.length === buffer.byteLength );
//  42  42  true

They are 1-to-1 match, the problem is only in Deno.readFile and Deno.readFileSync.

[andreubotella]

In general, you can't expect a Uint8Array's backing ArrayBuffer to be a one-to-one match:

const buffer = new ArrayBuffer(1024);
const view = new Uint8Array(buffer, 0, 42);
yourCode(view);

And @autoxbc in particular seems to be creating a Response object from the ArrayBuffer to pass to WebAssembly.instantiateStreaming(). But you could do that with a Uint8Array object just fine:

const {module, instance} = await WebAssembly.instantiateStreaming(
  await new Response(await Deno.readFile("/path/to/file.wasm"))
);

(Not to mention that, if you're not actually fetching a wasm module from the network, you should instead use WebAssembly.instantiate() instead:)

const {module, instance} = await WebAssembly.instantiate(await Deno.readFile("/path/to/file.wasm"));

[elisee]

#12057 is the change that's responsible but I wouldn't call it a bug. The only guarantee is that the Uint8Array is a sub-view of its backing ArrayBuffer, not that they're a 1-to-1 match.

Right, I updated my own code to work with the Uint8Array itself (and handle the offset & length as needed) instead of grabbing the underlying ArrayBuffer and assuming it can be used in full.

Feel free to close if you consider the previous behavior an implementation detail that users shouldn't rely on, seems fair enough to me.

[autoxbc]

In general, you can't expect a Uint8Array's backing ArrayBuffer to be a one-to-one match

const input = Deno.readFileSync('./test.js');
const { buffer } = input ;
const output = new Uint8Array(buffer);
console.log( input.length === output.length );
// false

This caused confusion.

@swc/wasm-web uses fetch() to import the wasm module, so I have to construct a URL to complete the initialization, in this way: Uint8Array => ArrayBuffer => Blob => URL, unable to bypass ArrayBuffer.

Slice out a new ArrayBuffer from the existing ArrayBuffer does work, but it is not elegant. Many apis of Deno return Uint8Array. If I can’t expect its consistency with ArrayBuffer, I’m afraid it’s a disaster.

[andreubotella]

@swc/wasm-web uses fetch() to import the wasm module, so I have to construct a URL to complete the initialization, in this way: Uint8Array => ArrayBuffer => Blob => URL, unable to bypass ArrayBuffer.

@swc/wasm-web uses fetch() if the input to init() is a string, a URL or a Request – but it seems like you can pass it a Uint8Array just fine.

[autoxbc]

@swc/wasm-web uses fetch() if the input to init() is a string, a URL or a Request – but it seems like you can pass it a Uint8Array just fine.

OK, it works, thank you.