feat(std/hash): reimplement all hashes in WASM #6292

skdltmxn · 2020-06-14T17:54:09Z

This PR is renewal of #6046.
WASM implementation is at https://github.com/skdltmxn/deno-hash

skdltmxn · 2020-06-14T17:57:39Z

Benchmark shows that WASM version is slower than native Rust but is faster than JS version.

10mb

Algorithm	JS	Rust	WASM
md5	227	75	125
sha1	256	76	162
sha224	400	105	202
sha256	392	101	203
sha512	339	84	174
sha3-224	273	88	197
sha3-256	282	91	201
sha3-384	328	101	239
sha3-512	418	118	281

100mb

Algorithm	JS	Rust	WASM
md5	1751	299	501
sha1	1961	309	904
sha224	3368	555	1301
sha256	3338	553	1298
sha512	2697	382	1026
sha3-224	1938	439	1275
sha3-256	2009	453	1338
sha3-384	2480	546	1546
sha3-512	3324	721	2250

kitsonk · 2020-06-15T05:38:07Z

I haven't dug into the PR in depth, but this is awesome! Not adding to the Deno binary, but providing hot path performance improvement. It also proves that WebAssembly doesn't perform better than Rust, so it isn't suitable for all hot paths, but it feels suitable for this one, as it is something that isn't part of the core of Deno, so it is a good mediation between all the options.

caspervonb · 2020-06-15T08:17:39Z

Generally speaking, pure computation that doesn't trampoline between JS/WASM in a hot path should be very close native code once v8 has warmed up so these measurements look a little off at first glance.

Based on the previous pull request, assuming these benchmarks are done the same way; the benchmarks seem to be including startup time?

def run_benchmark(deno_exe, method, input_file):
     # compile
     subprocess.call([deno_exe, "run", "cli/tests/hash/hash.ts"])

     for alg in algorithms:
         args = [
             deno_exe, "run", "--allow-read", "cli/tests/hash/hash.ts", method,
             alg, input_file
         ]

         start = time.time()
         subprocess.call(args)
         elapsed = time.time() - start

         print "[{}] {}ms".format(alg, int(elapsed * 1000))

It would be more accurate if the measurement did not include the time it takes for a WebAssembly module to be decoded from base64, compiled to a module and instantiated as an instance.

skdltmxn · 2020-06-15T15:03:07Z

@caspervonb Thank you for the tips! New benchmark for pure computation actually shows that WASM version is pretty fast!

10mb

Algorithm	JS	Rust	WASM
md5	199	75	35
sha1	196	76	75
sha224	352	105	116
sha256	328	101	115
sha512	281	84	88
sha3-224	213	88	110
sha3-256	223	91	115
sha3-384	274	101	150
sha3-512	352	118	191

100mb

Algorithm	JS	Rust	WASM
md5	1539	299	329
sha1	1819	309	721
sha224	3180	555	1120
sha256	3127	553	1151
sha512	2584	382	855
sha3-224	1794	439	1038
sha3-256	1884	453	1089
sha3-384	2337	546	1384
sha3-512	3188	721	1930

ry

@skdltmxn I agree with Kitson, this is awesome! Thanks for this work.

I would like to have the source code of how the wasm is generated in the tree - even tho we check-in the output. This will allow us to update and maintain it without depending on your repository. So could you please copy the files in https://github.com/skdltmxn/deno-hash over into std/hash/_wasm ?

skdltmxn · 2020-06-15T16:30:55Z

@ry Sure. That's a good idea. Would it be okay to copy whole repo into std/hash/_wasm?

ry · 2020-06-15T16:44:28Z

@skdltmxn Yes, ideally removing as many files as possible and with a readme that gives the explicit commands to run in order to regenerate the wasm file

marcosc90 · 2020-06-15T17:23:55Z

std/hash/_wasm/hash.ts

+ #hash: DenoHash;
+ #digested: boolean;
+
+ constructor(algorithm: string) {


Should this be changed to: algorithm: SupportedAlgorithm instead of algorithm: string?

And you can remove as string

deno/std/hash/mod.ts

Lines 32 to 34 in b59bab3

export function createHash(algorithm: SupportedAlgorithm): Hasher {

return new Hash(algorithm as string);

}

In order to use SupportedAlgorithm in hash.ts, I have to import mod.ts.
But mod.ts is also importing hash.ts, so I thought it would make cycle.

In architectural point of view, I think Hash class doesn't need to know about SupportedAlgorithm since Hash is just an wrapper for WASM which only uses string.
SupportedAlgorithm would do enough of its job for guarding algorithm for createHash defined in mod.ts.

skdltmxn · 2020-06-15T17:45:40Z

@ry I have copied deno-hash repo into std/hash/_wasm. Please check README for build!

Cargo.toml

ry · 2020-06-15T21:50:25Z

std/hash/_wasm/build.ts

+// Copyright 2018-2020 the Deno authors. All rights reserved. MIT license.
+
+import { encode as base64Encode } from "../../encoding/base64.ts";
+import Terser from "https://cdn.pika.dev/terser@^4.7.0";


What does terser do?

I see - it's a minifier. Is it possible to remove this dependency? Presumably minification doesn't help much?

In std we strive to not have any external dependencies: https://deno.land/manual/contributing/style_guide#minimize-dependencies-do-not-make-circular-imports
In this case, build.ts isn't part of std, it's just a script for generating the wasm, but I think it would be prudent for us to not rely on external code if possible.

Also I'm getting an error when running this:

> deno run --allow-read --allow-write --allow-run ./build.ts Download https://cdn.pika.dev/terser@^4.7.0 Download https://cdn.pika.dev/-/[email protected]/dist=es2019,mode=types/index.d.ts Download https://cdn.pika.dev/-/[email protected]/dist=es2019/terser.js Download https://cdn.pika.dev/-/[email protected]/dist=es2019,mode=types/tools/terser.d.ts Download https://cdn.pika.dev/-/[email protected]/dist=es2019/source-map.js Download https://cdn.pika.dev/-/[email protected]/dist=es2019,mode=types/tools/terser.d.ts Download https://cdn.pika.dev/-/[email protected]/dist=es2019,mode=types/source-map.d.ts Download https://cdn.pika.dev/-/[email protected]/dist=es2019,mode=types/index.d.ts Compile file:https:///Users/rld/src/deno/std/hash/_wasm/build.ts error: Uncaught SyntaxError: Invalid or unexpected token at https://cdn.pika.dev/-/[email protected]/dist=es2019/terser.js:10:342617

It should make a huge difference if we are targeting wasm. Mostly it would be shortening variable names, the rest simply shouldn't matter in Web Assembly.

Adding minification support into Deno longer term is an objective, but we would want that to come from Rust.

I will remove minifier :)

skdltmxn · 2020-06-16T13:09:47Z

@ry Removed Terser from WASM build script.

I'm still not sure whether it is better to include WASM project into root Cargo.toml or not since we don't have to build WASM everytime and building WASM is done using deno itself.

ry · 2020-06-16T13:22:38Z

@skdltmxn I'm not sure either. Let's exclude it for now. We'll have to develop our own best practices for managing wasm in std - thanks for blazing a path : )

ry · 2020-06-16T13:31:23Z

~~So I ran the build.ts script, and it was successful, but it creates an out directory:~~

> ls out README.md deno_hash.js deno_hash_bg.wasm deno_hash.d.ts deno_hash_bg.d.ts package.json

It seems there's probably a final step:

cp out/deno_hash.js ./wasm.js

~~Is this correct? If so, can you please update the readme?~~

ry · 2020-06-16T13:32:54Z

Oops, nope. It seems the build.ts does that. Sorry.

However, I noticed in build.ts it uses an absolute URL: https://deno.land/[email protected]/encoding/base64.ts

can you change this to be a relative URL?

skdltmxn · 2020-06-16T14:56:51Z

@ry All fixed!

ry

LGTM - thanks @skdltmxn !

I think we might have to continue iterating on this for a bit - but this is good enough for now

skdltmxn mentioned this pull request Jun 14, 2020

feat(std/hash): reimplement all hashes in rust #6046

Closed

ry requested changes Jun 15, 2020

View reviewed changes

marcosc90 reviewed Jun 15, 2020

View reviewed changes

ry reviewed Jun 15, 2020

View reviewed changes

Cargo.toml Show resolved Hide resolved

ry reviewed Jun 15, 2020

View reviewed changes

skdltmxn force-pushed the std-hash-wasm branch 2 times, most recently from 5975b66 to 784896b Compare June 16, 2020 12:51

feat(std/hash): reimplement all hashes in WASM

10171de

skdltmxn force-pushed the std-hash-wasm branch from 784896b to 10171de Compare June 16, 2020 14:21

bartlomieju added the std label Jun 16, 2020

ry approved these changes Jun 16, 2020

View reviewed changes

ry merged commit b3c72d1 into denoland:master Jun 16, 2020

skdltmxn deleted the std-hash-wasm branch June 17, 2020 01:00

oplik0 mentioned this pull request Jun 21, 2020

Start the move from typescript to web assembly denorg/scrypt#2

Closed

caspervonb mentioned this pull request Jun 26, 2020

perf(std/hash): optimize for speed (O3) #6499

Merged

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 21, 2021

feat(std/hash): reimplement all hashes in WASM (denoland/deno#6292)

25f86c7

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

cb96643

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

b85ebf2

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 24, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

65cbe59

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

1c48fa5

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

6ab1164

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

6714b09

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Jan 31, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

390a507

caspervonb pushed a commit to caspervonb/deno_std that referenced this pull request Feb 1, 2021

feat(hash): reimplement all hashes in WASM (denoland/deno#6292)

ef7576c

jeremyBanks mentioned this pull request Jul 12, 2021

feat(crypto): add std/crypto wrapping and extending runtime WebCrypto denoland/deno_std#1025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(std/hash): reimplement all hashes in WASM #6292

feat(std/hash): reimplement all hashes in WASM #6292

skdltmxn commented Jun 14, 2020

skdltmxn commented Jun 14, 2020

kitsonk commented Jun 15, 2020

caspervonb commented Jun 15, 2020 •

edited

Loading

skdltmxn commented Jun 15, 2020

ry left a comment

skdltmxn commented Jun 15, 2020

ry commented Jun 15, 2020

marcosc90 Jun 15, 2020

skdltmxn Jun 15, 2020 •

edited

Loading

skdltmxn Jun 15, 2020 •

edited

Loading

skdltmxn commented Jun 15, 2020

ry Jun 15, 2020

ry Jun 15, 2020

ry Jun 15, 2020

kitsonk Jun 15, 2020

skdltmxn Jun 15, 2020

skdltmxn commented Jun 16, 2020

ry commented Jun 16, 2020

ry commented Jun 16, 2020 •

edited

Loading

ry commented Jun 16, 2020

skdltmxn commented Jun 16, 2020

ry left a comment

	export function createHash(algorithm: SupportedAlgorithm): Hasher {
	return new Hash(algorithm as string);
	}

feat(std/hash): reimplement all hashes in WASM #6292

feat(std/hash): reimplement all hashes in WASM #6292

Conversation

skdltmxn commented Jun 14, 2020

skdltmxn commented Jun 14, 2020

10mb

100mb

kitsonk commented Jun 15, 2020

caspervonb commented Jun 15, 2020 • edited Loading

skdltmxn commented Jun 15, 2020

10mb

100mb

ry left a comment

Choose a reason for hiding this comment

skdltmxn commented Jun 15, 2020

ry commented Jun 15, 2020

marcosc90 Jun 15, 2020

Choose a reason for hiding this comment

skdltmxn Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

skdltmxn Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

skdltmxn commented Jun 15, 2020

ry Jun 15, 2020

Choose a reason for hiding this comment

ry Jun 15, 2020

Choose a reason for hiding this comment

ry Jun 15, 2020

Choose a reason for hiding this comment

kitsonk Jun 15, 2020

Choose a reason for hiding this comment

skdltmxn Jun 15, 2020

Choose a reason for hiding this comment

skdltmxn commented Jun 16, 2020

ry commented Jun 16, 2020

ry commented Jun 16, 2020 • edited Loading

ry commented Jun 16, 2020

skdltmxn commented Jun 16, 2020

ry left a comment

Choose a reason for hiding this comment

caspervonb commented Jun 15, 2020 •

edited

Loading

skdltmxn Jun 15, 2020 •

edited

Loading

skdltmxn Jun 15, 2020 •

edited

Loading

ry commented Jun 16, 2020 •

edited

Loading