Adding Node module for snowboy

[evancohen]

Authors

• @nekuz0r
• @evancohen

Info

This PR adds an npm package that uses node-gyp with Nan to create a JavaScript interface between the prebuilt snowboy native hotword detection and the Node runtime.

Using node-pre-gyp, we are able to host pre-compiled versions of the native addon for OSX, Linux x64, and the Raspberry Pi (arm), which makes using the package super easy!

More info: #4

Installation of this package uses a platform independent command:

npm install --save snowboy

Usage

There are two examples provided in this PR for streaming from a file and directly from a microphone. The short version of this is:

const record = require('node-record-lpcm16');
const {Detector, Models} = require('snowboy');

const models = new Models();

models.add({file: 'resources/snowboy.umdl', sensitivity: '0.5', hotwords : 'snowboy'});

const detector = new Detector({
  resource: "resources/common.res",
  models: models,
  audioGain: 2.0
});

detector.on('hotword', function (index, hotword) {
  console.log('hotword', index, hotword);
});

const mic = record.start({
  threshold: 0,
  verbose: true
});

mic.pipe(detector);

Requirements

Before accepting this PR, there are a few things that need to be taken care of. Pre-building the Nan extension is great so people don't have to compile on their own, but it requires some configuring on the snowboy end of things if you want it to continue to work:

• You'll have to enable Travis for snowboy if you want to leverage automatic building and publishing of the binary for Linux and OSX.
• I couldn't find any CI system that would work for ARM, so building and publishing for the Pi will have to be done manually :(
  • I have a bash script that you can use for this :)
• You'll have to add your own S3 bucket key and secret to .travis.yml (or you can give me the secure strings and I can add them to this PR)
  • Instructions for that here: https://github.com/mapbox/node-pre-gyp#s3-hosting
  • Once that's set up, any commit message containing [publish binary] or [republish binary] (see examples in my commit history) will publish/republish the native binary for the version specified in package.json.
• Give me an email address to add to https://www.npmjs.com/package/snowboy so you can publish updates to the package.

🚀

[chenguoguo]

Thanks a lot for the hard work guys! I'll take a closer look tomorrow.

[xuchen]

looks good to me! I had to additionally do npm install node-record-lpcm16 to install missing package.

[evancohen]

That might be worth including in the docs, but it's not a direct dependency of the module since you can pipe any arbitrary audio data (provided it's the correct format).

[chenguoguo]

I'll mention that in the docs. Will have to look into the pull request tomorrow, was working on some other stuff :-)

[chenguoguo]

I took a look at all the changed files, look good! I was thinking of squashing those commits into one, but on another thought, this contribution comes from two different developers, so let's keep those commits!

• I turned on Travis for Snowboy
• Could you share your bash script for raspberry pi?
• s3 bucket has been set up (I'll update after merging)
  • host: https://snowboy-release-node.s3-us-west-2.amazonaws.com/
  • secure strings generated using travis encrypt node_pre_gyp_accessKeyId=xxx and travis encrypt node_pre_gyp_secretAccessKey=yyy, where xxx and yyy are the actual keys. Let me know if this is not the right way.
• email: [email protected]

If things look good to you, I'm ready to merge! Thanks a million for the fantastic work @evancohen and @nekuz0r

[nekuz0r]

@chenguoguo @evancohen for me, commits can be squashed :)

[chenguoguo]

Thanks @nekuz0r ! I'll leave this to @evancohen . If we end up with squashing the commits, make sure you put @nekuz0r and @evancohen in the message so people know who contributed this commit :-)

[evancohen]

@chenguoguo I have the script sitting on my Pi at home, I can dig it up tonight. I'm also fine squashing the commits (or I can rebase on my side and squash everything into ~7 commits that make sense, instead of the 30 that we have).

Is that email address associated with an account on the npm site? I'm having trouble adding it as a collaborator.

I did one final sanity check (evancohen@e7d5472) incriminating the version number to 1.0.10, publishing the npm package, and running the builds on Travis to publish them:
https://travis-ci.org/evancohen/snowboy/builds/161123785

Everything looks green on my end!

There's one open question from @Mexxxo on evancohen#1 about how we name the event handlers...
tl;dr If snowboy uses Voice Activity Detection then we shouldn't use an event named "noise" it should be called "voice". Thoughts?

[chenguoguo]

@evancohen yes please rebase and squash if that's not too much trouble for you. Squashing to 7 sounds good!

Sorry about the email, please try again, the user name is kitt.ai.

For the VAD we have in Snowboy, it's more like silence v.s. non-silence. voice probably makes more sense than noise.

[evancohen]

I've rebased my branch and squashed both of our commits into larger changes, hopefully that's ok with @nekuz0r.

@chenguoguo I've added that email as a collaborator to the npm package.

As for VAD, if it's actually "non-silence" then noise and sound would make sense - My concern for using voice is if someone assumes that any time that event fires someone is actually speaking (and not just moving a chair or something). Happy to make this change either way, I just want to ensure that it makes sense to people :)

Edit: Also, it looks like you enabled builds for pull requests which leaves you vulnerable to someone hijacking the binary (If someone made a PR with [republish binary] in the commit then it would overwrite the existing binary in S3). Since Travis is only used for auto-publishing binaries I suggest that you either disable Travis for PRs, or I can remove the ability to republish the binary.

[chenguoguo]

@evancohen thanks, looks very clean now!

Let's go with sound then, noise itself is not well defined, people have different opinions on ,e.g., treating voice as noise or treating music as noise. If you are going to make the change, I'll wait for that and merge.

Good catch on the PR trigger! I turned it off.

[evancohen]

My auto-publish script for the pi: node-pre-gyp_publish.sh

All set 🚀

[chenguoguo]

I updated snowboy.umdl so it works better in noisy environment; also added alexa.umdl although it doesn't work as well as snowboy. I think I'm ready to release v1.1.0 now.

@evancohen what extra steps does it take to publish the node module to npmjs.com? I've never done this by myself.

[nekuz0r]

@chenguoguo npm publish into the repository directory should enough

[evancohen]

That should do it. I would recommend publishing the binaries for 1.1.0 first if you haven't already.

[chenguoguo]

I did a test publish with 1.0.11 and everything seems to be working. I'll bump it to 1.1.0 tomorrow after publishing the binaries (don't have a Raspberry Pi by hand now).

[lynxaegon]

Could this be tweaked to work in browser ?
I haven't looked how you use the microphone or how you read the mdl / pmdl files.

[arilotter]

@lynxaegon Not easily, since all the node module does is wrap a compiled library for some platform. Theoretically, it might be possible in the future for the kitt.ai team to compile the snowboy library for something like asm.js, but that part of the detection is closed source as of now.

[lynxaegon]

Ok, so i'm stuck with a python detector + a long poll request to detect a hotword.
Thanks for the quick reply 👍

[arilotter]

On the other hand, nothing's stopping you from running the node.js code on your server & streaming browser audio to it. Looks like that's what you're already doing with Python, though

[nekuz0r]

@lynxaegon

@arilotter beat me on this, indeed you can stream your browser audio capture to a node server via websocket and get the detected keyword back via the same channel :)

[nick-jonas]

Is there anyway to import this into an electron project? I get this error:

Error: Cannot find module '/Users/njonas/Code/<project_dir>/node_modules/snowboy/lib/node/binding/Release/electron-v1.4-darwin-x64/snowboy.node'

...when node-v48-darwin-x64 is the one node binding that gets installed.

[evancohen]

@nick-jonas yup! It's for Sonus, but it'd be easy to adapt to use just for snowboy: https://github.com/evancohen/sonus-electron-boilerplate

[nick-jonas]

@evancohen that's great! I get the same error though :(

Uncaught Error: Cannot find module '/Users/njonas/Downloads/sonus-electron-boilerplate-master/node_modules/snowboy/lib/node/binding/Release/electron-v1.6-darwin-x64/snowboy.node'

[evancohen]

@nick-jonas So you ran through the install steps in the README and the boilerplate didn't work?

It's possible that I inadvertently broke it when trying to fix something for ARM evancohen/sonus-electron-boilerplate@3a5bd4e.

If that's the case then you can check out the parent commit before that change and rm -r node_modules then go through the install steps again.

[nick-jonas]

Changing this in package.json did the trick (from ^1.4):

"devDependencies": {
    "electron": "1.4.x"
  }

[evancohen]

What version do you have installed now? npm ls electron

[nick-jonas]

Electron version = 1.4.15.

Also, I'm not totally in the clear as I now get this error:

[evancohen]

@nick-jonas looks like you didn't install sox https://github.com/evancohen/sonus#for-macos

[nick-jonas]

ahh right that was it, thanks @evancohen

[Wqrld]

https://gist.github.com/Wqrld/a34ee45c8cf8edefad58eae5e33cdd7a it seems like the prebuilt files are inaccessible and compiling also doesnt work on ubuntu 16.04 Intel

Edit because i dont want to email everyone again: make sure to download the correct cblas lib, there are multiple on apt but only one of them works, i believe its the dev one

[nekuz0r]

@Wqrld Your build seems to failed because the lib cblas can't be found on your system, install it and try again.
You might need to compile it by yourself.
https://www.netlib.org/blas/