Add support of speech recognition for microphone node (node-red#52)

* update speech recognition * add support of speech recognition for microphone node * update speech recognition & add examples * add description of duration=0 * use done property instead of second output port
hardillb · Dec 28, 2020 · 31d45a1 · 31d45a1
1 parent 8913b58
commit 31d45a1
Show file tree

Hide file tree

Showing 10 changed files with 648 additions and 87 deletions.
diff --git a/node-red-node-ui-microphone/README.md b/node-red-node-ui-microphone/README.md
@@ -1,7 +1,7 @@
 node-red-node-ui-microphone
 ============================
 
-A Node-RED UI widget node that allows audio to be recorded from the dashboard.
+A Node-RED UI widget node that allows audio to be recorded and allows speech recognition from the dashboard.
 
 ## Install
 
@@ -14,23 +14,29 @@ Either use the Editor - Menu - Manage Palette - Install option, or run the follo
 The node provides a single button that, when clicked, will begin to capture audio.
 
 This node provides a single button widget in the dashboard that, when pressed,
-will begin to capture audio.
-The button can be configured in two modes.
+will begin to capture audio or speech recognition.
+The button can be configured in two modes for audio capture mode.
 
  - Click begins to capture audio, and then continues to capture audio until the button is pressed again, or it reaches its configured maximum duration.
  - Or the button can be configured to record only while the button is pressed.
 
-The audio is captured in WAV format and published by the node as a Buffer object.
+For audio capture mode, the audio is captured in WAV format and published by the node as a Buffer object.
 This can be written straight to a file or passed to any other node that expects
 audio data.
 
 ## Browser Support
 
-This node will work in most modern browsers as it uses the standard MediaRecorder API.
+For audio capture mode, this node will work in most modern browsers as it uses the standard MediaRecorder API.
 
  - IE : not supported
  - Safari : MediaRecorder needs to be enabled (Develop -> Experimental Features -> MediaRecorder)
 
+For speech recognition mode, this node will work in more restricted browsers as it uses SpeechRecognition API.
+
+ - IE, Safari : not supported
+ - Firefox : SpeechRecognition needs to be enabled (about:config -> `media.webspeech.recognition.enable`).
+ - Chrome : Supported
+
 If you are accessing the dashboard remotely (not via `localhost`), then you must
 use HTTPS otherwise the browser will block access to the microphone.
 
@@ -45,3 +51,7 @@ has given their permission.
 ## Notices
 
 This node uses `recorderWorker.js` Copyright © 2013 Matt Diamond, under the MIT License
+
+## Example
+
+See examples under Node-RED editor menu of *Import/Examples/node-red-node-ui-microphone*.
diff --git a/node-red-node-ui-microphone/examples/01 - record and play audio.json b/node-red-node-ui-microphone/examples/01 - record and play audio.json
@@ -0,0 +1,179 @@
+[
+ {
+ "id": "5b5a51f.aaa1eb",
+ "type": "change",
+ "z": "80638a41.e07d68",
+ "name": "",
+ "rules": [
+ {
+ "t": "set",
+ "p": "audio",
+ "pt": "flow",
+ "to": "payload",
+ "tot": "msg"
+ }
+ ],
+ "action": "",
+ "property": "",
+ "from": "",
+ "to": "",
+ "reg": false,
+ "x": 560,
+ "y": 240,
+ "wires": [
+ []
+ ]
+ },
+ {
+ "id": "98386b47.1f9578",
+ "type": "ui_button",
+ "z": "80638a41.e07d68",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "order": 2,
+ "width": "3",
+ "height": "1",
+ "passthru": false,
+ "label": "Play",
+ "tooltip": "",
+ "color": "",
+ "bgcolor": "",
+ "icon": "",
+ "payload": "audio",
+ "payloadType": "flow",
+ "topic": "",
+ "x": 390,
+ "y": 360,
+ "wires": [
+ [
+ "f03beedb.7ca0d"
+ ]
+ ]
+ },
+ {
+ "id": "f03beedb.7ca0d",
+ "type": "ui_audio",
+ "z": "80638a41.e07d68",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "voice": "Google US English",
+ "always": "",
+ "x": 540,
+ "y": 360,
+ "wires": []
+ },
+ {
+ "id": "6a2eefef.95866",
+ "type": "ui_text",
+ "z": "80638a41.e07d68",
+ "group": "a9cf3f4b.a6172",
+ "order": 5,
+ "width": 0,
+ "height": 0,
+ "name": "",
+ "label": "text→",
+ "format": "{{msg.payload}}",
+ "layout": "row-left",
+ "x": 530,
+ "y": 280,
+ "wires": []
+ },
+ {
+ "id": "b3eef9ed.456698",
+ "type": "switch",
+ "z": "80638a41.e07d68",
+ "name": "",
+ "property": "done",
+ "propertyType": "msg",
+ "rules": [
+ {
+ "t": "false"
+ },
+ {
+ "t": "else"
+ }
+ ],
+ "checkall": "true",
+ "repair": false,
+ "outputs": 2,
+ "x": 390,
+ "y": 180,
+ "wires": [
+ [
+ "b65bff76.ccfaa"
+ ],
+ [
+ "5b5a51f.aaa1eb",
+ "6a2eefef.95866"
+ ]
+ ]
+ },
+ {
+ "id": "b65bff76.ccfaa",
+ "type": "ui_text",
+ "z": "80638a41.e07d68",
+ "group": "a9cf3f4b.a6172",
+ "order": 4,
+ "width": 0,
+ "height": 0,
+ "name": "",
+ "label": "interim→",
+ "format": "{{msg.payload}}",
+ "layout": "row-left",
+ "x": 540,
+ "y": 180,
+ "wires": []
+ },
+ {
+ "id": "2340ced4.6f9712",
+ "type": "comment",
+ "z": "80638a41.e07d68",
+ "name": "↑select appropriate voice",
+ "info": "",
+ "x": 590,
+ "y": 400,
+ "wires": []
+ },
+ {
+ "id": "9e9dd20a.deefa",
+ "type": "ui_microphone",
+ "z": "80638a41.e07d68",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "order": 4,
+ "width": 0,
+ "height": 0,
+ "maxLength": 5,
+ "maxRecogLength": 0,
+ "timeslice": 0,
+ "press": "click",
+ "mode": "recog",
+ "interimResults": true,
+ "x": 230,
+ "y": 180,
+ "wires": [
+ [
+ "b3eef9ed.456698"
+ ]
+ ]
+ },
+ {
+ "id": "a9cf3f4b.a6172",
+ "type": "ui_group",
+ "name": "Default",
+ "tab": "a96b5d8e.0a5a3",
+ "order": 1,
+ "disp": true,
+ "width": "6",
+ "collapse": false
+ },
+ {
+ "id": "a96b5d8e.0a5a3",
+ "type": "ui_tab",
+ "name": "Speech Recognition Example",
+ "icon": "dashboard",
+ "order": 3,
+ "disabled": false,
+ "hidden": false
+ }
+]
diff --git a/node-red-node-ui-microphone/examples/02 - recognize and play speech.json b/node-red-node-ui-microphone/examples/02 - recognize and play speech.json
@@ -0,0 +1,125 @@
+[
+ {
+ "id": "3ed6351e.b4548a",
+ "type": "ui_microphone",
+ "z": "701c5025.117cc",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "order": 1,
+ "width": "3",
+ "height": "1",
+ "maxLength": 5,
+ "maxRecogLength": 0,
+ "timeslice": 0,
+ "press": "click",
+ "mode": "recog",
+ "interimResults": false,
+ "outputs": 1,
+ "x": 170,
+ "y": 480,
+ "wires": [
+ [
+ "5b5a51f.aaa1eb",
+ "6a2eefef.95866"
+ ]
+ ]
+ },
+ {
+ "id": "5b5a51f.aaa1eb",
+ "type": "change",
+ "z": "701c5025.117cc",
+ "name": "",
+ "rules": [
+ {
+ "t": "set",
+ "p": "audio",
+ "pt": "flow",
+ "to": "payload",
+ "tot": "msg"
+ }
+ ],
+ "action": "",
+ "property": "",
+ "from": "",
+ "to": "",
+ "reg": false,
+ "x": 340,
+ "y": 480,
+ "wires": [
+ []
+ ]
+ },
+ {
+ "id": "98386b47.1f9578",
+ "type": "ui_button",
+ "z": "701c5025.117cc",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "order": 2,
+ "width": "3",
+ "height": "1",
+ "passthru": false,
+ "label": "Play",
+ "tooltip": "",
+ "color": "",
+ "bgcolor": "",
+ "icon": "",
+ "payload": "audio",
+ "payloadType": "flow",
+ "topic": "",
+ "x": 190,
+ "y": 580,
+ "wires": [
+ [
+ "f03beedb.7ca0d"
+ ]
+ ]
+ },
+ {
+ "id": "f03beedb.7ca0d",
+ "type": "ui_audio",
+ "z": "701c5025.117cc",
+ "name": "",
+ "group": "a9cf3f4b.a6172",
+ "voice": "Google US English",
+ "always": "",
+ "x": 320,
+ "y": 580,
+ "wires": []
+ },
+ {
+ "id": "6a2eefef.95866",
+ "type": "ui_text",
+ "z": "701c5025.117cc",
+ "group": "a9cf3f4b.a6172",
+ "order": 3,
+ "width": 0,
+ "height": 0,
+ "name": "",
+ "label": "text→",
+ "format": "{{msg.payload}}",
+ "layout": "row-left",
+ "x": 310,
+ "y": 520,
+ "wires": []
+ },
+ {
+ "id": "a9cf3f4b.a6172",
+ "type": "ui_group",
+ "name": "Default",
+ "tab": "a96b5d8e.0a5a3",
+ "order": 1,
+ "disp": true,
+ "width": "6",
+ "collapse": false
+ },
+ {
+ "id": "a96b5d8e.0a5a3",
+ "type": "ui_tab",
+ "name": "Speech Recognition Example",
+ "icon": "dashboard",
+ "order": 3,
+ "disabled": false,
+ "hidden": false
+ }
+]
diff --git a/node-red-node-ui-microphone/locales/en-US/ui_microphone.html b/node-red-node-ui-microphone/locales/en-US/ui_microphone.html
@@ -0,0 +1,40 @@
+<script type="text/html" data-help-name="ui_microphone">
+ <p>A Node-RED dashboard ui node to capture audio or to recognize
+ spoken text from the brower's microphone.</p>
+ <h3>Outputs</h3>
+ <dl class="message-properties">
+ <dt>payload<span class="property-type">Buffer or string</span></dt>
+ <dd>The captured audio in the required format or recognized text.</dd>
+ <dt>done<span class="property-type">Bool</span></dt>
+ <dd>Result is final result(speech recognition mode only)</dd>
+ </dl>
+ <h3>Details</h3>
+ <p>This node provides a single button widget in the dashboard.</p>
+ <p><b>Audio Input</b> mode and <b>Speech Recognition</b> mode can
+ be selected.</p>
+ <dl>
+ <dt>Audio Input mode</dt>
+ <dd>
+ <p>The audio is captured in WAV format and published by the node as a Buffer object.</p>
+ </dd>
+ <dt>Speech Recognition mode</dt>
+ <dd>
+ <p>Speech recognition is performed and the recognized text
+ is sent as payload value.<p>
+ <p>If <b>Interim Results</b> is checked, the node sends 
+ interim results. In addition, <code>done</code> 
+ property is set to <code>true</code> if the result is interim,
+ otherwise <code>false</code>.</p>
+ <p>Note: This mode needs internet access for operation and
+ only works on browser that supports SpeechRecognition
+ feature of Web Speech API (e.g. Chrome). Mode selector is
+ not shown if SpeechRecognition feature is not supported. </p>
+ </dd>
+ </dl>
+ <p>The button can be configured in two modes. Either click begin
+ to capture audio/recognize speech, and then
+ continue to capture audio/recognize text until the button is pressed again, or it reaches its configured maximum duration. Or the button can be configured to record/recognize only while the button is pressed.</p>
+ <p>If value of the maximum duration is 0, no limit is applied for
+ audio capture/speech recognition.</p>
+ <p></p>
+</script>