[Clojure] Clojure BERT QA example #14691

gigasquid · 2019-04-13T16:04:29Z

Description

Thanks to @lanking520 and the JVM team - we were able to convert the Java BERT QA example to the Clojure package 💯

This makes a slight change to it by having an external edn file to store the sample question and answers along with the ground truths to be able to process multiple examples and make it easier to edit and add more.

Example output:

===============================
      Question Answer Data
{:input-answer
 "Computational complexity theory is a branch of the theory of computation in theoretical computer science that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other. A computational problem is understood to be a task that is in principle amenable to being solved by a computer, which is equivalent to stating that the problem may be solved by mechanical application of mathematical steps, such as an algorithm.",
 :input-question
 "By what main attribute are computational problems classified utilizing computational complexity theory?",
 :ground-truth-answers
 ["Computational complexity theory"
  "Computational  complexity theory"
  "complexity theory"]}

  Predicted Answer:  [inherent difficulty]
===============================
===============================
      Question Answer Data
{:input-answer
 "Steam engines are external combustion engines, where the working fluid is separate from the combustion products. Non-combustion heat sources such as solar power, nuclear power or geothermal energy may be used. The ideal thermodynamic cycle used to analyze this process is called the Rankine cycle. In the cycle, water is heated and transforms into steam within a boiler operating at a high pressure. When expanded through pistons or turbines, mechanical work is done. The reduced-pressure steam is then condensed and pumped back into the boiler.",
 :input-question
 "Along with geothermal and nuclear, what is a notable non-combustion heat source?",
 :ground-truth-answers
 ["solar"
  "solar power"
  "solar power, nuclear power or geothermal energysolar"]}

  Predicted Answer:  [solar power]
===============================
===============================
      Question Answer Data
{:input-answer
 "In the 1960s, a series of discoveries, the most important of which was seafloor spreading, showed that the Earth's lithosphere, which includes the crust and rigid uppermost portion of the upper mantle, is separated into a number of tectonic plates that move across the plastically deforming, solid, upper mantle, which is called the asthenosphere. There is an intimate coupling between the movement of the plates on the surface and the convection of the mantle: oceanic plate motions and mantle convection currents always move in the same direction, because the oceanic lithosphere is the rigid upper thermal boundary layer of the convecting mantle. This coupling between rigid plates moving on the surface of the Earth and the convecting mantle is called plate tectonics.",
 :input-question
 "What was the most important discovery that led to the understanding that Earth's lithosphere is separated into tectonic plates?",
 :ground-truth-answers ["seafloor spreading"]}

  Predicted Answer:  [seafloor spreading]
===============================
===============================
      Question Answer Data
{:input-answer
 "Susan had a cat named Sammy when she lived in the green house.",
 :input-question "What was Susan's cat named?",
 :ground-truth-answers ["Sammy" "sammy"]}

  Predicted Answer:  [sammy]
===============================
===============================
      Question Answer Data
{:input-answer
 "Rich Hickey is the creator of the Clojure language. Before Clojure, he developed dotLisp, a similar project based on the .NET platform, and three earlier attempts to provide interoperability between Lisp and Java: a Java foreign language interface for Common Lisp, A Foreign Object Interface for Lisp, and a Lisp-friendly interface to Java Servlets.",
 :input-question "Who created Clojure?",
 :ground-truth-answers ["rich" "hickey"]}

  Predicted Answer:  [rich hickey]
===============================

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

New example code for BERT QA based off the Java example along with a new integration test

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

rename core to infer add integration test

kedarbellare · 2019-04-14T00:36:04Z

contrib/clojure-package/examples/bert-qa/README.md

+By default, this model are using `bert_12_768_12` model with extra layers for QA jobs.
+
+After that, to be able to use it in Java, we need to export the dictionary from the script to parse the text
+to actual indexes. Please add the following lines after [this line](https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/staticbert/static_finetune_squad.py#L262).


should we add an option to this file itself (i.e. create a PR) to export the vocabulary?

Interesting idea. @lanking520 did the original exploration and documentation on this. What do you think?

kedarbellare · 2019-04-14T00:37:08Z

contrib/clojure-package/examples/bert-qa/README.md

+
+For this tutorial, you can get the model and vocabulary by running following bash file. This script will use `wget` to download these artifacts from AWS S3.
+
+From the `scala-package/examples/scripts/infer/bert/` folder run:


shouldn't this be within the clojure bert-qa example folder?

kedarbellare · 2019-04-14T00:42:04Z

contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj

+ :token2idx (get vocab "token_to_idx")}))
+
+(defn tokens->idxs [token2idx tokens]
+ (mapv #(get token2idx % (get token2idx "[UNK]")) tokens))


(let [unk-idx (get token2idx "[UNK]")] ...)?

much better

kedarbellare · 2019-04-14T00:44:06Z

contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj

+
+(defn get-vocab []
+ (let [vocab (json/parse-stream (clojure.java.io/reader "model/vocab.json"))]
+ {:idx2token (get vocab "idx_to_token")


nit:

{:idx->token ... :token->idx ...}

?

kedarbellare · 2019-04-14T01:00:28Z

contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj

+ (break-out-punctuation s target-char)
+ [s]))
+
+(defn tokenizer [s]


tokenize?

kedarbellare

lgtm! this is great!! only minor comments/suggestions

gigasquid · 2019-04-14T10:55:18Z

Thanks so much for the feedback @kedarbellare - I'll work on implementing it 😸

lanking520 · 2019-04-14T16:10:49Z

contrib/clojure-package/examples/bert-qa/src/bert_qa/infer.clj

+ (map #(string/replace % "<punc>" str-match))))
+
+(defn break-out-punctuations [s]
+ (if-let [target-char (first (re-seq #"[.,?!]" s))]


I do saw some tokens like ... in your example, maybe get it covered as well.

Good point. I was changing the data examples around and I don't have one with that in there now, but I'll keep my eye out for them in the future.

@kedarbellare

* Initial working example for bert qa * add RAT rename core to infer add integration test * add rat for project.clj * Couldn’t resist adding a qa about Clojure * rat for readme * feedback from @kedarbellare

@kedarbellare

* Initial working example for bert qa * add RAT rename core to infer add integration test * add rat for project.clj * Couldn’t resist adding a qa about Clojure * rat for readme * feedback from @kedarbellare

Initial working example for bert qa

d8cb2a7

gigasquid added Clojure pr-work-in-progress PR is still work in progress labels Apr 13, 2019

add RAT

8e0069e

rename core to infer add integration test

gigasquid removed the pr-work-in-progress PR is still work in progress label Apr 13, 2019

add rat for project.clj

32cb4fe

gigasquid changed the title ~~[Clojure] [WIP] Clojure BERT QA example~~ [Clojure] Clojure BERT QA example Apr 13, 2019

gigasquid added 2 commits April 13, 2019 16:27

Couldn’t resist adding a qa about Clojure

661a2d4

rat for readme

4df5cf8

kedarbellare reviewed Apr 14, 2019

View reviewed changes

kedarbellare approved these changes Apr 14, 2019

View reviewed changes

feedback from @kedarbellare

8c80af5

lanking520 reviewed Apr 14, 2019

View reviewed changes

gigasquid merged commit c2ba51b into apache:master Apr 14, 2019

gigasquid deleted the clojure-bert-qa-example branch April 14, 2019 19:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Clojure] Clojure BERT QA example #14691

[Clojure] Clojure BERT QA example #14691

gigasquid commented Apr 13, 2019 •

edited

Loading

kedarbellare Apr 14, 2019

gigasquid Apr 14, 2019

kedarbellare Apr 14, 2019

kedarbellare Apr 14, 2019

gigasquid Apr 14, 2019

kedarbellare Apr 14, 2019

kedarbellare Apr 14, 2019

kedarbellare left a comment

gigasquid commented Apr 14, 2019

lanking520 Apr 14, 2019

gigasquid Apr 14, 2019


		For this tutorial, you can get the model and vocabulary by running following bash file. This script will use `wget` to download these artifacts from AWS S3.

		From the `scala-package/examples/scripts/infer/bert/` folder run:

[Clojure] Clojure BERT QA example #14691

[Clojure] Clojure BERT QA example #14691

Conversation

gigasquid commented Apr 13, 2019 • edited Loading

Description

Checklist

Essentials

Changes

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kedarbellare left a comment

Choose a reason for hiding this comment

gigasquid commented Apr 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gigasquid commented Apr 13, 2019 •

edited

Loading