This is project for releasing some open-source natural language models from Joint Lab of BAAI and JDAI. Different from other open-source Chinese NLP models, we mainly focus on some basic models for dialogue systems, especially in E-commerce domain. Our corpus is very huge, currently we are using 42 GB Customer Service Dialogue Data (CSDD) for training, and it contain about 1.2 billion sentences.

We provide the pre-trained BERT and word embeddings. The charts below shows the data we use.

The links to the models are here.

JD-BERT.tar.gz file contains items:
  |—— BAAI-JDAI-BERT
      |—— bert_model.cpkt.*                  # pre-trained weights
      |—— bert_config.json                   # hyperparamters of the model
      |—— vocab.txt                          # vocabulary for WordPiece
      |—— JDAI-BERT.md & INTRO.md            # summary and details
JD-WORD-EMBEDDING.tar.gz file contains items:
  |—— BAAI-JDAI-WORD-EMBEDDING
      |—— JDAI-Word-Embedding.txt            # word vectors, each line separated by whitespace
      |—— JDAI-WORD-EMBEDDING.md & INTRO.md  # summary and details

• When pre-training, we do the data preprocessing with our preprocessor including some generalization processing.
• The init checkpoint we use is <12-layer, 768-hidden, 12-heads, 110M parameters>, Chinese, and bert_config.json and vocab.txt are identical to Google's original settings.
• We do not use Chinese Whole Word Masking(WWM) in our current pre-training but use google's original WWM which is on the Chinese character level.

We use Train data of LCQMC(Large-scale Chinese Question Matching Corpus) and CSDQMC(Customer Service Dialog Question Matching Corpus) for fine-tuning, and then just train 2 epochs with a proper init learning rate 2e-5 on each dataset respectively.

We evaluate our pre-trained model on the FAQ task with Test data of LCQMC and CSDQMC.

We quote the BERT and ERNIE's results on LCQMC from the Chinese-BERT-wwm report.

• When pre-training, we use our tools for preprocessing and word segmentaion.
• We train the vectors based on Skip-Gram.

We show top3 similar words for some sample words below. We use cosine distance to compute the distance of two words.