bert-base-chinese Incompatible shapes error

[Wind-Ward]

Rasa version:
2.4.3
Rasa SDK version (if used & relevant):
2.4.1
Rasa X version (if used & relevant):
0.37.1
Python version:
3.6.12
Operating system (windows, osx, ...):
Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
Issue:
Hi,I'm try to use "bert-base-chinese" component to train my data,but I ran into problems.
I have upgraded transformers already,but the problem still exists.Could you help me?Thanks a lot.

Error (including full traceback):

Traceback (most recent call last):
  File "/disk1/conda_env/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/cli/train.py", line 207, in train_nlu
    finetuning_epoch_fraction=args.epoch_fraction,
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/train.py", line 711, in train_nlu
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/utils/common.py", line 307, in run_in_loop
    result = loop.run_until_complete(f)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/train.py", line 758, in train_nlu_async
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/train.py", line 819, in _train_nlu_with_validated_data
    **additional_arguments,
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/nlu/train.py", line 115, in train
    interpreter = trainer.train(training_data, **kwargs)
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/nlu/model.py", line 209, in train
    updates = component.train(working_data, self.config, **context)
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 861, in train
    shuffle=False,  # we use custom shuffle inside data generator
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/disk1/conda_env/lib/python3.6/site-packages/rasa/utils/tensorflow/temp_keras_modules.py", line 188, in fit
    tmp_logs = train_function(iterator)
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/disk1/conda_env/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Incompatible shapes: [64,14] vs. [64,7]
         [[{{node cond/else/_13/cond/add_1}}]]
         [[crf/cond/else/_1/crf/cond/Cast/_260]]
  (1) Invalid argument:  Incompatible shapes: [64,14] vs. [64,7]
         [[{{node cond/else/_13/cond/add_1}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_29835]

Function call stack:
train_function -> train_function

2021-04-07 20:54:54.849304: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]

Command or request that led to error:

rasa train nlu

Content of configuration file (config.yml) (if relevant):

language: zh

pipeline:
  - name: HFTransformersNLP
    model_name: "bert"
    model_weights: "bert-base-chinese"
    cache_dir: "data/bert-base-chinese"
  - name: JiebaTokenizer
    dictionary_path: "data/dict"
  - name: LanguageModelFeaturizer
  - name: DIETClassifier
    epochs: 50
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 50

[muximuxi]

same error

[Dustyposa]

Why the error is not clear.
But in my opinion:
Do you want to use two tokenizer(Jieba and Bert)？
And HFTransformersNLP is deprecated and will be removed in a future release.You can direct use

 - name: LanguageModelFeaturizer
    # Name of the language model to use
    model_name: "bert"
    # Pre-Trained weights to be loaded
    model_weights: "bert-base-chinese"

    # An optional path to a specific directory to download and cache the pre-trained model weights.
    # The `default` cache_dir is the same as https://huggingface.co/transformers/serialization.html#cache-directory .
    cache_dir: data/bert-base-chinese

[sara-tagger]

Thanks for raising this issue, @dakshvar22 will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[Wind-Ward]

@Dustyposa As I know,I should put JiebaTokenizer before LanguageModelFeaturizer to run rasa.But I don't think this caused the error,Maybe I was wrong?The use of HFTransformersNLP doesn't matter the error.

[Dustyposa]

It's looks like diet_classifier raised this error.
The reason maybe is your datas, maybe is your cache model(eg. the model had finetuning? ).
You can use the the smallest demo(like rasa init) to validate your config.yml , if no error, the config is ok.
Then check your model or datas.

[muximuxi]

It seems the Tokenizer, do not use jieba when use bert
Should using rasa_chinese and rasa_chinese_service
and set pipline :- name: "rasa_chinese.nlu.tokenizers.lm_tokenizer.LanguageModelTokenizer"

[Wind-Ward]

@muximuxi Thank you,you are right.I use rasa_chinese and rasa_chinese_service.The problem has been solved. @Dustyposa Thank you for your help as well.

[laoyin]

@Wind-Ward @muximuxi using rasa_chinese and rasa_chinese_service just solving the intent-training ,

when i put entity nlu data, this error occur 。

- intent: wechat_num examples: | - 微信号[wechat_num](wechat_num) - 微信[wechat_num](wechat_num) - 我微信[wechat_num](wechat_num) - 我微信号[wechat_num](wechat_num) - 你加这个[wechat_num](wechat_num) - 你加[wechat_num](wechat_num)这个 - 你加[wechat_num](wechat_num)这个微信 - 你加我的微信[wechat_num](wechat_num)

File "/home/yinxingpan/outbound-robot/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call ctx=ctx) File "/home/yinxingpan/outbound-robot/venv/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,10] vs. [64,5] [[{{node cond/else/_13/cond/add_1}}]] [Op:__inference_train_function_25073]

[linglingfengty]

same error,and i delete all my cache models,it run