English | 简体中文
python3
pip install -r requirements.txt
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ner/standard
- Create and enter the python virtual environment.
- Install dependencies:
pip install -r requirements.txt
.
-
Dataset
-
Download the dataset to this directory.
wget 120.27.214.45/Data/ner/standard/data.tar.gz tar -xzvf data.tar.gz
-
Three types of data formats are supported,including
json
,docx
andtxt
. The dataset is stored indata
:train.txt
: Training setvalid.txt
: Validation settest.txt
: Test set
-
-
Training
-
Parameters for training are in the
conf
folder and users can modify them before training.This task supports multi card training. Modify trian.yaml's parameter 'use_multi_gpu' to true.'os.environ['CUDA_VISIBLE_DEVICES']' in 'run.py' set to the selected gpus. The first card is the main card for calculation, which requires a little more memory. -
With BERT, it is recommended that the batch size is no less than 64.
-
Logs for training are in the
log
folder and the trained model is saved in thecheckpoints
folder.
python run_bert.py or python run_crflstm.py
- W2NER(The new state-of-the-art ner model, which involvs with three major types, including flat, overlapped (aka. nested), and discontinuous NER.)
cd w2ner python run.py
-
-
Prediction
Chinese datasets are supported by default. If English datasets are used, 'nltk' need to be installed and download the corresponding vocabulary by running 'nltk.download('punkt')'. Meanwhile before prediction, 'lan' in config.yaml also need to be set en.
python predict.py
BiLSTM + CRF
BERT
W2NER
If you only have text data and corresponding dictionaries, but no canonical training data.
You can get weakly supervised formatted training data through automated labeling methods.
Please make sure that:
- Provide high-quality dictionaries
- Enough text data