Merge pull request #3467 from WenmuZhou/table2

replace image in layoutparse doc
PaddlePaddle · Aug 2, 2021 · c86ee9f · c86ee9f
2 parents e903fb8 + bcfea0c
commit c86ee9f
Show file tree

Hide file tree

Showing 18 changed files with 503 additions and 86 deletions.
diff --git a/doc/table/PaddleDetection_config.png b/doc/table/PaddleDetection_config.png
diff --git a/doc/table/layout.jpg b/doc/table/layout.jpg
diff --git a/doc/table/result_all.jpg b/doc/table/result_all.jpg
diff --git a/doc/table/result_text.jpg b/doc/table/result_text.jpg
diff --git a/doc/table/tableocr_pipeline.jpg b/doc/table/tableocr_pipeline.jpg
diff --git a/doc/table/tableocr_pipeline_en.jpg b/doc/table/tableocr_pipeline_en.jpg
diff --git a/ppocr/postprocess/rec_postprocess.py b/ppocr/postprocess/rec_postprocess.py
@@ -346,14 +346,14 @@ def load_char_elem_dict(self, character_dict_path):
  list_elem = []
  with open(character_dict_path, "rb") as fin:
  lines = fin.readlines()
- substr = lines[0].decode('utf-8').strip("\n").split("\t")
+ substr = lines[0].decode('utf-8').strip("\n").strip("\r\n").split("\t")
  character_num = int(substr[0])
  elem_num = int(substr[1])
  for cno in range(1, 1 + character_num):
- character = lines[cno].decode('utf-8').strip("\n")
+ character = lines[cno].decode('utf-8').strip("\n").strip("\r\n")
  list_character.append(character)
  for eno in range(1 + character_num, 1 + character_num + elem_num):
- elem = lines[eno].decode('utf-8').strip("\n")
+ elem = lines[eno].decode('utf-8').strip("\n").strip("\r\n")
  list_elem.append(elem)
  return list_character, list_elem
 

diff --git a/ppstructure/README.md b/ppstructure/README.md
@@ -7,7 +7,7 @@ PaddleStructure is an OCR toolkit for complex layout analysis. It can divide doc
 
 **install layoutparser**
 ```sh
-pip3 install https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
+pip3 install -U premailer paddleocr https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
 ```
 **install paddlestructure**
 
@@ -57,8 +57,28 @@ im_show = draw_result(image, result,font_path=font_path)
 im_show = Image.fromarray(im_show)
 im_show.save('result.jpg')
 ```
+#### 1.2.3 返回结果说明
+The return result of PaddleStructure is a list composed of a dict, an example is as follows
+
+```shell
+[
+ { 'type': 'Text', 
+ 'bbox': [34, 432, 345, 462], 
+ 'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]], 
+ [('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)])
+ }
+]
+```
+The description of each field in dict is as follows
+
+| Parameter | Description | 
+| --------------- | -------------|
+|type|Type of image area|
+|bbox|The coordinates of the image area in the original image, respectively [left upper x, left upper y, right bottom x, right bottom y]|
+|res|OCR or table recognition result of image area。<br> Table: HTML string of the table; <br> OCR: A tuple containing the detection coordinates and recognition results of each single line of text|
+
 
-#### 1.2.3 Parameter Description：
+#### 1.2.4 Parameter Description：
 
 | Parameter | Description | Default value |
 | --------------- | ---------------------------------------- | ------------------------------------------- |
@@ -80,22 +100,22 @@ In PaddleStructure, the image will be analyzed by layoutparser first. In the lay
 
 ### 2.1 LayoutParser
 
-Layout analysis divides the document data into regions, including the use of Python scripts for layout analysis tools, extraction of special category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README.md).
+Layout analysis divides the document data into regions, including the use of Python scripts for layout analysis tools, extraction of special category detection boxes, performance indicators, and custom training layout analysis models. For details, please refer to [document](layout/README_en.md).
 
 ### 2.2 Table OCR
 
 Table OCR converts table image into excel documents, which include the detection and recognition of table text and the prediction of table structure and cell coordinates. For detailed, please refer to [document](table/README.md)
 
-### 3. Predictive by inference engine
+## 3. Predictive by inference engine
 
 Use the following commands to complete the inference. 
 
 ```python
-python3 table/predict_system.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output ../output/table
+python3 table/predict_system.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output=../output/table --vis_font_path=../doc/fonts/simfang.ttf
 ```
 After running, each image will have a directory with the same name under the directory specified in the output field. Each table in the picture will be stored as an excel, and the excel file name will be the coordinates of the table in the image.
 
-# 3. Model List
+**Model List**
 
 
 |model name|description|config|model size|download|

diff --git a/ppstructure/README_ch.md b/ppstructure/README_ch.md
@@ -8,7 +8,7 @@ PaddleStructure是一个用于复杂版面分析的OCR工具包，其能够对
 
 **安装 layoutparser**
 ```sh
-pip3 install https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
+pip3 install -U premailer paddleocr https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
 ```
 **安装 paddlestructure**
 
@@ -58,8 +58,28 @@ im_show = Image.fromarray(im_show)
 im_show.save('result.jpg')
 ```
 
+#### 1.2.3 返回结果说明
+PaddleStructure 的返回结果为一个dict组成的list，示例如下
+
+```shell
+[
+ { 'type': 'Text', 
+ 'bbox': [34, 432, 345, 462], 
+ 'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]], 
+ [('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent ', 0.465441)])
+ }
+]
+```
+dict 里各个字段说明如下
+
+| 字段 | 说明 | 
+| --------------- | -------------|
+|type|图片区域的类型|
+|bbox|图片区域的在原图的坐标，分别[左上角x，左上角y，右下角x，右下角y]|
+|res|图片区域的OCR或表格识别结果。<br> 表格: 表格的HTML字符串; <br> OCR: 一个包含各个单行文字的检测坐标和识别结果的元组|
 
-#### 1.2.3 参数说明
+
+#### 1.2.4 参数说明
 
 | 字段 | 说明 | 默认值 |
 | --------------- | ---------------------------------------- | ------------------------------------------- |
@@ -80,28 +100,27 @@ im_show.save('result.jpg')
 
 在PaddleStructure中，图片会先经由layoutparser进行版面分析，在版面分析中，会对图片里的区域进行分类，包括**文字、标题、图片、列表和表格**5类。对于前4类区域，直接使用PP-OCR完成对应区域文字检测与识别。对于表格类区域，经过Table OCR处理后，表格图片转换为相同表格样式的Excel文件。
 
-### 2.1 LayoutParser
+### 2.1 版面分析
 
 版面分析对文档数据进行区域分类，其中包括版面分析工具的Python脚本使用、提取指定类别检测框、性能指标以及自定义训练版面分析模型，详细内容可以参考[文档](layout/README.md)。
 
-### 2.2 Table OCR
+### 2.2 表格识别
 
 Table OCR将表格图片转换为excel文档，其中包含对于表格文本的检测和识别以及对于表格结构和单元格坐标的预测，详细说明参考[文档](table/README_ch.md)
 
-### 3. 预测引擎推理
+## 3. 预测引擎推理
 
 使用如下命令即可完成预测引擎的推理
 
 ```python
-python3 table/predict_system.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output ../output/table
+python3 table/predict_system.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output=../output/table --vis_font_path=../doc/fonts/simfang.ttf
 ```
 运行完成后，每张图片会output字段指定的目录下有一个同名目录，图片里的每个表格会存储为一个excel，excel文件名为表格在图片里的坐标。
 
-# 3. Model List
-
+**Model List**
 
 |模型名称|模型简介|配置文件|推理模型大小|下载地址|
 | --- | --- | --- | --- | --- |
 |en_ppocr_mobile_v2.0_table_det|英文表格场景的文字检测|[ch_det_mv3_db_v2.0.yml](../configs/det/ch_ppocr_v2.0/ch_det_mv3_db_v2.0.yml)| 4.7M |[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) |
-|en_ppocr_mobile_v2.0_table_rec|英文表格场景的文字识别|[rec_chinese_lite_train_v2.0.yml](..//configs/rec/rec_mv3_none_bilstm_ctc.yml)|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) |
+|en_ppocr_mobile_v2.0_table_rec|英文表格场景的文字识别|[rec_chinese_lite_train_v2.0.yml](../configs/rec/rec_mv3_none_bilstm_ctc.yml)|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) |
 |en_ppocr_mobile_v2.0_table_structure|英文表格场景的表格结构预测|[table_mv3.yml](../configs/table/table_mv3.yml)|18.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) |
diff --git a/ppstructure/layout/README.md b/ppstructure/layout/README.md
@@ -1,17 +1,20 @@
 # 版面分析使用说明
 
-* [1. 安装whl包](#安装whl包)
-* [2. 使用](#使用)
-* [3. 后处理](#后处理)
-* [4. 指标](#指标)
-* [5. 训练版面分析模型](#训练版面分析模型)
+[1. 安装whl包](#安装whl包)
+
+[2. 使用](#使用)
+
+[3. 后处理](#后处理)
+
+[4. 指标](#指标)
+
+[5. 训练版面分析模型](#训练版面分析模型)
 
 <a name="安装whl包"></a>
 
 ## 1. 安装whl包
 ```bash
-wget https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
-pip install -U layoutparser-0.0.0-py3-none-any.whl
+pip install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
 ```
 
 <a name="使用"></a>
@@ -21,8 +24,9 @@ pip install -U layoutparser-0.0.0-py3-none-any.whl
 使用layoutparser识别给定文档的布局：
 
 ```python
+import cv2
 import layoutparser as lp
-image = cv2.imread("imags/paper-image.jpg")
+image = cv2.imread("doc/table/layout.jpg")
 image = image[..., ::-1]
 
 # 加载模型
@@ -35,7 +39,8 @@ model = lp.PaddleDetectionLayoutModel(config_path="lp:https://PubLayNet/ppyolov2_r50vd
 layout = model.detect(image)
 
 # 显示结果
-lp.draw_box(image, layout, box_width=3, show_element_type=True)
+show_img = lp.draw_box(image, layout, box_width=3, show_element_type=True)
+show_img.show()
 ```
 
 下图展示了结果，不同颜色的检测框表示不同的类别，并通过`show_element_type`在框的左上角显示具体类别：
@@ -67,7 +72,7 @@ lp.draw_box(image, layout, box_width=3, show_element_type=True)
 | [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) | lp:https://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config | {0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"} |
 
 * TableBank word和TableBank latex分别在word文档、latex文档数据集训练；
-* 下载TableBank数据集同时包含word和latex。
+* 下载的TableBank数据集里同时包含word和latex。
 
 <a name="后处理"></a>
 
@@ -76,6 +81,7 @@ lp.draw_box(image, layout, box_width=3, show_element_type=True)
 版面分析检测包含多个类别，如果只想获取指定类别(如"Text"类别)的检测框、可以使用下述代码：
 
 ```python
+# 接上面代码
 # 首先过滤特定文本类型的区域
 text_blocks = lp.Layout([b for b in layout if b.type=='Text'])
 figure_blocks = lp.Layout([b for b in layout if b.type=='Figure'])
@@ -99,9 +105,10 @@ right_blocks.sort(key = lambda b:b.coordinates[1])
 text_blocks = lp.Layout([b.set(id = idx) for idx, b in enumerate(left_blocks + right_blocks)])
 
 # 显示结果
-lp.draw_box(image, text_blocks,
+show_img = lp.draw_box(image, text_blocks,
  box_width=3, 
  show_element_id=True)
+show_img.show()
 ```
 
 显示只有"Text"类别的结果：

diff --git a/ppstructure/layout/README_en.md b/ppstructure/layout/README_en.md
@@ -0,0 +1,139 @@
+# Getting Started
+
+[1. Install whl package](#Install whl package)
+
+[2. Quick Start](#Quick Start)
+
+[3. PostProcess](#PostProcess)
+
+[4. Results](#Results)
+
+[5. Training](#Training)
+
+<a name="Install whl package"></a>
+
+## 1. Install whl package
+```bash
+wget https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
+pip install -U layoutparser-0.0.0-py3-none-any.whl
+```
+
+<a name="Quick Start"></a>
+
+## 2. Quick Start
+
+Use LayoutParser to identify the layout of a given document:
+
+```python
+import cv2
+import layoutparser as lp
+image = cv2.imread("doc/table/layout.jpg")
+image = image[..., ::-1]
+
+# load model
+model = lp.PaddleDetectionLayoutModel(config_path="lp:https://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config", 
+ threshold=0.5,
+ label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"},
+ enforce_cpu=False, 
+ enable_mkldnn=True)
+# detect
+layout = model.detect(image)
+
+# show result
+show_img = lp.draw_box(image, layout, box_width=3, show_element_type=True)
+show_img.show()
+```
+
+The following figure shows the result, with different colored detection boxes representing different categories and displaying specific categories in the upper left corner of the box with `show_element_type`
+
+<div align="center">
+<img src="../../doc/table/result_all.jpg" width = "600" />
+</div>
+`PaddleDetectionLayoutModel`parameters are described as follows:
+
+| parameter | description | default | remark |
+| :------------: | :------------------------------------------------------: | :---------: | :----------------------------------------------------------: |
+| config_path | model config path | None | Specify config_ path will automatically download the model (only for the first time,the model will exist and will not be downloaded again) |
+| model_path | model path | None | local model path, config_ path and model_ path must be set to one, cannot be none at the same time |
+| threshold | threshold of prediction score | 0.5 | \ |
+| input_shape | picture size of reshape | [3,640,640] | \ |
+| batch_size | testing batch size | 1 | \ |
+| label_map | category mapping table | None | Setting config_ path, it can be none, and the label is automatically obtained according to the dataset name_ map |
+| enforce_cpu | whether to use CPU | False | False to use GPU, and True to force the use of CPU |
+| enforce_mkldnn | whether mkldnn acceleration is enabled in CPU prediction | True | \ |
+| thread_num | the number of CPU threads | 10 | \ |
+
+The following model configurations and label maps are currently supported, which you can use by modifying '--config_path' and '--label_map' to detect different types of content:
+
+| dataset | config_path | label_map |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------- |
+| [TableBank](https://doc-analysis.github.io/tablebank-page/index.html) word | lp:https://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_word/config | {0:"Table"} |
+| TableBank latex | lp:https://TableBank/ppyolov2_r50vd_dcn_365e_tableBank_latex/config | {0:"Table"} |
+| [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) | lp:https://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config | {0: "Text", 1: "Title", 2: "List", 3:"Table", 4:"Figure"} |
+
+* TableBank word and TableBank latex are trained on datasets of word documents and latex documents respectively;
+* Download TableBank dataset contains both word and latex。
+
+<a name="PostProcess"></a>
+
+## 3. PostProcess
+
+Layout parser contains multiple categories, if you only want to get the detection box for a specific category (such as the "Text" category), you can use the following code:
+
+```python
+# follow the above code
+# filter areas for a specific text type
+text_blocks = lp.Layout([b for b in layout if b.type=='Text'])
+figure_blocks = lp.Layout([b for b in layout if b.type=='Figure'])
+
+# text areas may be detected within the image area, delete these areas
+text_blocks = lp.Layout([b for b in text_blocks \
+ if not any(b.is_in(b_fig) for b_fig in figure_blocks)])
+
+# sort text areas and assign ID
+h, w = image.shape[:2]
+
+left_interval = lp.Interval(0, w/2*1.05, axis='x').put_on_canvas(image)
+
+left_blocks = text_blocks.filter_by(left_interval, center=True)
+left_blocks.sort(key = lambda b:b.coordinates[1])
+
+right_blocks = [b for b in text_blocks if b not in left_blocks]
+right_blocks.sort(key = lambda b:b.coordinates[1])
+
+# the two lists are merged and the indexes are added in order
+text_blocks = lp.Layout([b.set(id = idx) for idx, b in enumerate(left_blocks + right_blocks)])
+
+# display result
+show_img = lp.draw_box(image, text_blocks,
+ box_width=3, 
+ show_element_id=True)
+show_img.show()
+```
+
+Displays results with only the "Text" category：
+
+<div align="center">
+<img src="../../doc/table/result_text.jpg" width = "600" />
+</div>
+<a name="Results"></a>
+
+## 4. Results
+
+| Dataset | mAP | CPU time cost | GPU time cost |
+| --------- | ---- | ------------- | ------------- |
+| PubLayNet | 93.6 | 1713.7ms | 66.6ms |
+| TableBank | 96.2 | 1968.4ms | 65.1ms |
+
+**Envrionment：** 
+
+ **CPU：** Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz，24core
+
+ **GPU：** a single NVIDIA Tesla P40
+
+<a name="Training"></a>
+
+## 5. Training
+
+The above model is based on PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection) ，if you want to train your own layout parser model，please refer to：[train_layoutparser_model](train_layoutparser_model_en.md)
+