CN111445420A - Image operation method and device of convolutional neural network and electronic equipment - Google Patents

Image operation method and device of convolutional neural network and electronic equipment Download PDF

Info

Publication number
CN111445420A
CN111445420A CN202010276826.6A CN202010276826A CN111445420A CN 111445420 A CN111445420 A CN 111445420A CN 202010276826 A CN202010276826 A CN 202010276826A CN 111445420 A CN111445420 A CN 111445420A
Authority
CN
China
Prior art keywords
pooling
layer
image
neural network
image block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010276826.6A
Other languages
Chinese (zh)
Other versions
CN111445420B (en
Inventor
王艳龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Aixin Technology Co ltd
Original Assignee
Beijing Aixin Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Aixin Technology Co ltd filed Critical Beijing Aixin Technology Co ltd
Priority to CN202010276826.6A priority Critical patent/CN111445420B/en
Publication of CN111445420A publication Critical patent/CN111445420A/en
Application granted granted Critical
Publication of CN111445420B publication Critical patent/CN111445420B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an image operation method, an image operation device and electronic equipment of a convolutional neural network, wherein the image operation method comprises the steps of obtaining preset operation parameters of the convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; acquiring image blocks from the image to be processed according to the operation parameters; calculating a pooling feature matrix of each image block through the convolutional neural network; and forming a pooling feature matrix of the image to be processed according to the pooling feature matrix of each image block. According to the method, the scale of the feature matrix output by the pooling layer is deduced according to the internal cache of the electronic equipment, and the window scale parameter matched with the network parameter of the convolutional neural network is further set, so that redundant data does not appear in the convolutional feature matrix obtained through convolution operation according to the image block acquired by the window scale parameter, the pooling calculation can be directly carried out on the convolutional result output by the convolutional layer, the convolutional result does not need to be cached to the DDR, and the memory resource and the network broadband are saved.

Description

Image operation method and device of convolutional neural network and electronic equipment
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image operation method and apparatus for a convolutional neural network, and an electronic device.
Background
The convolutional neural network comprises an input layer, a hidden layer and an output layer; wherein the hidden layer comprises a convolutional layer and a pooling layer, which is also called a sampling layer. In the related art, a convolutional neural network accelerator usually adopts a window with a fixed scale to collect an image block, performs convolution calculation on the collected image block through a convolutional layer, caches a convolution result to a DDR (Double Data Rate Dynamic Random Access Memory), acquires the convolution result from the DDR, performs pooling calculation through a pooling layer, and outputs a pooling result. In the method, because the scale of the acquisition window of the image block is fixed, if the scale of the acquisition window is too large relative to the convolution calculation of a certain convolution layer, the acquired image block is large and redundant data exists; in addition, the convolution result output from the convolution layer may have redundant data with respect to the subsequent pooling layer. Therefore, the above-mentioned manner of buffering the convolution result to the DDR and then reading the convolution result from the DDR is required to solve the problem of redundant data. However, the way to access the convolution result in DDR is wasteful of a lot of memory resources and network bandwidth.
Overall, the conventional convolutional neural network operation method results in a large amount of memory resources and network bandwidth waste.
Disclosure of Invention
In view of the above, the present invention provides an image operation method and apparatus for a convolutional neural network, and an electronic device, which can reduce the occupation of memory resources and save network bandwidth.
In a first aspect, an embodiment of the present invention provides an image operation method for a convolutional neural network, where the method is applied to an electronic device that implements operation of the convolutional neural network; the method comprises the following steps: acquiring operation parameters of a preset convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; acquiring image blocks from the image to be processed according to the operation parameters; for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block; and according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed.
In a preferred embodiment of the present invention, the step of obtaining the operation parameter of the predetermined convolutional neural network includes: acquiring hardware parameters of the electronic equipment; and determining the operation parameters of the preset convolutional neural network according to the hardware parameters.
In a preferred embodiment of the present invention, the step of determining the operation parameters of the predetermined convolutional neural network according to the hardware parameters includes: determining a constraint condition of a preset convolutional neural network according to the hardware parameter; wherein the hardware parameter includes an internal cache size of the electronic device; the constraint conditions include: the maximum scale of a convolution characteristic matrix output by a convolution layer in the convolution neural network, or the maximum scale of an image block input by the convolution layer; and determining the operation parameters of the convolutional neural network according to the constraint conditions.
In a preferred embodiment of the present invention, the step of determining the operation parameters of the convolutional neural network according to the constraint condition includes: if the constraint condition comprises the maximum scale of the convolution characteristic matrix output by the convolution layer, determining the maximum scale of the image block input by the convolution layer according to the maximum scale of the convolution characteristic matrix; determining the scale of a pooling feature matrix output by a pooling layer in the convolutional neural network according to the maximum scale of the image block input by the convolutional layer; and determining the operation parameters of the convolutional neural network according to the scale of the pooling feature matrix output by the pooling layer.
In a preferred embodiment of the present invention, the step of determining the maximum scale of the image block input to the convolutional layer according to the maximum scale of the convolutional feature matrix includes: determining the maximum scale of the image block input by the convolutional layer by the following formula: i _ cov _ width _ valid _ max ═ o _ cov _ width _ max-1) × cs _ w + 1; i _ cov _ height _ valid _ max ═ (o _ cov _ height _ max-1) × cs _ h + 1; wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; o _ cov _ width _ max is the maximum width of the convolution feature matrix output by the convolution layer; o _ cov _ height _ max is the maximum height of the convolution feature matrix output by the convolution layer; cs _ w is the horizontal step size of the convolutional layer; cw _ h is the vertical step of the convolutional layer.
In a preferred embodiment of the present invention, the step of determining the scale of the pooled feature matrix output by the pooling layer in the convolutional neural network according to the maximum scale of the image block input by the convolutional layer includes: determining the scale of the pooled feature matrix output by the pooling layer in the convolutional neural network by the following formula: (pw _ w + (pl _ width-1) ps _ w-1) cs _ w +1 ═ i _ cov _ width _ valid _ max; (pw _ h + (pl _ height-1) ps _ h-1) cs _ h +1 ═ i _ cov _ height _ valid max; wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; pl _ width is the width of the pooling characteristic matrix output by the pooling layer; pl _ height is the height of the pooled feature matrix output by the pooled layer; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer.
In a preferred embodiment of the present invention, the step of determining the operation parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooling layer includes: rounding the scale of the pooling characteristic matrix output by the pooling layer; and determining the operation parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooled layer after the integration.
In a preferred embodiment of the present invention, the step of determining the operation parameters of the convolutional neural network according to the rounded scale of the pooled feature matrix output by the pooled layer includes: calculating the window scale for acquiring the image block from the image to be processed in the operation parameters by the following formula: i _ cov _ width ═ pw _ w + (o _ pl _ width-1) ps _ w-1) cs _ w + cw _ w; i _ cov _ height ═ pw _ h + (o _ pl _ height-1) ps _ h-1) cs _ h + cw _ h; wherein i _ cov _ width is the width of the window; i _ cov _ height is the height of the window; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer; o _ pl _ width is the width of the pooled feature matrix output by the pooled layer after the completion; o _ pl _ height is the height of the pooled feature matrix output by the pooled layer after the completion; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer.
In a preferred embodiment of the present invention, the step of determining the operation parameters of the convolutional neural network according to the rounded scale of the pooled feature matrix output by the pooled layer includes: calculating the window displacement for collecting the image block from the image to be processed in the operation parameters by the following formula: cb _ ws ═ ((pw _ w + (o _ pl _ width-1) × ps _ w-1) × cs _ w +1- (pw _ w-ps _ w)) × cs _ w; cb _ hs ═ ((pw _ h + (o _ pl _ height-1) × ps _ h-1) × cs _ h +1- (pw _ h-ps _ h)) × cs _ h; wherein cb _ ws is the horizontal displacement step of the window; the cb _ hs is the vertical shift step size of the window.
In a preferred embodiment of the present invention, the operation parameters include: the step of acquiring the image block from the image to be processed according to the operation parameter comprises the following steps: determining a first image block from the image to be processed according to the window scale parameter in the operation parameter and a preset default initial position; traversing the image to be processed according to a preset sequence according to the window scale parameter and the window displacement parameter to obtain image blocks except the first image block.
In a preferred embodiment of the present invention, the step of forming the pooled feature matrix of each image block into the pooled feature matrix of the image to be processed according to the position of each image block in the image to be processed includes: and splicing the pooling characteristic matrixes of the image blocks adjacent to each other according to the position of each image block in the image to be processed to obtain a pooling characteristic matrix of the image to be processed.
In a second aspect, an embodiment of the present invention further provides an image operation apparatus for a convolutional neural network, where the apparatus is applied to an electronic device that implements operation of the convolutional neural network; the device includes: the operation parameter acquisition module is used for acquiring operation parameters of a preset convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; the image block acquisition module is used for acquiring an image block from the image to be processed according to the operation parameter; the image block pooling feature matrix calculating module is used for performing the following processing on each image block: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block; and the image pooling feature matrix calculating module is used for forming the pooling feature matrix of each image block into the pooling feature matrix of the image to be processed according to the position of each image block in the image to be processed.
In a fourth aspect, an embodiment of the present invention further provides an electronic device for implementing a convolutional neural network operation, where the electronic device includes a processing device and a storage device; the storage means has stored thereon a computer program which, when run by the processing device, performs the steps of the image computation method of the convolutional neural network described above.
In a fifth aspect, the embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processing device, the computer program performs the steps of the image operation method of the convolutional neural network.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides an image operation method, an image operation device and electronic equipment of a convolutional neural network, which are characterized by firstly obtaining preset operation parameters of the convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; then, acquiring image blocks from the image to be processed according to the operation parameters; for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block; and then according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed. This method is based on the following principle: under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, if the size of an image block input into the convolutional layer is fixed, the scale of a convolution characteristic matrix output by the convolutional layer and the scale of a pooling characteristic matrix output by the pooling layer are also determined; therefore, under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, the scale of the feature matrix output by the pooling layer can be deduced according to the storage attribute of the electronic device, and then an appropriate window scale parameter is set (namely, the operation parameter of the convolutional neural network is deduced in advance based on the incidence relation of the window scale parameter), so that the size of the acquired image block is matched with the convolution kernel parameter and the pooling window parameter of the convolutional neural network, and further redundant data does not appear in the convolution feature matrix obtained after convolution operation of the image block, so that the convolution result output by the convolutional layer can be directly subjected to pooling calculation without caching the convolution result to a DDR (double data rate), the pipeline design from convolution operation to pooling operation is realized, the occupation of memory resources can be reduced, and the network bandwidth is saved.
In addition, the operation method can be suitable for the neural network with different convolution kernel parameters and pooling window parameters, so that the method has good universality.
Additional features and advantages of the disclosure will be set forth in the description which follows, or in part may be learned by the practice of the above-described techniques of the disclosure, or may be learned by practice of the disclosure.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of an image operation method of a convolutional neural network according to an embodiment of the present invention;
FIG. 3 is a schematic view of a window displacement diagram of an image block acquired in a convolutional neural network image operation according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an image block acquisition according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating an image operation method of a convolutional neural network according to another embodiment of the present invention;
FIG. 6 is a schematic diagram of a convolutional neural network operation process of a pipeline design according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an image computing device of a convolutional neural network according to an embodiment of the present invention.
Icon: 100-an electronic device; 102-a processing device; 104-a storage device; 106-an input device; 108-an output device; 110-an image acquisition device; 112-a bus system; 71-an operation parameter acquisition module; 72-image block acquisition module; 73-image block pooling feature matrix calculation module; 74-image pooling feature matrix calculation module.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In view of the problem that the conventional convolutional neural network operation mode causes waste of a large amount of memory resources and network bandwidth, the image operation, device and electronic device of the convolutional neural network provided by the embodiment of the invention can be applied to hardware design for realizing the convolutional neural network and various application scenes related to convolutional neural network operation. For the convenience of understanding the embodiment, first, the image operation method of the convolutional neural network disclosed in the embodiment of the present invention is described in detail.
Here, an example electronic device of an image operation, an apparatus, and an electronic device for implementing a convolutional neural network of an embodiment of the present invention is described with reference to fig. 1.
As shown in fig. 1, the electronic device 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and one or more image capture devices 110, which are interconnected via a bus system 112 and/or other type of connection mechanism (not shown). It should be noted that the components and structure of the electronic device shown in fig. 1 are exemplary only, and not limiting, and the electronic device may have other components and structures as desired.
The processing device 102 may be a smart terminal, or a device including a Central Processing Unit (CPU) or other form of processing unit having data processing capability and/or instruction execution capability, may process data of other components in the electronic device, and may also control other components in the electronic device to perform the function of target object statistics.
Storage 104 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, Random Access Memory (RAM), cache memory (or the like). The non-volatile memory may include, for example, Read Only Memory (ROM), a hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer-readable storage medium and executed by processing device 102 to implement the client functionality (implemented by the processing device) of the embodiments of the invention described below and/or other desired functionality. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
Image capture device 110 may capture a training picture and store the captured preview video frame or image data in storage 104 for use by other components.
Illustratively, the devices in the image operation, apparatus and electronic device for implementing the convolutional neural network according to the embodiment of the present invention may be integrally arranged, or may be dispersedly arranged, such as integrally arranging the processing device 102, the storage device 104, the input device 106 and the output device 108, and arranging the image capturing device 110 at a designated position where a picture can be captured. When the above-described devices in the electronic apparatus are integrally provided, the electronic apparatus may be implemented as a smart terminal such as a camera, a smart phone, a tablet computer, a vehicle-mounted terminal, and the like.
As shown in fig. 2, which is a schematic flowchart of an image operation method of a convolutional neural network according to an embodiment of the present invention, the method is applied to an electronic device for implementing operation of the convolutional neural network, and as can be seen from fig. 2, the method includes the following steps:
step S202: acquiring operation parameters of a preset convolutional neural network; the operation parameter is associated with a storage attribute of the electronic device.
Here, the electronic device for implementing the convolutional neural network operation may be an FPGA (Field programmable gate Array) chip, an Asic (Application Specific integrated circuit) chip, or the like.
The convolutional neural network is constructed according to preset operation parameters, the operation parameters are associated with storage attributes of the electronic equipment, and the storage attributes comprise caches in hardware circuits. In one possible implementation, the operation parameters include a window scale parameter and a window displacement parameter for acquiring an image block from an image to be processed. In actual operation, image block acquisition is performed according to window scale parameters, and the window scale parameters generally include the width and height of a window. Moreover, the window displacement parameter is a moving parameter of the acquisition window when the image is acquired, and generally includes a displacement direction and a displacement step length. Here, referring to fig. 3, a schematic diagram of window displacement of an acquired image block in a convolutional neural network image operation is shown, and in the embodiment shown in fig. 3, the horizontal displacement and the vertical displacement of the acquisition window are shown.
In an embodiment of the present invention, the window scale parameter is associated with a scale of a feature matrix output by a pooling layer in the convolutional neural network, wherein the convolutional neural network comprises an input layer, a hidden layer and an output layer, and the hidden layer comprises a convolutional layer and a pooling layer. In actual operation, sampling an image to be processed according to the window scale parameters to obtain an image block, inputting the image block into a convolution layer to perform convolution operation to obtain a convolution characteristic matrix, inputting the convolution characteristic into a pooling layer, and outputting the pooling characteristic matrix of the image block. Here, since in the case where the convolution kernel parameter and the pooling window parameter are determined, if the size of the input image block is fixed, the scale of the convolution feature matrix output by the convolution layer is also determined, and the scale of the pooled feature matrix output by the pooling layer is also determined; therefore, such a reverse reasoning can be performed, when the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, according to the storage attribute (for example, RAM cache) of the electronic device, the scale of the feature matrix output by the pooling layer can be derived, and then an appropriate window scale parameter is set, so that the size of the acquired image block is matched with the convolution kernel parameter and the pooling window parameter of the convolutional neural network, for example, redundant data does not appear in the convolution feature matrix obtained after the convolution operation of the image block with an appropriate size.
Compared with the prior art for acquiring the image block pooled feature matrix, the method does not need to cache the convolution feature matrix to the DDR, and can realize the pipeline design from the convolution operation to the pooled operation.
It should be noted that, in practical implementation, the window scale parameter in the above operation parameters is not only associated with the scale of the feature matrix output by the pooling layer in the convolutional neural network, but also associated with parameters of each layer of the convolutional neural network, such as the convolutional layer convolutional kernel size, step size, pooling layer window, step size, and the like, and meanwhile, the window scale parameter is also associated with the cache capacity of the system.
Step S204: and acquiring image blocks from the image to be processed according to the operation parameters.
In practical operation, the step of acquiring image blocks from the image to be processed may be implemented by the following steps 21-22:
(21) and determining a first image block from the image to be processed according to the window scale parameter in the operation parameter and a preset default initial position.
The initial position may be any position in the image to be processed, and taking the position of the upper left corner of the acquisition window as an example, the initial position may be set as the position of the pixel point where the upper left corner of the image to be processed is located.
(22) Traversing the image to be processed according to a preset sequence according to the window scale parameter and the window displacement parameter to obtain image blocks except the first image block.
Acquiring an image block through the window scale parameters to obtain an image block with a specific size; and the acquisition window is displaced according to the window displacement parameters, wherein the preset sequence is a direction route of the window displacement, the to-be-processed image is traversed, and the image blocks are sequentially acquired until the last image block in the to-be-processed image is acquired.
Referring to fig. 4, which is a schematic diagram illustrating an image block acquisition, in the embodiment shown in fig. 4, the window scale parameters include a window width and a window height, and the window displacement parameters include a horizontal displacement step and a vertical displacement step, wherein the window displacement is acquired in a sequence of moving horizontally from the top left corner of the image to the right corner of the image, and then, changing lines according to the vertical displacement step, continuing to move horizontally and acquiring the image block.
Step S206: for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; and performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block.
In this embodiment, after each image block is subjected to convolution calculation to obtain a convolution feature matrix, pooling calculation is directly performed on the convolution feature matrix through a pooling layer of the convolution neural network to obtain a pooling feature matrix corresponding to the image block.
Step S208: and according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed.
In one possible implementation manner, according to the position of each image block in the image to be processed, the pooling characteristic matrices of image blocks adjacent to each other in position may be subjected to stitching processing, so as to obtain the pooling characteristic matrix of the image to be processed.
The image operation method of the convolutional neural network provided by the embodiment of the invention comprises the steps of firstly obtaining the operation parameters of the preset convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; then, acquiring image blocks from the image to be processed according to the operation parameters; for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block; and then according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed. This method is based on the following principle: under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, if the size of an image block input into the convolutional layer is fixed, the scale of a convolution characteristic matrix output by the convolutional layer and the scale of a pooling characteristic matrix output by the pooling layer are also determined; therefore, under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, the scale of the feature matrix output by the pooling layer can be deduced according to the storage attribute of the electronic device, and then an appropriate window scale parameter is set (namely, the operation parameter of the convolutional neural network is deduced in advance based on the incidence relation of the window scale parameter), so that the size of the acquired image block is matched with the convolution kernel parameter and the pooling window parameter of the convolutional neural network, and further redundant data does not appear in the convolution feature matrix obtained after convolution operation of the image block, so that the convolution result output by the convolutional layer can be directly subjected to pooling calculation without caching the convolution result to a DDR (double data rate), the pipeline design from convolution operation to pooling operation is realized, the occupation of memory resources can be reduced, and the network bandwidth is saved.
On the basis of the image operation method of the convolutional neural network shown in fig. 2, the present embodiment further provides another image operation method of the convolutional neural network, which is applied to an electronic device for implementing operation of the convolutional neural network, where the method focuses on a specific implementation process of step S202 (i.e., obtaining preset operation parameters of the convolutional neural network) in the foregoing embodiment. Fig. 5 is a schematic flow chart of the operation method of the convolutional neural network, and as can be seen from fig. 5, the method includes the following steps:
step S502: and acquiring hardware parameters of the electronic equipment.
Here, the electronic device is an electronic device for implementing a convolutional neural network operation, that is, an electronic device for operating an operation method of the convolutional neural network, where the electronic device may be a CNN hardware accelerator. The hardware parameters generally include the type of processor used by the device, the host frequency, the memory capacity, and the like.
Step S504: determining the operation parameters of a preset convolutional neural network according to the hardware parameters; the operation parameter is associated with a storage attribute of the electronic device.
In one possible embodiment, the operation parameters of the convolutional neural network can be determined through the following steps 31 to 32:
(31) determining a constraint condition of a preset convolutional neural network according to the hardware parameter; wherein the hardware parameter includes an internal cache size of the electronic device; the constraint conditions include: the maximum scale of the convolution layer output convolution feature matrix in the convolution neural network, or the maximum scale of the image block input by the convolution layer.
Here, for a preset convolutional neural network, that is, under the condition that the convolutional kernel parameter and the pooling window parameter of the convolutional neural network are determined, if the internal cache size of the electronic device operating the convolutional neural network is known, the maximum scale of the convolutional feature matrix output by the convolutional layer in the convolutional neural network can be correspondingly determined, and for the convolutional layer of the convolutional neural network, the scale of the convolutional feature matrix output by the convolutional layer can also be correspondingly obtained based on the scale of the convolutional feature matrix output by the convolutional layer, so that the maximum scale of the image block input by the convolutional layer is obtained.
(32) And determining the operation parameters of the convolutional neural network according to the constraint conditions.
In at least one possible embodiment, if the constraint includes a maximum dimension of the convolution feature matrix of the convolutional layer output, the step of determining the operation parameters of the convolutional neural network according to the constraint may be implemented by the following steps 41 to 44:
(41) and determining the maximum scale of the image block input by the convolutional layer according to the maximum scale of the convolutional characteristic matrix.
In this embodiment, the maximum scale of the image block input to the convolutional layer is determined by the following formula:
i _ cov _ width _ valid _ max ═ (o _ cov _ width _ max-1) × cs _ w +1 (equation 1.1)
i _ cov _ height _ valid _ max (o _ cov _ height _ max-1) × cs _ h +1 (equation 1.2)
Wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; o _ cov _ width _ max is the maximum width of the convolution feature matrix output by the convolution layer; o _ cov _ height _ max is the maximum height of the convolution feature matrix output by the convolution layer; cs _ w is the horizontal step size of the convolutional layer; cw _ h is the vertical step of the convolutional layer.
(42) And determining the scale of the pooled feature matrix output by the pooled layer in the convolutional neural network according to the maximum scale of the image block input by the convolutional layer.
In this embodiment, the scale of the pooled feature matrix output by the pooling layer in the convolutional neural network is determined by the following formula:
(pw _ w + (pl _ width-1) ps _ w-1) cs _ w +1 i _ cov _ width _ valid _ max (equation 1.3)
(pw _ h + (pl _ height-1) ps _ h-1) cs _ h +1 i _ cov _ height _ valid max (equation 1.4)
Wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; pl _ width is the width of the pooling characteristic matrix output by the pooling layer; pl _ height is the height of the pooled feature matrix output by the pooled layer; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer.
(43) And determining the operation parameters of the convolutional neural network according to the scale of the pooling feature matrix output by the pooling layer.
In at least one possible embodiment, the scale of the pooled feature matrix output by the pooling layer may be first rounded; and then, determining the operation parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooled layer after the integration.
When rounding the scale of the pooled feature matrix, rounding can be performed through the following formula:
o _ pl _ width (pl _ width) (equation 1.5)
o _ pl _ height (pl _ height) (equation 1.6)
In the above formula, o _ pl _ width is the width of the pooled feature matrix output by the pooled layer after the completion; o _ pl _ height is the height of the pooled feature matrix output by the pooled layer after the completion; floor denotes rounding. In practice, rounding down is usually done, for example, if the calculated scale is 5.3, then rounding to 5.
Here, substituting the above equations 1.5 and 1.6 into the above equations 1.3 and 1.4 can obtain the following equations:
i _ cov _ width _ valid ═ pw _ w + (o _ pl _ width-1) ps _ w-1) cs _ w +1 (equation 1.7)
i _ cov _ height _ valid ═ pw _ h + (o _ pl _ height-1) ps _ h-1) cs _ h +1 (equation 1.8)
In actual operation, since the size of the image block input to the convolutional layer is related to the size of the convolutional core, considering padding, after obtaining the scale of the pooled feature matrix output by the pooled layer after being rounded, the window scale used for acquiring the image block from the image to be processed in the operation parameter can be calculated by the following formula:
i _ cov _ width ═ pw _ w + (o _ pl _ width-1) × ps _ w-1) × cs _ w + cw _ w (equation 1.9)
i _ cov _ height ═ pw _ h + (o _ pl _ height-1) ps _ h-1) cs _ h + cw _ h (formula 1.10), where i _ cov _ width is the width of the window; i _ cov _ height is the height of the window; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer; o _ pl _ width is the width of the pooled feature matrix output by the pooled layer after the completion; o _ pl _ height is the height of the pooled feature matrix output by the pooled layer after the completion; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer.
Here, the above-mentioned window scale of the acquired image block is also the scale of the image block of the input convolutional layer, and accordingly, the scale of the convolutional feature matrix output by the convolutional layer may also be obtained as follows:
o _ cov _ width ═ i _ cov _ width _ valid-1)/cs _ w +1 (equation 1.11)
o _ cov _ height ═ (i _ cov _ height _ valid-1)/cs _ h +1 (equation 1.12)
Wherein o _ cov _ width is the width of the convolution feature matrix; o _ cov _ height is the height of the convolution feature matrix.
In one possible embodiment, the window displacement of the operation parameter for acquiring the image block from the image to be processed may also be calculated by the following formula:
cb _ ws ═ cs _ w ((pw _ w + (o _ pl _ width-1) × ps _ w-1) × cs _ w +1- (pw _ w-ps _ w)) × cs _ w (formula 1.13)
cb _ hs ═ cs _ h ((pw _ h + (o _ pl _ height-1) × ps _ h-1) × cs _ h +1- (pw _ h-ps _ h)) (formula 1.14)
Wherein cb _ ws is the horizontal displacement step of the window; the cb _ hs is the vertical shift step size of the window.
As can be seen from equations 1.13 and 1.14 above, the window displacement of an image block is related to the pooling parameter in addition to the convolution parameter and the first image block size, and when pw _ w is equal to ps _ w and pw _ h is equal to ps _ h, the window displacement is related to the convolution parameter and the size of the first image block only, and adjacent image blocks of the pooled output can be obtained by the above two equations without any overlap. As shown in fig. 6, which is a schematic diagram of an operation process of a convolutional neural network designed by a pipeline, it can be seen from fig. 6 that four image blocks in the diagram complete seamless docking of each image block through convolution and pooling operations, thereby truly realizing a pipeline design from convolution to pooling.
Therefore, the window scale parameter and the window displacement parameter of the convolutional neural network are determined according to the scale of the pooling feature matrix output by the pooling layer, and then the image block acquisition can be performed according to the operation parameter. And the operation parameters of the convolutional neural network calculated according to the steps are matched with the hardware configuration of the electronic equipment for operating the convolutional neural network, so that redundant data does not appear in a convolutional characteristic matrix obtained after convolution operation according to the image blocks acquired according to the operation parameters, and the convolutional result output by the convolutional layer can be directly subjected to pooling calculation without caching the convolutional result to a DDR (double data rate) to solve the problem of redundant data.
Step S506: and acquiring image blocks from the image to be processed according to the operation parameters.
Step S508: for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; and performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block.
Step S510: and according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed.
Here, steps S506 to S510 in this embodiment correspond to steps S204 to S208 in the above embodiment, and corresponding contents may refer to corresponding parts of the above embodiment, which are not described herein again.
In order to more clearly understand the process of acquiring the operation parameters of the convolutional neural network in the image operation method of the convolutional neural network provided in this embodiment, an application example is described below.
Assuming that the constraint conditions of the convolutional neural network are determined according to the internal RAM buffer size of some electronic equipment as shown in the following tables 1 and 2:
TABLE 1 image Block Width hardware constraints
cs_w I_cov_width_valid_max
1 32
2 63
3 64
TABLE 2 image Block height hardware constraints
cs_h i_cov_height_valid_max
1 16
2 31
3 46
And, some layer network characteristics in the convolutional neural network are as follows: pw _ w is 5, ps _ w is 4, cs _ w is 2, cw _ w is 3; pw _ h is 5, ps _ h is 4, cs _ h is 2, and cw _ h is 3.
Looking up a table by cs _ w ═ 2 to obtain: i _ cov _ width _ valid _ max is 63.
Substituting the value of i _ cov _ width _ valid _ max into the above equation (1.3) yields:
(5+ (pl _ w-1) × 4-1) × 2+1 ═ 63, yielding pl _ width ═ 7.75.
Further, it is obtained from the formula (formula 1.5): o _ pl _ width is 7.
The value of o _ pl _ width is substituted into (equation 1.7) and I _ cov _ width _ valid is calculated to be 57, which is seen to be the value that is closest to the ideal value I _ cov _ width _ valid _ max.
Further, i _ cov _ width is calculated from (equation 1.9) to 59;
substituting the value of i _ cov _ width into (equation 1.11) yields: o _ cov _ width is 29.
Therefore, the image block width (i.e., the width of the window) i _ cov _ width of the input convolutional layer is 59, the width o _ cov _ widt of the convolution feature matrix of the output convolutional layer is 29, and the width o _ pl _ w of the pooling feature matrix of the output pooling layer is 7. Similarly, it can be found that the image block height (i.e., the window width) of the output convolutional layer i _ cov _ height is 27, the height of the convolutional feature matrix of the output convolutional layer o _ cov _ height is 13, and the height of the pooled feature matrix of the output pooled layer o _ pl _ height is 3.
Therefore, the operation parameters of the preset convolutional neural network are determined according to the hardware parameters of the electronic equipment. In actual operation, according to the network parameters of the convolutional neural network on the electronic device and the requirement of hardware design storage resource size, the window information (convolutional input, convolutional output, pooling output) and the shift information of the window can be calculated according to the above formulas 1.1 to 1.14. Because the window parameters are irrelevant to the number of channels of the convolutional neural network, the operation parameters can be configured to a register in a design module through a cpu during hardware design, and a certain control logic is combined, so that the pipeline design from convolution to pooling can be realized. Here, the operation method has good versatility because the operation method can be applied to neural networks with different convolution kernel parameters and pooling window parameters.
According to the image operation method of the convolutional neural network, the constraint condition of the convolutional neural network is determined according to the hardware parameter of the electronic device for realizing the operation of the convolutional neural network, the operation parameter of the convolutional neural network is further determined according to the constraint condition, and the image block is collected based on the operation parameter, so that redundant data does not appear in a convolutional characteristic matrix obtained after the convolution operation of the image block, the convolutional result output by a convolutional layer can be directly subjected to pooling calculation without caching the convolutional result to a DDR (double data rate), the pipeline design from the convolutional operation to the pooling operation is realized, the occupation of memory resources can be reduced, the CNN hardware realization area is reduced, and the network broadband is saved more.
Corresponding to the image operation method of the convolutional neural network shown in fig. 2, an embodiment of the present invention further provides an image operation device of the convolutional neural network, which is applied to an electronic device for implementing operation of the convolutional neural network, and referring to fig. 7, the image operation device of the convolutional neural network is shown as a schematic structural diagram, as can be seen from fig. 7, the device includes an operation parameter obtaining module 71, an image block collecting module 72, an image block pooling feature matrix calculating module 73, and an image pooling feature matrix calculating module 74, which are connected in sequence, where functions of each module are as follows:
an operation parameter obtaining module 71, configured to obtain operation parameters of a preset convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment;
an image block acquisition module 72, configured to acquire an image block from the image to be processed according to the operation parameter;
an image block pooling feature matrix calculating module 73, configured to perform the following processing for each image block: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block;
and the image pooling feature matrix calculating module 74 is configured to combine the pooling feature matrix of each image block into a pooling feature matrix of the image to be processed according to the position of each image block in the image to be processed.
The embodiment of the invention provides an image operation device of a convolutional neural network, which comprises the following steps of firstly, obtaining preset operation parameters of the convolutional neural network; the operation parameter is associated with the storage attribute of the electronic equipment; then, acquiring image blocks from the image to be processed according to the operation parameters; for each image block, the following processing is performed: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block; and then according to the position of each image block in the image to be processed, forming the pooling characteristic matrix of each image block into the pooling characteristic matrix of the image to be processed. The device is based on the following principle: under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, if the size of an image block input into the convolutional layer is fixed, the scale of a convolution characteristic matrix output by the convolutional layer and the scale of a pooling characteristic matrix output by the pooling layer are also determined; therefore, under the condition that the convolution kernel parameter and the pooling window parameter of the convolutional neural network are determined, the scale of the feature matrix output by the pooling layer can be deduced according to the storage attribute of the electronic device, and then an appropriate window scale parameter is set (namely, the operation parameter of the convolutional neural network is deduced in advance based on the incidence relation of the window scale parameter), so that the size of the acquired image block is matched with the convolution kernel parameter and the pooling window parameter of the convolutional neural network, and further redundant data does not appear in the convolution feature matrix obtained after convolution operation of the image block, so that the convolution result output by the convolutional layer can be directly subjected to pooling calculation without caching the convolution result to a DDR (double data rate), the pipeline design from convolution operation to pooling operation is realized, the occupation of memory resources can be reduced, and the network bandwidth is saved.
In one possible implementation, the operation parameter obtaining module 71 is further configured to: acquiring hardware parameters of the electronic equipment; and determining the operation parameters of the preset convolutional neural network according to the hardware parameters.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: determining a constraint condition of a preset convolutional neural network according to the hardware parameter; wherein the hardware parameter includes an internal cache size of the electronic device; the constraint conditions include: the maximum scale of a convolution characteristic matrix output by a convolution layer in the convolution neural network, or the maximum scale of an image block input by the convolution layer; and determining the operation parameters of the convolutional neural network according to the constraint conditions.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: if the constraint condition comprises the maximum scale of the convolution characteristic matrix output by the convolution layer, determining the maximum scale of the image block input by the convolution layer according to the maximum scale of the convolution characteristic matrix; determining the scale of a pooling feature matrix output by a pooling layer in the convolutional neural network according to the maximum scale of the image block input by the convolutional layer; and determining the operation parameters of the convolutional neural network according to the scale of the pooling feature matrix output by the pooling layer.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: determining the maximum scale of the image block input by the convolutional layer by the following formula: i _ cov _ width _ valid _ max ═ o _ cov _ width _ max-1) × cs _ w + 1; i _ cov _ height _ valid _ max ═ (o _ cov _ height _ max-1) × cs _ h + 1; wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; o _ cov _ width _ max is the maximum width of the convolution feature matrix output by the convolution layer; o _ cov _ height _ max is the maximum height of the convolution feature matrix output by the convolution layer; cs _ w is the horizontal step size of the convolutional layer; cw _ h is the vertical step of the convolutional layer.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: determining the scale of the pooled feature matrix output by the pooling layer in the convolutional neural network by the following formula: (pw _ w + (pl _ width-1) ps _ w-1) cs _ w +1 ═ i _ cov _ width _ valid _ max; (pw _ h + (pl _ height-1) ps _ h-1) cs _ h +1 ═ i _ cov _ height _ valid max; wherein, i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; pl _ width is the width of the pooling characteristic matrix output by the pooling layer; pl _ height is the height of the pooled feature matrix output by the pooled layer; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: rounding the scale of the pooling characteristic matrix output by the pooling layer; and determining the operation parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooled layer after the integration.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: calculating the window scale for acquiring the image block from the image to be processed in the operation parameters by the following formula: i _ cov _ width ═ pw _ w + (o _ pl _ width-1) ps _ w-1) cs _ w + cw _ w; i _ cov _ height ═ pw _ h + (o _ pl _ height-1) ps _ h-1) cs _ h + cw _ h; wherein i _ cov _ width is the width of the window; i _ cov _ height is the height of the window; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer; o _ pl _ width is the width of the pooled feature matrix output by the pooled layer after the completion; o _ pl _ height is the height of the pooled feature matrix output by the pooled layer after the completion; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer.
In another possible implementation, the operation parameter obtaining module 71 is further configured to: calculating the window displacement for collecting the image block from the image to be processed in the operation parameters by the following formula: cb _ ws ═ ((pw _ w + (o _ pl _ width-1) × ps _ w-1) × cs _ w +1- (pw _ w-ps _ w)) × cs _ w; cb _ hs ═ ((pw _ h + (o _ pl _ height-1) × ps _ h-1) × cs _ h +1- (pw _ h-ps _ h)) × cs _ h; wherein cb _ ws is the horizontal displacement step of the window; the cb _ hs is the vertical shift step size of the window.
In another possible implementation, the operation parameter includes: the window scale parameter and the window displacement parameter are used to acquire an image block from an image to be processed, and the image block acquisition module 72 is further configured to: determining a first image block from the image to be processed according to the window scale parameter in the operation parameter and a preset default initial position; traversing the image to be processed according to a preset sequence according to the window scale parameter and the window displacement parameter to obtain image blocks except the first image block.
In another possible implementation, the image pooling feature matrix calculating module 74 is further configured to: and splicing the pooling characteristic matrixes of the image blocks adjacent to each other according to the position of each image block in the image to be processed to obtain a pooling characteristic matrix of the image to be processed.
The implementation principle and the generated technical effect of the image operation device of the convolutional neural network provided by the embodiment of the present invention are the same as those of the above-mentioned embodiment of the image operation method of the convolutional neural network, and for brief description, reference may be made to the corresponding contents in the above-mentioned embodiment of the image operation method of the convolutional neural network for some portions of the embodiments of the image operation device of the convolutional neural network that are not mentioned.
The embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the image operation method of the convolutional neural network, and specific implementation may refer to the foregoing method embodiment, and is not described herein again.
The image operation method of the convolutional neural network, the image operation device of the convolutional neural network, and the computer program product of the electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program codes, where instructions included in the program codes may be used to execute the image operation method of the convolutional neural network described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims (14)

1. An image operation method of a convolutional neural network is characterized in that the method is applied to electronic equipment for realizing convolutional neural network operation; the method comprises the following steps:
acquiring operation parameters of a preset convolutional neural network; the operation parameter is associated with a storage attribute of the electronic equipment;
acquiring image blocks from the image to be processed according to the operation parameters;
for each image block, performing the following processing: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block;
and according to the position of each image block in the image to be processed, forming the pooling feature matrix of each image block into the pooling feature matrix of the image to be processed.
2. The method of claim 1, wherein the step of obtaining the operation parameters of the predetermined convolutional neural network comprises:
acquiring hardware parameters of the electronic equipment;
and determining the operation parameters of the preset convolutional neural network according to the hardware parameters.
3. The method of claim 2, wherein the step of determining the operation parameters of the predetermined convolutional neural network according to the hardware parameters comprises:
determining a preset constraint condition of the convolutional neural network according to the hardware parameter; wherein the hardware parameter comprises an internal cache size of the electronic device; the constraint conditions include: the maximum scale of a convolution characteristic matrix output by a convolution layer in the convolution neural network, or the maximum scale of an image block input by the convolution layer;
and determining the operation parameters of the convolutional neural network according to the constraint conditions.
4. The method of claim 3, wherein the step of determining the operation parameters of the convolutional neural network according to the constraint condition comprises:
if the constraint condition comprises the maximum scale of the convolution characteristic matrix output by the convolution layer, determining the maximum scale of the image block input by the convolution layer according to the maximum scale of the convolution characteristic matrix;
determining the scale of a pooling feature matrix output by a pooling layer in the convolutional neural network according to the maximum scale of the image block input by the convolutional layer;
and determining the operation parameters of the convolutional neural network according to the scale of the pooling feature matrix output by the pooling layer.
5. The method of claim 4, wherein the step of determining the maximum scale of the image blocks of the convolutional layer input based on the maximum scale of the convolutional feature matrix comprises:
determining a maximum scale of an image block input to the convolutional layer by the following formula:
i_cov_width_valid_max＝(o_cov_width_max-1)*cs_w+1；
i_cov_height_valid_max＝(o_cov_height_max-1)*cs_h+1；
wherein i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; o _ cov _ width _ max is the maximum width of the convolution feature matrix output by the convolution layer; o _ cov _ height _ max is the maximum height of the convolution feature matrix output by the convolution layer; cs _ w is the horizontal step size of the convolutional layer; cw _ h is the vertical step of the convolutional layer.
6. The method of claim 4, wherein the step of determining the scale of the pooled feature matrix of the pooled layer output in the convolutional neural network based on the maximum scale of the image block input to the convolutional layer comprises:
determining a scale of a pooled feature matrix output by a pooling layer in the convolutional neural network by:
(pw_w+(pl_width-1)*ps_w-1)*cs_w+1＝i_cov_width_valid_max；
(pw_h+(pl_height-1)*ps_h-1)*cs_h+1＝i_cov_height_valid max；
wherein i _ cov _ width _ valid _ max is the maximum width of the image block input by the convolutional layer; i _ cov _ height _ valid _ max is the maximum height of the image block input by the convolutional layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step length of the convolutional layer; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; pl _ width is the width of the pooling feature matrix output by the pooling layer; pl _ height is the height of the pooling feature matrix output by the pooling layer; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer.
7. The method of claim 4, wherein the step of determining the operational parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooling layer comprises:
rounding the scale of the pooling feature matrix output by the pooling layer;
and determining the operation parameters of the convolutional neural network according to the scale of the pooled feature matrix output by the pooled layer after the rounding.
8. The method of claim 7, wherein the step of determining the operation parameters of the convolutional neural network according to the rounded scale of the pooled feature matrix output by the pooled layer comprises:
calculating the window scale for collecting the image block from the image to be processed in the operation parameters by the following formula:
i_cov_width＝(pw_w+(o_pl_width-1)*ps_w-1)*cs_w+cw_w；
i_cov_height＝(pw_h+(o_pl_height-1)*ps_h-1)*cs_h+cw_h；
wherein i _ cov _ width is the width of the window; i _ cov _ height is the height of the window; pw _ w is the pooling window width of the pooling layer; pw _ h is the pooling window height of the pooling layer; o _ pl _ width is the width of the pooling feature matrix output by the pooled layer after the completion; o _ pl _ height is the height of the pooling feature matrix output by the pooled layer after the completion; ps _ w is the horizontal step size of the pooling layer; ps _ h is the vertical step size of the pooling layer; cs _ w is the horizontal step size of the convolutional layer; cs _ h is the vertical step size of the convolutional layer.
9. The method of claim 8, wherein the step of determining the operation parameters of the convolutional neural network according to the rounded scale of the pooled feature matrix output by the pooled layer comprises:
calculating the window displacement of the operation parameter for collecting the image block from the image to be processed by the following formula:
cb_ws＝((pw_w+(o_pl_width-1)*ps_w-1)*cs_w+1-(pw_w-ps_w))*cs_w；
cb_hs＝((pw_h+(o_pl_height-1)*ps_h-1)*cs_h+1-(pw_h-ps_h))*cs_h；
wherein cb _ ws is a horizontal displacement step of the window; the cb _ hs is a vertical displacement step of the window.
10. The method of claim 1, wherein the operation parameters comprise: the step of acquiring the image blocks from the image to be processed according to the operation parameters comprises the following steps:
determining a first image block from the image to be processed according to the window scale parameter in the operation parameters and a preset default initial position;
traversing the image to be processed according to a preset sequence according to the window scale parameter and the window displacement parameter to obtain image blocks except the first image block.
11. The method according to claim 1, wherein the step of combining the pooled feature matrix of each image block into the pooled feature matrix of the image to be processed according to the position of each image block in the image to be processed comprises:
and splicing the pooling characteristic matrixes of the image blocks adjacent to each other according to the position of each image block in the image to be processed to obtain a pooling characteristic matrix of the image to be processed.
12. An image operation device of a convolutional neural network is characterized in that the device is applied to electronic equipment for realizing operation of the convolutional neural network; the device comprises:
the operation parameter acquisition module is used for acquiring operation parameters of a preset convolutional neural network; the operation parameter is associated with a storage attribute of the electronic equipment;
the image block acquisition module is used for acquiring an image block from the image to be processed according to the operation parameters;
an image block pooling feature matrix calculating module, configured to perform the following processing for each image block: performing convolution calculation on the current image block through the convolution layer of the convolution neural network to obtain a convolution characteristic matrix of the current image block; performing pooling calculation on the convolution characteristic matrix through a pooling layer of the convolution neural network to obtain a pooling characteristic matrix of the current image block;
and the image pooling feature matrix calculating module is used for forming the pooling feature matrix of each image block into the pooling feature matrix of the image to be processed according to the position of each image block in the image to be processed.
13. An electronic device, characterized in that the electronic device comprises a processing device and a storage means;
the storage device has stored thereon a computer program that, when executed by the processing apparatus, performs the steps of the image operation method of a convolutional neural network according to any one of 1 to 11.
14. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processing device, performs the steps of the image operation method of a convolutional neural network of any one of 1 to 11.
CN202010276826.6A 2020-04-09 2020-04-09 Image operation method and device of convolutional neural network and electronic equipment Active CN111445420B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010276826.6A CN111445420B (en) 2020-04-09 2020-04-09 Image operation method and device of convolutional neural network and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010276826.6A CN111445420B (en) 2020-04-09 2020-04-09 Image operation method and device of convolutional neural network and electronic equipment

Publications (2)

Publication Number Publication Date
CN111445420A true CN111445420A (en) 2020-07-24
CN111445420B CN111445420B (en) 2023-06-06

Family

ID=71651422

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010276826.6A Active CN111445420B (en) 2020-04-09 2020-04-09 Image operation method and device of convolutional neural network and electronic equipment

Country Status (1)

Country Link
CN (1) CN111445420B (en)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183732A (en) * 2020-10-22 2021-01-05 中国人民解放军国防科技大学 Convolutional neural network acceleration method and device and computer equipment
CN112883983A (en) * 2021-02-09 2021-06-01 北京迈格威科技有限公司 Feature extraction method and device and electronic system
US11403069B2 (en) 2017-07-24 2022-08-02 Tesla, Inc. Accelerated mathematical engine
US11487288B2 (en) 2017-03-23 2022-11-01 Tesla, Inc. Data synthesis for autonomous control systems
US11537811B2 (en) 2018-12-04 2022-12-27 Tesla, Inc. Enhanced object detection for autonomous vehicles based on field view
US11561791B2 (en) 2018-02-01 2023-01-24 Tesla, Inc. Vector computational unit receiving data elements in parallel from a last row of a computational array
US11562231B2 (en) 2018-09-03 2023-01-24 Tesla, Inc. Neural networks for embedded devices
US11567514B2 (en) 2019-02-11 2023-01-31 Tesla, Inc. Autonomous and user controlled vehicle summon to a target
US11610117B2 (en) 2018-12-27 2023-03-21 Tesla, Inc. System and method for adapting a neural network model on a hardware platform
US11636333B2 (en) 2018-07-26 2023-04-25 Tesla, Inc. Optimizing neural network structures for embedded systems
US11665108B2 (en) 2018-10-25 2023-05-30 Tesla, Inc. QoS manager for system on a chip communications
US11681649B2 (en) 2017-07-24 2023-06-20 Tesla, Inc. Computational array microprocessor system using non-consecutive data formatting
CN116306823A (en) * 2023-04-27 2023-06-23 北京爱芯科技有限公司 Method, device and chip for providing data for MAC array
US11734562B2 (en) 2018-06-20 2023-08-22 Tesla, Inc. Data pipeline and deep learning system for autonomous driving
US11748620B2 (en) 2019-02-01 2023-09-05 Tesla, Inc. Generating ground truth for machine learning from time series elements
US11790664B2 (en) 2019-02-19 2023-10-17 Tesla, Inc. Estimating object properties using visual image data
US11816585B2 (en) 2018-12-03 2023-11-14 Tesla, Inc. Machine learning models operating at different frequencies for autonomous vehicles
US11841434B2 (en) 2018-07-20 2023-12-12 Tesla, Inc. Annotation cross-labeling for autonomous control systems
US11893774B2 (en) 2018-10-11 2024-02-06 Tesla, Inc. Systems and methods for training machine models with augmented data
US11893393B2 (en) 2017-07-24 2024-02-06 Tesla, Inc. Computational array microprocessor system with hardware arbiter managing memory requests
CN117763399A (en) * 2024-02-21 2024-03-26 电子科技大学 Neural network classification method for self-adaptive variable-length signal input
US12014553B2 (en) 2019-02-01 2024-06-18 Tesla, Inc. Predicting three-dimensional features for autonomous driving
US12086097B2 (en) 2017-07-24 2024-09-10 Tesla, Inc. Vector computational unit
US12136030B2 (en) 2023-03-16 2024-11-05 Tesla, Inc. System and method for adapting a neural network model on a hardware platform

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160104056A1 (en) * 2014-10-09 2016-04-14 Microsoft Technology Licensing, Llc Spatial pyramid pooling networks for image processing
CN105701507A (en) * 2016-01-13 2016-06-22 吉林大学 Image classification method based on dynamic random pooling convolution neural network
CN106407891A (en) * 2016-08-26 2017-02-15 东方网力科技股份有限公司 Target matching method based on convolutional neural network and device
WO2018052586A1 (en) * 2016-09-14 2018-03-22 Konica Minolta Laboratory U.S.A., Inc. Method and system for multi-scale cell image segmentation using multiple parallel convolutional neural networks
CN109117897A (en) * 2018-08-09 2019-01-01 百度在线网络技术（北京）有限公司 Image processing method, device and readable storage medium storing program for executing based on convolutional neural networks
CN109726798A (en) * 2018-12-27 2019-05-07 北京灵汐科技有限公司 A kind of data processing method and device
WO2019084788A1 (en) * 2017-10-31 2019-05-09 深圳市大疆创新科技有限公司 Computation apparatus, circuit and relevant method for neural network
WO2019127838A1 (en) * 2017-12-29 2019-07-04 国民技术股份有限公司 Method and apparatus for realizing convolutional neural network, terminal, and storage medium
CN110276444A (en) * 2019-06-04 2019-09-24 北京清微智能科技有限公司 Image processing method and device based on convolutional neural networks
CN110858398A (en) * 2018-08-24 2020-03-03 深圳市前海安测信息技术有限公司 Tongue segmentation device and method based on deep learning and storage medium

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160104056A1 (en) * 2014-10-09 2016-04-14 Microsoft Technology Licensing, Llc Spatial pyramid pooling networks for image processing
CN105701507A (en) * 2016-01-13 2016-06-22 吉林大学 Image classification method based on dynamic random pooling convolution neural network
CN106407891A (en) * 2016-08-26 2017-02-15 东方网力科技股份有限公司 Target matching method based on convolutional neural network and device
WO2018036146A1 (en) * 2016-08-26 2018-03-01 东方网力科技股份有限公司 Convolutional neural network-based target matching method, device and storage medium
WO2018052586A1 (en) * 2016-09-14 2018-03-22 Konica Minolta Laboratory U.S.A., Inc. Method and system for multi-scale cell image segmentation using multiple parallel convolutional neural networks
WO2019084788A1 (en) * 2017-10-31 2019-05-09 深圳市大疆创新科技有限公司 Computation apparatus, circuit and relevant method for neural network
WO2019127838A1 (en) * 2017-12-29 2019-07-04 国民技术股份有限公司 Method and apparatus for realizing convolutional neural network, terminal, and storage medium
CN109117897A (en) * 2018-08-09 2019-01-01 百度在线网络技术（北京）有限公司 Image processing method, device and readable storage medium storing program for executing based on convolutional neural networks
CN110858398A (en) * 2018-08-24 2020-03-03 深圳市前海安测信息技术有限公司 Tongue segmentation device and method based on deep learning and storage medium
CN109726798A (en) * 2018-12-27 2019-05-07 北京灵汐科技有限公司 A kind of data processing method and device
CN110276444A (en) * 2019-06-04 2019-09-24 北京清微智能科技有限公司 Image processing method and device based on convolutional neural networks

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
于波 等: "基于深度卷积神经网络的图像重建算法" *
李强: "基于Adaboost算法的人脸实时检测方法研究" *
李锐: ""基于卷积神经网络的人群计数技术研究"" *
李锐: "基于卷积神经网络的人群计数技术研究" *
王显宏: "面向深度学习平台的内存管理器的设计与实现" *

Cited By (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11487288B2 (en) 2017-03-23 2022-11-01 Tesla, Inc. Data synthesis for autonomous control systems
US12020476B2 (en) 2017-03-23 2024-06-25 Tesla, Inc. Data synthesis for autonomous control systems
US11681649B2 (en) 2017-07-24 2023-06-20 Tesla, Inc. Computational array microprocessor system using non-consecutive data formatting
US12086097B2 (en) 2017-07-24 2024-09-10 Tesla, Inc. Vector computational unit
US11403069B2 (en) 2017-07-24 2022-08-02 Tesla, Inc. Accelerated mathematical engine
US11893393B2 (en) 2017-07-24 2024-02-06 Tesla, Inc. Computational array microprocessor system with hardware arbiter managing memory requests
US11561791B2 (en) 2018-02-01 2023-01-24 Tesla, Inc. Vector computational unit receiving data elements in parallel from a last row of a computational array
US11797304B2 (en) 2018-02-01 2023-10-24 Tesla, Inc. Instruction set architecture for a vector computational unit
US11734562B2 (en) 2018-06-20 2023-08-22 Tesla, Inc. Data pipeline and deep learning system for autonomous driving
US11841434B2 (en) 2018-07-20 2023-12-12 Tesla, Inc. Annotation cross-labeling for autonomous control systems
US12079723B2 (en) 2018-07-26 2024-09-03 Tesla, Inc. Optimizing neural network structures for embedded systems
US11636333B2 (en) 2018-07-26 2023-04-25 Tesla, Inc. Optimizing neural network structures for embedded systems
US11983630B2 (en) 2018-09-03 2024-05-14 Tesla, Inc. Neural networks for embedded devices
US11562231B2 (en) 2018-09-03 2023-01-24 Tesla, Inc. Neural networks for embedded devices
US11893774B2 (en) 2018-10-11 2024-02-06 Tesla, Inc. Systems and methods for training machine models with augmented data
US11665108B2 (en) 2018-10-25 2023-05-30 Tesla, Inc. QoS manager for system on a chip communications
US11816585B2 (en) 2018-12-03 2023-11-14 Tesla, Inc. Machine learning models operating at different frequencies for autonomous vehicles
US11537811B2 (en) 2018-12-04 2022-12-27 Tesla, Inc. Enhanced object detection for autonomous vehicles based on field view
US11908171B2 (en) 2018-12-04 2024-02-20 Tesla, Inc. Enhanced object detection for autonomous vehicles based on field view
US11610117B2 (en) 2018-12-27 2023-03-21 Tesla, Inc. System and method for adapting a neural network model on a hardware platform
US11748620B2 (en) 2019-02-01 2023-09-05 Tesla, Inc. Generating ground truth for machine learning from time series elements
US12014553B2 (en) 2019-02-01 2024-06-18 Tesla, Inc. Predicting three-dimensional features for autonomous driving
US11567514B2 (en) 2019-02-11 2023-01-31 Tesla, Inc. Autonomous and user controlled vehicle summon to a target
US11790664B2 (en) 2019-02-19 2023-10-17 Tesla, Inc. Estimating object properties using visual image data
CN112183732A (en) * 2020-10-22 2021-01-05 中国人民解放军国防科技大学 Convolutional neural network acceleration method and device and computer equipment
CN112883983A (en) * 2021-02-09 2021-06-01 北京迈格威科技有限公司 Feature extraction method and device and electronic system
US12136030B2 (en) 2023-03-16 2024-11-05 Tesla, Inc. System and method for adapting a neural network model on a hardware platform
CN116306823B (en) * 2023-04-27 2023-08-04 北京爱芯科技有限公司 Method, device and chip for providing data for MAC array
CN116306823A (en) * 2023-04-27 2023-06-23 北京爱芯科技有限公司 Method, device and chip for providing data for MAC array
CN117763399B (en) * 2024-02-21 2024-05-14 电子科技大学 Neural network classification method for self-adaptive variable-length signal input
CN117763399A (en) * 2024-02-21 2024-03-26 电子科技大学 Neural network classification method for self-adaptive variable-length signal input

Also Published As

Publication number Publication date
CN111445420B (en) 2023-06-06

Similar Documents

Publication Publication Date Title
CN111445420B (en) Image operation method and device of convolutional neural network and electronic equipment
JP3781634B2 (en) Image processing apparatus, image processing method, and portable video device
EP3779769B1 (en) Optical flow computing method and computing device
WO2022247568A1 (en) Image restoration method and apparatus, and device
WO2020000665A1 (en) Image processing method, device and apparatus, and storage medium
CN110689496B (en) Method and device for determining noise reduction model, electronic equipment and computer storage medium
CN111080571A (en) Camera shielding state detection method and device, terminal and storage medium
CN112883983B (en) Feature extraction method, device and electronic system
CN112801876B (en) Information processing method and device, electronic equipment and storage medium
CN110545427A (en) PDF document compression method and device and electronic equipment
CN109871814B (en) Age estimation method and device, electronic equipment and computer storage medium
US7466862B2 (en) Image expansion and display method, image expansion and display device, and program for image expansion and display
JP5683153B2 (en) Image processing apparatus and image processing method
CN113706639B (en) Image compression method and device based on rectangular NAM, storage medium and computing equipment
WO2021212498A1 (en) Image processing method, system on chip, and electronic device
JP4158147B2 (en) Storage preview method, apparatus, and compression rate prediction method in Lossy compression
CN114119423A (en) Image processing method, image processing device, electronic equipment and storage medium
CN109429067B (en) Dynamic picture compression method and device, computer equipment and storage medium
JP2009003953A (en) Image processor
JP2007028208A (en) Image processing apparatus and imaging apparatus
CN112750136B (en) Image processing method and system
KR102293883B1 (en) Simultaneous image compression and thumbnail generation
JP4865021B2 (en) Image processing apparatus and image processing method
KR100595067B1 (en) Apparatus and method for resizing image
JP3655814B2 (en) Enlarged image generation apparatus and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant