CN110390657B - Image fusion method - Google Patents
Image fusion method Download PDFInfo
- Publication number
- CN110390657B CN110390657B CN201810358534.XA CN201810358534A CN110390657B CN 110390657 B CN110390657 B CN 110390657B CN 201810358534 A CN201810358534 A CN 201810358534A CN 110390657 B CN110390657 B CN 110390657B
- Authority
- CN
- China
- Prior art keywords
- image
- target image
- boundary
- pixel processing
- virtual pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 116
- 238000000034 method Methods 0.000 claims abstract description 62
- 230000004927 fusion Effects 0.000 claims abstract description 48
- 230000006870 function Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 14
- 230000007704 transition Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image fusion method. The method comprises the following steps: determining the range of the original target image boundary to be expanded by taking the original target image boundary to be fused as a reference; performing virtual pixel processing on the expanded range to obtain a target image after the virtual pixel processing; and fusing the target image after the virtual pixel processing and the background image by utilizing Poisson image editing. The image fusion method of the invention eliminates the problems of unnatural transition of the target image and fuzzy appearance at the boundary of the target image after image fusion, thereby finally obtaining ideal image fusion effect.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an image fusion method.
Background
The image fusion is to combine two or more images into a new image by a specific method, and the basic idea is to adopt a certain method to fuse the information of a plurality of images imaged by various image sensors working in different wavelength ranges and having different imaging mechanisms to the same scene into a new image, so that the fused image has higher reliability, less blur and better intelligibility, or is more suitable for human vision or computer detection, classification, identification, understanding and the like. The image fusion technology is widely applied to the fields of remote sensing image processing, computer vision, intelligent robots, military monitoring, medical scanning imaging and the like.
The principle of image fusion is that a target object or a target area in a source image is embedded into a background image to generate a new image, so that smooth transition and seamless fusion between the target image and the background image are realized, and the visual effect of a fusion transition zone is improved. At present, image fusion methods mainly include a weighted average method, a multi-resolution method, a gradient domain-based fusion method, and the like. The weighted average method, also called feathering method, is simple and fast in calculation speed, and has the defects of poor fusion effect and difficulty in eliminating ghost effect of motion on a target; the multi-resolution method is based on the principle that an image is decomposed into a series of sub-band images with different resolutions, transition regions with different sizes are used for fusion in different sub-bands, and then an image of an overlapped region under the original resolution is synthesized by using a reconstruction algorithm, but the method needs to be filtered for many times, has large calculation amount, and is easy to cause signal attenuation, thereby causing image blurring; the fusion method based on the gradient domain is essentially to realize the gradient migration of the source image to the target image by solving a Poisson equation, and simultaneously ensure the seamless fusion at the boundary and adjust the brightness deviation to obtain the final fusion image. The gradient reflects the most remarkable part of the local brightness change of the image, and the method is more suitable for the characteristic that the human visual system is very sensitive to the brightness change of the image.
At present, the technology which is widely applied is to apply some new technology to image fusion based on gradient domain. For example, poisson image editing is one of the hottest research directions. Poisson image editing is an image editing method based on Poisson equation and proposed by Perez et al, and the method utilizes an image gradient field to conduct guided interpolation on a region to be fused, resolves the image fusion problem into a problem of minimizing the difference value between the gradient field of the region to be synthesized and a target image guidance gradient field, and solves the variation problem by utilizing a Poisson equation, and achieves good image fusion effect.
However, in the prior art, when the target image and the background image are fused by using the poisson image editing method, at least two problems exist: 1) when the target image is blended into the background image, the boundary of the target image is blurred; 2) when the target image is blended to the boundary close to the background image, the visual effect of the boundary area is not ideal, a fuzzy phenomenon occurs, and when the color of the target image is greatly different from that of the background image, the original color of the target image cannot be ensured by the poisson image editing method.
Therefore, it is necessary to improve the prior art and improve the quality of the fused image to meet the requirements of people on the mode and quality of image fusion.
Disclosure of Invention
The present invention is directed to overcome the above-mentioned drawbacks of the prior art, and to provide an image fusion method to improve the display effect after image fusion.
According to a first aspect of the present invention, an image fusion method is provided. The method comprises the following steps:
step 1: determining the range of the original target image boundary to be expanded by taking the original target image boundary to be fused as a reference;
step 2: performing virtual pixel processing on the expanded range to obtain a target image after the virtual pixel processing;
and step 3: and fusing the target image after the virtual pixel processing and the background image by utilizing Poisson image editing.
In one embodiment, the range of the original target image boundary to be expanded is determined according to the following steps:
step 21: dividing the background image into n sub-blocks in parallel, wherein n is an integer greater than or equal to 1;
step 22: determining the times threshold of each sub-block for virtual pixel processing by using the received M frames of images of the original target image, and respectively marking the times threshold as T1，T2，…，TnWherein M is an integer greater than or equal to 1;
step 23: and for the subsequent frames of the received original target image, calculating a range threshold T of the original target image for virtual pixel processing by using the determined number threshold of the virtual pixel processing of each sub-block.
In one embodiment, in step 22, for the received M frames of the original target image, one sub-block b of the divided background image is selectedkPerforming the following substeps:
step 221: determining the boundary of the original target image to appear in the sub-block b each time by optimizing the target function by using an iterative methodkThe number of times of virtual pixel processing is performed is denoted as C1、C2、……、CmThe objective function is expressed as:
wherein Q isiIs the average pixel value, Q, of the target image boundary after the ith virtual pixel processing0Representing the average pixel value, Q, over the boundary of the original target image without virtual pixel processingiIs shown as
Omega is the sub-block bkThe boundary of the target image after the ith virtual pixel processing, p is a pixel point on the boundary, ft(p) is the color value of p in the background image, fs(p) is the color value of p in the target image;
step 222: according to the obtained C1、C2、……、CmDetermining the sub-block b by weighted averagekNumber threshold T for virtual pixel processingkAnd k is 1 to n.
In one embodiment, in step 23, for the subsequent frame of the received original target image, the range threshold T of the original target image for performing virtual pixel processing is determined according to the following sub-steps:
step 231: judging the number of the subblock of the background image at which the boundary of the original target image in the subsequent frame is positioned;
step 232: and acquiring a virtual pixel processing time threshold corresponding to the sub-block number of the background image, and taking the maximum value of the virtual pixel processing time threshold as a range threshold T for performing virtual pixel processing on the original target image.
In one embodiment, M ranges from 1000 to 2000 frames.
In one embodiment, step 3 comprises:
step 31: when the distance between the boundary of the target image after the virtual pixel processing and the boundary of the background image is smaller than a distance threshold value, intercepting a pixel value of a predetermined area from the target image after the virtual pixel processing;
step 32: and copying the pixel values of the intercepted area to the corresponding position of the background image, and then utilizing Poisson image editing for fusion.
In one embodiment, the distance threshold is a distance of 1 to 10 pixels.
In one embodiment, the original target image boundary is a minimum bounding rectangle surrounding the original target image.
Compared with the prior art, the invention has the advantages that: the virtual pixel processing is carried out on the boundary of the original target image to be fused, so that the fuzzy phenomenon can not occur at the boundary of the target image after the images are fused; under the condition that the boundary of the target image is close to the boundary of the background image, the pixel value of the appropriate area of the target image is copied to the corresponding position of the background image, so that the problems of unnatural transition of the target image and blurring at the boundary of the target image after image fusion are solved, and the ideal image fusion effect is finally obtained.
Drawings
The invention is illustrated and described only by way of example and not by way of limitation in the scope of the invention as set forth in the following drawings, in which:
fig. 1(a) to 1(b) show schematic diagrams of an image fusion process;
FIG. 2 shows a flow diagram of an image fusion method according to one embodiment of the invention;
FIG. 3 illustrates a schematic diagram of virtual pixel processing of an original target image according to one embodiment of the invention;
FIG. 4 shows a schematic diagram of Poisson image fusion according to one embodiment of the present invention;
FIG. 5 illustrates a flow diagram for determining a virtual pixel processing range based on a self-learning approach in accordance with one embodiment of the present invention;
FIG. 6(a) is a diagram illustrating the effect of Poisson image fusion when the original target image is not subjected to virtual pixel processing;
FIG. 6(b) is a diagram illustrating the effect of Poisson image fusion after the original target image is subjected to virtual pixel processing;
fig. 6(c) is a diagram illustrating an effect of poisson image fusion after virtual pixel processing and boundary processing are performed on an original target image at the same time.
Detailed Description
In order to make the objects, technical solutions, design methods, and advantages of the present invention more apparent, the present invention will be further described in detail by specific embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The principle and preferred embodiment of image fusion of the present invention will be described below by taking poisson image editing as an example.
Image fusion is to embed a target image in a source image into a background image to generate a new image. For example, the basic process applied to image fusion in a video surveillance camera is: firstly, respectively acquiring a frame of non-target background image and a frame of target-existing background image by a transmitting end in the same scene, and acquiring the minimum circumscribed rectangle surrounding the target image by the acquired non-target background image and the target-existing image through preprocessing processes such as a background difference method and the like; transmitting the processed result (namely the target image in the minimum bounding rectangle surrounding the target image) to a receiving end; and the receiving end fuses the target image and the background image which are transmitted in real time. Referring to fig. 1, wherein fig. 1(a) illustrates a background image S without an object, fig. 1(b) illustrates a schematic diagram after embedding an object image into the background image, and a region I is a region after image fusion. In practical applications, the target image is usually a moving object, such as a person or a vehicle, and the background image is usually a stationary object, such as a road, a building, and the like. Compared with the method for simultaneously transmitting the background image containing the target to the receiving end in real time, the method for displaying the video monitoring by using the image fusion process can save transmission flow and save bandwidth. In the description herein, the target image region determined with the minimum bounding rectangle as the boundary is referred to as the original target image.
FIG. 2 shows a flow diagram of an image fusion method according to one embodiment of the invention. Briefly, the image fusion method comprises: expanding the boundary of the original target image to the periphery, and performing virtual pixel processing on the expanded region to obtain a target image subjected to virtual pixel processing; and fusing the target image subjected to the virtual pixel processing to the background image by using a Poisson image editing method to obtain a fusion result. Optionally, before the poisson image fusion is performed, if it is determined that the boundary of the target image after the virtual pixel processing is performed is close to the boundary of the background image, the target image area close to the boundary of the background image is intercepted, the pixel value of the intercepted area is copied to the corresponding position of the background image, and then the poisson image fusion is performed.
Specifically, the image fusion method of the present invention includes the steps of:
and step S210, determining the position of the original target image fused into the background image.
In this step, a target image and a background image are acquired, and position coordinates of the target image fused into the background image are determined, and available information includes pixel values of the background image, pixel values of the target image, positions of the target image and the background image, and the like. The step can be implemented by using the prior art, for example, the target image is extracted by using a background difference method, that is, the image of the current frame is differentiated from a preset background image in a video image sequence, so that the information such as the position, the size and the like of the original target image can be obtained.
Step S220, determining a range of the original target image to be expanded with reference to the boundary of the original target image, so as to perform virtual pixel processing on the expanded region.
The boundary of the original target image refers to a minimum circumscribed figure comprising the original target image, wherein the minimum circumscribed figure can be a minimum circumscribed rectangle, a minimum circumscribed circle or a minimum circumscribed figure of an irregular shape determined according to the shape of the target image.
The expanded region is obtained by expanding the boundary of the original target image to the periphery to obtain a new target image with a boundary range larger than that of the original target image, and performing virtual pixel processing on the expanded region. Referring to the schematic diagram of fig. 3, the inner rectangle represents the boundary of the original target image, the gray area represents the expanded range (i.e., the range that needs to be processed by virtual pixels), and the outer rectangle represents the boundary of the target image after being processed by virtual pixels.
In one embodiment, the pixel values may be used to indicate the range of the original target image that needs to be extended, for example, by a predetermined threshold (e.g., 5-10 pixel distance) with respect to the boundary of the original target image.
In a preferred embodiment, the number of times the original target image needs to be processed by virtual pixel processing is determined by an initialization and self-learning method (also referred to herein simply as a self-learning method), thereby determining the range where virtual pixel processing is needed. In short, the self-learning method comprises the following steps: determining the times of virtual pixel processing required by the received previous M frames of target images through an initialization and self-learning method; then, virtual pixel processing is performed on the target image of the subsequent frame using the value obtained by the learning. The detailed procedure of the self-learning method will be described below.
In step S230, the expanded region is subjected to virtual pixel processing.
In this step, the expanded region is subjected to virtual pixel processing to obtain a target image after the virtual pixel processing, which is also referred to as performing virtual pixel processing on the original target image herein.
In the patent, virtual pixel processing refers to extending to the periphery on the basis of the boundary of the original target image, and if virtual pixel processing is performed on the basis of the original target image once in the previous M-frame initialization-based method, the boundary of the original target image extends to the periphery by a pixel distance; in the subsequent frame (after M frames) based on the self-learning method, only one virtual pixel processing is needed, the processing range is T pixels, the distance of the threshold T pixels is directly expanded to the periphery from the boundary of the original target image, and the target image is changed into a new target image. As a result of the virtual pixel processing, the region of the target image is enlarged.
In step S240, it is determined whether the boundary of the target image after the virtual pixel processing is close to the boundary of the background image.
After the virtual pixel processing, optionally, the method further includes further determining a relative position between a boundary (i.e., an expanded boundary) of the target image after the virtual pixel processing and a boundary of the background image, and if the boundary distance of the target image after the virtual pixel processing is close to the boundary of the background image, performing step S250, otherwise, directly performing a poisson fusion operation on the target image after the virtual pixel processing, that is, performing step S260.
In one embodiment, a predetermined pixel threshold is used to determine whether the target image after the virtual pixel processing is close to the boundary of the background image, for example, when the boundary of the target image after the virtual pixel processing is 1 to 10 rows of pixel points away from the boundary of the background image, the target image is considered to be close to the background image, and preferably, the target image after the virtual pixel processing is judged to be close when the boundary is 1 to 6 rows of pixel points away.
And step S250, intercepting a target image area close to the boundary of the background image and copying the pixel value of the intercepted area to the corresponding position of the background image.
When it is determined that the boundary of the target image after the virtual pixel processing is close to the boundary of the background image, several rows of matrix pixel values (for example, 6 rows of pixel values) close to the boundary of the background image are cut out from the region of the target image after the virtual pixel processing, and then the cut-out region is directly copied to the corresponding position in the background image. By the method, the phenomena of unnatural transition of the target image and fuzzy appearance at the boundary after Poisson fusion can be eliminated, so that an ideal fusion effect is obtained.
And step S260, carrying out image fusion by using a Poisson image editing method.
And carrying out image fusion on the target image subjected to the virtual pixel processing and the background image by using a Poisson image editing method.
The idea of poisson image editing is that under the condition that a boundary (the boundary of a background image) is ensured to be unchanged, a group of specific gradient change graphs are used as guidance to obtain an image of a fusion part, so that the gradient change trend of a fusion area is closest to the change trend of corresponding pixels of a source image (in the application, a target image processed by virtual pixels is used for representing the source image). The method and the device eliminate the splicing trace during image fusion through the virtual pixel processing process, and achieve the visual effect of seamless fusion.
Specifically, in connection with fig. 4, the principle of poisson image editing is to introduce a gradient vector field V such that the difference between the gradient field and the target gradient field is minimized, in order to solve for the unknown scalar f, i.e.:
where Ω is a closed subset on the background image, i.e. the fusion region,
is an edge portion of the fused region Ω, # f denotes the first order gradient of f (i.e., the gradient of the sought image), f is the pixel value inside Ω of the fused image (i.e., the sought image region) (which is an unknown scalar function), f is the pixel value outside Ω of the fused image Ω, which is defined at the boundary of Ω
A known scalar function of (a) the (c),
is gradient operator, (x, y) is image pixel point coordinate, g is target imageAnd V is the guide field of the target image (i.e., image g in fig. 4).
The solution vector of equation (1) can be expressed by a poisson equation with Dirichlet boundary conditions (Dirichlet boundary conditions) as follows:
wherein V is a guide field of the target image,
being the laplacian, div (i.e., # means a gradient of V, i.e., # g (g is the target image),
(u, v) represent the gradient fields of the target image in the x, y directions, respectively. And solved on the three RGB color channels using equation (2), respectively.
Carrying out finite differential discretization on the formula (1) to enable fpThe value of the function f at the point p of the pixel is calculated, the aim is to solve the problem of fΩ＝{fpP is equal to omega. The optimal solution of equation (1) satisfies the following equation (3):
wherein, | NpFour-way set N with | as pixel point ppThe number of elements, | Np|∈[1,4]，<p，q>Represents a pair of pixels, and q ∈ Np，
Is that
At directed edges [ p, q ]]The projected value of (a).
The formula (3) is a linear equation, and the pixel value in omega, that is, the fused image, can be obtained by solving the formula (3). In the request ofWhen solving, for example, the method can be used to solve f by using a super-relaxation Gauss-Seidel iterative method or a multiple-grid methodp，fpI.e. the pixel value of the fused p point, the solving process belongs to the prior art and is not described herein again.
It should be emphasized again that, during the poisson fusion operation executed in step S260, the referred target image refers to the target image after the virtual pixel processing of the present invention; in the case where step S240 and step S250 are included, the background image involved in the poisson image fusion refers to the background image to be obtained after the processing of step S250.
The following describes a process of determining a virtual pixel processing range required for the original target image by using initialization and self-learning methods, and an example of processing one pixel at a time when the background image is blocked and virtual pixel processing is performed in an initialization stage is described.
Referring to FIG. 5, a method for determining a virtual pixel processing range based on initialization and self-learning methods according to one embodiment of the present invention is shown, which briefly comprises: dividing a background image acquired in the same scene into a plurality of sub-blocks; initializing the original target image boundary in each sub-block to calculate a learning value C of each sub-block where the M frames of original target image boundaries are located and which needs to be subjected to virtual pixel processing; obtaining the fixed threshold value T of each sub-block by carrying out weighted average on the learning value of each sub-blockkAfter M frames, when the target image of the subsequent frame is received again, directly analyzing sub-blocks of the background image in which the boundary of the target image of the subsequent frame is distributed, and self-learning the obtained historical threshold value T by utilizing the sub-blockskAnd determining the range of the original target image needing virtual pixel processing.
Specifically, the embodiment of fig. 5 includes the following steps:
in step S510, the background image is divided into n sub-blocks in parallel.
Dividing the background image collected in the same scene into a plurality of sub-blocks marked as b1、b2、…，bnFor example, the stroke may be determined based on the size of the background image or the quality requirements for the fused imageThe number n of sub-blocks can be any integer larger than or equal to 1 theoretically, the background image is not partitioned when n is equal to 1, the larger the value of n is, the more accurate the threshold value obtained by self-learning is, and the higher the quality of the fused image is.
And step S520, calculating a learning value C of the original target image in each sub-block for virtual pixel processing by a self-learning method for the original target image of the previous M frames.
In this step, theoretically, M may be any integer greater than or equal to 1, and the larger the value of M, the more accurate the result of self-learning will be, but the speed of self-learning will be reduced, and in a preferred embodiment, M is set to 1000 + 2000 frames in order to balance the accuracy of self-learning and the learning speed.
Specifically, step S520 includes the following sub-steps:
step S521, initialization phase
Sub-block b of background image1For example, for the previous M frames, when the original target image boundary appears in the sub-block b for the first time1In pair b1Each pixel point on the boundary of the original target image sequentially extends to the boundary where the adjacent pixel point is located, and each time the pixel point extends (for example, one pixel point), the pixel point is recorded as performing virtual pixel processing once, and the average pixel value of each pixel point on the boundary before virtual pixel processing and after each virtual pixel processing is calculated, and the formula is expressed as follows:
wherein Q isiRepresenting the average pixel value of the boundary of the target image after the ith virtual pixel processing, omega representing the boundary of the current target image (i.e. after the ith virtual pixel processing) in the sub-block, p representing the pixel point on the boundary after the ith virtual pixel processing, ft(p) color value of p in background image, fs(p) represents a color value of p in the target image, i represents the number of times the original target image is subjected to the virtual pixel processing to the surroundings, and may be an arbitrary integer of 1 or more, and in this embodiment, i is taken as 20 as an exampleAnd (4) explanation.
In order to eliminate the phenomenon of fuzzy boundary of the fused target image and reduce the color change at the boundary, for the sub-block b1The absolute difference between the average pixel value before the virtual pixel processing and the average pixel value after the ith virtual pixel processing is minimized, that is:
wherein Q is0Representing the average pixel value, Q, over the boundary of the original target image without virtual pixel processingiRepresenting the average pixel value at the boundary after the i-th virtual pixel processing.
It should be understood that in this embodiment, though with sub-block b1For example, when any sub-block b of the divided background image is usedk(i.e., k is 1 to n), similar processing is performed, and the process is equally applicable.
Step S522, iterative optimization procedure
i) B sub-block b1The boundary which is not processed by the virtual pixel is used as the initial boundary, the pixel values of each point on the boundary and the background image at the corresponding position are substituted into the formula (4), and the average pixel value Q on the boundary at the moment is calculated0。
ii), extending from the current boundary to the boundary where the adjacent pixel points are located, namely performing virtual pixel processing once, wherein the virtual pixel processing once in the initialization stage represents that the original target image extends a pixel distance to the periphery, substituting the processed boundary and the pixel values of all points on the background image at the corresponding position into a formula (4), and calculating the average pixel value Q on the boundary at the moment1。
iii) substituting the results obtained in step i and step ii into the above formula (5), and recording the results as
iv) returning to the steps ii and iii, and finding out 20 times of iteration processes
And recording the value of i at the moment, namely expanding the distance of i pixels to the periphery on the basis of the initial boundary to meet the requirement that the blur around the boundary of the fused target image disappears, and setting the value of i at the moment as a learning value C1。
Step S523, self-learning stage
For the previous M frames of target images, the boundaries of the target images when other frames are received appear in sub-block b independently 2 nd time, 3 rd time, … … th time1Then, the learning values are updated in real time according to the procedure of step S522 and recorded as C2、C3、……、Cm。
Next, a sub-block b is determined based on the obtained learning value1Range threshold T for which virtual pixel processing is required1. For example, the threshold T may be obtained by calculating each learning value obtained by m iterations by a weighted average method1If T is1And if the number is a decimal number, rounding up. T is1That is, when the target image of any frame boundary in the b1 sub-block in the M-frame video image is poisson-fused with the background image, the range region in which the initial boundary needs to be processed by virtual pixels to the periphery in order to make the target image boundary blur just disappear.
Similarly, the boundary of the received previous M frames of target images calculated according to the method is respectively present in the sub-block b2Sub-block b3… … sub-block bnThreshold value T in2、T3……TnAnd setting the threshold value in the sub-block to be 0 if the boundary of the target image does not appear in the sub-block.
To further understand the self-learning process, the background image is divided into 25 sub-blocks, i.e., n is 25, and the threshold of each sub-block is labeled as follows.
| T1 | T6 | T11 | T16 | T21 |
| T2 | T7 | T12 | T17 | T22 |
| T3 | T8 | T13 | T18 | T23 |
| T4 | T9 | T14 | T19 | T24 |
| T5 | T10 | T15 | T20 | T25 |
Obtaining the threshold value T corresponding to each sub-block in the background image after initialization and self-learning stageskSuppose a threshold TkAs follows:
| 5 | 7 | 8 | 0 | 5 |
| 4 | 4 | 6 | 5 | 4 |
| 6 | 3 | 5 | 7 | 8 |
| 5 | 6 | 3 | 5 | 6 |
| 3 | 5 | 5 | 6 | 7 |
step S530, determining the range of the target image of the subsequent frame needing virtual pixel processing according to the self-learning obtained value.
After the initialization and self-learning process of the target image of the previous M frames, the threshold value T of the virtual pixel processing of each sub-block is obtainedkWhen a subsequent frame is received, directly analyzing sub-blocks of the background image in which the boundary of the original target image in the frame is distributed, and recording the history T obtained from the sub-blockskAnd at each value TkWhere the maximum value is found and is denoted as T, the rectangular box represents the smallest rectangle surrounding the target image whose boundary is distributed at T7、T8、T9、T12、T14、T17、T18、T19In the sub-block, the corresponding threshold value TkRespectively 4, 3, 6, 3, 5, 7, and 5, and if the maximum value T of these values is 7, then the range distance of 7 pixels is processed by virtual pixels to the periphery on the basis of the minimum bounding rectangle (i.e. the range distance of 7 pixels is directly extended to the periphery on the basis of the minimum bounding rectangle), that is, for the subsequent frame, the range distance of T pixels is processed by virtual pixels to the periphery on the boundary of the original target image, so as to obtain the target image after virtual pixel processing.
It should be noted that: in the initialization and self-learning embodiment of the present invention, the method includes comparing the number of frames of the original target image with M, and when the number of frames of the original target image is less than or equal to M, performing virtual pixel processing by performing steps S521 and S522, processing the distance of one pixel at a time; when the number of frames of the original target image is greater than M, virtual pixel processing is carried out according to the range region processed by the virtual pixels obtained by self-learning, for example, if the obtained range region T is 7 pixels, 7 pixels are expanded to the periphery by taking the boundary of the original target image as a reference, and at the moment, the virtual pixel processing efficiency of the subsequent frame can be improved by taking the range of 7 pixels as a unit through one-time virtual pixel processing.
Compared with the prior art, the method and the device have the advantages that the virtual pixel processing is carried out on the original target image, and the effect of fusing the images can be remarkably improved according to the fact that the target image after the virtual pixel processing is close to the boundary of the background image. Referring to fig. 6(a) to 6(c), fig. 6(a) illustrates an effect diagram of poisson image fusion when the original target image is not subjected to virtual pixel processing, fig. 6(b) illustrates an effect diagram of poisson image fusion after the original target image is subjected to virtual pixel processing, fig. 6(c) illustrates an effect diagram of poisson image fusion after the original target image is subjected to virtual pixel processing and boundary processing at the same time, wherein a human body is the target image, and it can be seen that, in the poisson fusion image diagram 6(a) which is not subjected to the method of the present invention, the boundary of the target image, for example, the head and leg regions have a blur phenomenon, in fig. 6(b) of the fusion image after being subjected to the virtual pixel processing of the present invention, the blur phenomenon of the head region substantially disappears, and the leg region near the boundary of the background image has a certain blur phenomenon, when the boundary of the target image is further improved by the present invention (see fig. 6 (c)), the blurring phenomenon of the leg region is almost eliminated, so that the target image is fused to the background image in a natural transition without the blurring phenomenon.
It should be noted that, although the steps are described in a specific order, the steps are not necessarily performed in the specific order, and in fact, some of the steps may be performed concurrently or even in a changed order as long as the required functions are achieved.
The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that retains and stores instructions for use by an instruction execution device. The computer readable storage medium may include, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (9)
1. An image fusion method comprising the steps of:
step 1: determining the range of the original target image boundary to be expanded by taking the original target image boundary to be fused as a reference;
step 2: performing virtual pixel processing on the expanded range to obtain a target image after the virtual pixel processing;
and step 3: fusing the target image after the virtual pixel processing and the background image by utilizing Poisson image editing;
determining the range of the original target image boundary needing to be expanded according to the following steps:
step 21: dividing the background image into n sub-blocks in parallel, wherein n is an integer greater than or equal to 1;
step 22: determining the times threshold of each sub-block for virtual pixel processing by using the received M frames of images of the original target image, and respectively marking the times threshold as T1，T2，…，TnWherein M is an integer greater than or equal to 1;
step 23: and for the subsequent frames of the received original target image, calculating a range threshold T of the original target image for virtual pixel processing by using the determined number threshold of the virtual pixel processing of each sub-block.
2. The image fusion method according to claim 1, wherein in step 22, for the received M frames of the original target image, one sub-block b of the divided background image is usedkPerforming the following substeps:
step 221: determining the boundary of the original target image to appear in the sub-block b each time by optimizing the target function by using an iterative methodkThe number of times of virtual pixel processing is performed is denoted as C1、C2、…、CmThe objective function is expressed as:
wherein Q isiIs the average pixel value, Q, of the target image boundary after the ith virtual pixel processing0Representing the average pixel value, Q, over the boundary of the original target image without virtual pixel processingiIs shown as
Omega is the boundary of the target image after the ith virtual pixel processing in the sub-block k, p is the pixel point on the boundary, ft(p) is p in the backgroundColor values in images, fs(p) is the color value of p in the target image;
step 222: according to the obtained C1、C2、…、CmDetermining the sub-block b by weighted averagekNumber threshold T for virtual pixel processingkAnd k is 1 to n.
3. The image fusion method according to claim 1, wherein, in step 23, for the subsequent frame of the received original target image, the range threshold T of the original target image for virtual pixel processing is determined according to the following sub-steps:
step 231: judging the number of the subblock of the background image at which the boundary of the original target image in the subsequent frame is positioned;
step 232: and acquiring a virtual pixel processing time threshold corresponding to the sub-block number of the background image, and taking the maximum value of the virtual pixel processing time threshold as a range threshold T for performing virtual pixel processing on the original target image.
4. The image fusion method according to claim 1, wherein M has a value in the range of 1000 to 2000 frames.
5. The image fusion method according to any one of claims 1 to 3, wherein step 3 comprises:
step 31: when the distance between the boundary of the target image after the virtual pixel processing and the boundary of the background image is smaller than a distance threshold value, intercepting a pixel value of a predetermined area from the target image after the virtual pixel processing;
step 32: and copying the pixel values of the intercepted area to the corresponding position of the background image, and then utilizing Poisson image editing for fusion.
6. The image fusion method of claim 5, wherein the distance threshold is 1 to 10 pixel distance.
7. The image fusion method according to any one of claims 1 to 3, wherein the original target image boundary is a minimum bounding rectangle surrounding the original target image.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
9. A computer device comprising a memory and a processor, on which memory a computer program is stored which is executable on the processor, characterized in that the steps of the method according to any of claims 1 to 7 are implemented when the processor executes the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810358534.XA CN110390657B (en) | 2018-04-20 | 2018-04-20 | Image fusion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810358534.XA CN110390657B (en) | 2018-04-20 | 2018-04-20 | Image fusion method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110390657A CN110390657A (en) | 2019-10-29 |
| CN110390657B true CN110390657B (en) | 2021-10-15 |
Family
ID=68283561
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810358534.XA Active CN110390657B (en) | 2018-04-20 | 2018-04-20 | Image fusion method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110390657B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111445408A (en) * | 2020-03-25 | 2020-07-24 | 浙江大华技术股份有限公司 | Method, device and storage medium for performing differentiation processing on image |
| CN111524100B (en) * | 2020-04-09 | 2023-04-18 | 武汉精立电子技术有限公司 | Defect image sample generation method and device and panel defect detection method |
| CN112288666B (en) * | 2020-10-28 | 2024-07-05 | 维沃移动通信有限公司 | Image processing method and device |
| CN117278726A (en) * | 2020-12-31 | 2023-12-22 | 上海丹诺西诚智能科技有限公司 | Projection pattern splicing method and system for multiple projection light sources |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101472162A (en) * | 2007-12-25 | 2009-07-01 | 北京大学 | Method and device for embedding and recovering prime image from image with visible watermark |
| CN101770649A (en) * | 2008-12-30 | 2010-07-07 | 中国科学院自动化研究所 | Automatic synthesis method for facial image |
| CN101945223A (en) * | 2010-09-06 | 2011-01-12 | 浙江大学 | Video consistent fusion processing method |
| CN102663766A (en) * | 2012-05-04 | 2012-09-12 | 云南大学 | Non-photorealistic based art illustration effect drawing method |
| CN102903093A (en) * | 2012-09-28 | 2013-01-30 | 中国航天科工集团第三研究院第八三五八研究所 | Poisson image fusion method based on chain code mask |
| CN104717574A (en) * | 2015-03-17 | 2015-06-17 | 华中科技大学 | Method for fusing events in video summarization and backgrounds |
| CN105096287A (en) * | 2015-08-11 | 2015-11-25 | 电子科技大学 | Improved multi-time Poisson image fusion method |
| CN105608716A (en) * | 2015-12-21 | 2016-05-25 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| CN106056537A (en) * | 2016-05-20 | 2016-10-26 | 沈阳东软医疗系统有限公司 | Medical image splicing method and device |
| CN106530265A (en) * | 2016-11-08 | 2017-03-22 | 河海大学 | Adaptive image fusion method based on chromaticity coordinates |
| CN106846241A (en) * | 2015-12-03 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of method of image co-registration, device and equipment |
-
2018
- 2018-04-20 CN CN201810358534.XA patent/CN110390657B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101472162A (en) * | 2007-12-25 | 2009-07-01 | 北京大学 | Method and device for embedding and recovering prime image from image with visible watermark |
| CN101770649A (en) * | 2008-12-30 | 2010-07-07 | 中国科学院自动化研究所 | Automatic synthesis method for facial image |
| CN101945223A (en) * | 2010-09-06 | 2011-01-12 | 浙江大学 | Video consistent fusion processing method |
| CN102663766A (en) * | 2012-05-04 | 2012-09-12 | 云南大学 | Non-photorealistic based art illustration effect drawing method |
| CN102903093A (en) * | 2012-09-28 | 2013-01-30 | 中国航天科工集团第三研究院第八三五八研究所 | Poisson image fusion method based on chain code mask |
| CN104717574A (en) * | 2015-03-17 | 2015-06-17 | 华中科技大学 | Method for fusing events in video summarization and backgrounds |
| CN105096287A (en) * | 2015-08-11 | 2015-11-25 | 电子科技大学 | Improved multi-time Poisson image fusion method |
| CN106846241A (en) * | 2015-12-03 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of method of image co-registration, device and equipment |
| CN105608716A (en) * | 2015-12-21 | 2016-05-25 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| CN106056537A (en) * | 2016-05-20 | 2016-10-26 | 沈阳东软医疗系统有限公司 | Medical image splicing method and device |
| CN106530265A (en) * | 2016-11-08 | 2017-03-22 | 河海大学 | Adaptive image fusion method based on chromaticity coordinates |
Non-Patent Citations (3)
| Title |
|---|
| Poisson image editing;Patrick Pérez 等;《ACM Transactions on Graphics》;20030731;第22卷(第3期);第313-318页 * |
| 图像融合与修复处理关键技术研究;谌明;《中国博士学位论文全文数据库信息科技辑》;20180115(第1期);I138-80 * |
| 基于泊松图像编辑方法的图像无缝拼合技术研究;张满满;《广东技术师范学院学报》;20150515(第5期);第59-62页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110390657A (en) | 2019-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7236545B2 (en) | Video target tracking method and apparatus, computer apparatus, program | |
| CN110390657B (en) | Image fusion method | |
| CN112991413A (en) | Self-supervision depth estimation method and system | |
| CN110675407B (en) | Image instance segmentation method and device, electronic equipment and storage medium | |
| CN110163188B (en) | Video processing and method, device and equipment for embedding target object in video | |
| CN113888437A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN113658197B (en) | Image processing method, device, electronic equipment and computer readable storage medium | |
| CN114565508B (en) | Virtual reloading method and device | |
| CN108961304B (en) | Method for identifying moving foreground in video and method for determining target position in video | |
| CN111382647B (en) | Picture processing method, device, equipment and storage medium | |
| CN111429485A (en) | Cross-modal filtering tracking method based on self-adaptive regularization and high-reliability updating | |
| CN115439803A (en) | Smoke optical flow identification method based on deep learning model | |
| CN108961293B (en) | Background subtraction method, device, equipment and storage medium | |
| Minematsu et al. | Adaptive background model registration for moving cameras | |
| CN113723272A (en) | Object detection method, system, device and medium based on multi-picture real-time splicing | |
| CN113065534A (en) | Method, system and storage medium based on portrait segmentation precision improvement | |
| Singh et al. | Gaussian and Laplacian of Gaussian weighting functions for robust feature based tracking | |
| CN111178200A (en) | Identification method of instrument panel indicator lamp and computing equipment | |
| CN113160297B (en) | Image depth estimation method and device, electronic equipment and computer readable storage medium | |
| CN113259605A (en) | Video matting method, system and storage medium based on prediction foreground mask prediction | |
| EP3018626B1 (en) | Apparatus and method for image segmentation | |
| KR20220064857A (en) | Segmentation method and segmentation device | |
| CN112132753A (en) | Infrared image super-resolution method and system for multi-scale structure guide image | |
| CN114255493A (en) | Image detection method, face detection device, face detection equipment and storage medium | |
| KR20230083212A (en) | Apparatus and method for estimating object posture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |