CN106600572A - Adaptive low-illumination visible image and infrared image fusion method - Google Patents

Abstract

The invention relates to an adaptive low-illumination visible image and infrared image fusion method, and belongs to the field of digital image processing. The method comprises the steps: image preprocessing, NSCT transformation, frequency domain coefficient fusion and an NSCT inversion. The method mainly achieves the multi-scale decomposition of a bright image and an infrared visible image, extracts high-frequency components and low-frequency components, and selects different fusion rules to carry out the frequency domain fusion according to the characteristics of the high-frequency components and low-frequency components. The NSCT inversion part is mainly used for carrying out the multi-scale inversion of the obtained fusion high-frequency and low-frequency components, obtaining the gray scale image after fusion, then carrying out the weighting of the gray scale image after fusion and an original color visible image, and obtaining a color fusion image. The method can effectively keep more detail information of the original image, can improve the contrast and resolution of the fusion image, and can be widely used in the fields of intelligent traffic, video monitoring, medical diagnosis, target detection and national defense security.

Description

Self-adaptive low-illumination visible light image and infrared image fusion method

Technical Field

The present invention belongs to the field of digital image processing.

Background

With the rapid development of sensor technology, the research of image fusion technology gradually becomes the hot field of research today. Different kinds of sensors, the image information of the captured scene is usually very different. In addition, the amount of information obtained by a single sensor is limited, and the application requirements of people are difficult to meet. The image fusion technology is to combine the image information obtained by two or more sensors to generate a new image. The fused new image has the advantages of rich detail information, high contrast and the like, is widely applied to various fields of medical diagnosis, target recognition and tracking, video monitoring, intelligent transportation, national defense safety and the like, and has high social application value. The problems of determination of fusion criteria, improvement of quality of fused images, registration of original images and the like are still technical difficulties in the field of image fusion.

The image fusion technology is mainly classified into a fusion method based on pixel point analysis, a fusion method based on feature calculation, and a fusion method based on decision analysis. The fusion method based on pixel point analysis directly analyzes and comprehensively processes the corresponding pixel points of the original image to obtain the fusion pixel value, and is the most widely and simply applied method in the field of image fusion. The image fusion method based on the traditional wavelet transform is poor in directional selection and easy to introduce blocking effect, so that the fused image is low in contrast and not beneficial to observation and judgment of human eyes. The image fusion method based on the traditional Contourlet transformation has the problems of small brightness dynamic range, missing detail information and the like of the fused image due to improper selection of the fusion criterion, and the function of a visual system is seriously influenced.

In recent years, the interest in the research of image fusion technology has increased. The low-illumination visible light image is often low in brightness and unclear, and is inconvenient for observation and identification of the target. At present, a unified mathematical model is not formed yet in an image fusion method based on pixel point analysis, and the quality of a fused image needs to be further improved. The traditional image fusion method has the problems of low contrast, small brightness dynamic range, less retained detail information and the like, and most of the methods only aim at fusion of gray level images and are difficult to be widely applied to various fields.

Disclosure of Invention

The invention provides a self-adaptive low-illumination visible light image and infrared image fusion method, which aims to solve the problems of low contrast and small brightness dynamic range of a fusion image.

The technical scheme adopted by the invention is that the method comprises the following steps:

(1) acquiring an original low-illumination visible light image and an infrared image:

under the condition of low illumination, an infrared camera and a visible light camera on a holder are used for respectively collecting an original color visible light image and an original infrared image, and the resolution ratios of the original color visible light image and the original color visible light image are 640 x 480;

(2) and matching the obtained original infrared image and the obtained color visible light image with scale-invariant feature points:

according to the scale invariant features of the original infrared image and the color visible light image, image registration is carried out, and images at any same position in a scene are ensured to be at the same position in the acquired image;

(3) extracting a brightness image of the registered color visible light image by using HIS transformation; according to the characteristics of the infrared sensor, enhancing the contrast of the infrared image after registration:

after registration, the color visible light image is a three-dimensional image, the infrared image is a two-dimensional image, and the image space dimensions are different, so that the brightness image I of the color visible light image is extracted by adopting color space HIS transformation_visibleThe calculation formula is shown as (1):

wherein, I_r、I_g、I_bPixel values of RGB three channels respectively expressed as a low-illumination visible light image;

the pixel brightness value of the infrared image is inverted, so that the contrast of the infrared image is enhanced;

I_IR＝L-I_ir(2)

wherein, I_IRFor the infrared image after reflection removal, I_irFor the registered infrared image, L is the gray level of the infrared image, and when the pixel of the infrared image is 8bit, L is 2^8 ^ 256;

(4) by using NSCT conversion of luminance image I_visibleAnd an infrared image I_IRPerforming multi-scale decomposition to obtain corresponding low-frequency components and high-frequency components respectively:

the device comprises two parts: the system comprises a non-subsampling pyramid filter and a non-subsampling direction filter bank, wherein the non-subsampling pyramid filter bank is used for realizing a multi-scale decomposition process, and the non-subsampling direction filter bank is used for realizing the decomposition of a frequency domain direction;

in the non-subsampled pyramid filter bank, the kth-level non-subsampled pyramid filter can be obtained by the following formula:

the non-subsampled pyramid filter also needs to satisfy the Bezout identity:

H₀(z)G₀(z)+H₁(z)G₁(z)＝1 (4)

wherein H₀(z)、G₀(z) low pass decomposition and synthesis filters that are non-subsampled pyramid filters; h₁(z)、G₁(z) high pass decomposition and synthesis filters that are non-subsampled pyramid filters:

the non-subsampling direction filter bank consists of a single sector filter, and performs upsampling operation on the non-subsampling direction filter, so that the frequency spectrum aliasing phenomenon can be effectively eliminated;

according to the non-subsampling pyramid and the non-subsampling direction filter bank, multi-scale transformation is carried out on the brightness image and the infrared image, and corresponding high-frequency components are extractedAnd low frequency componentsWherein J ≧ d ≧ 1, J denotes the imageThe total decomposition layer number is larger, the longer the algorithm operation time is, and J is 2, k is [2,16 ] to ensure the multi-scale and multi-direction characteristics of image decomposition]；

(5) Using different fusion criteria for high-frequency componentsAnd low frequency componentsRespectively fusing to obtain fused low-frequency components and fused high-frequency components;

the low-frequency component reflects the general picture information of the original image, the high-frequency component represents the texture detail information of the image, and different criteria are adopted for fusion respectively according to the characteristics of the high-frequency component and the low-frequency component to obtain the fused high-frequency component and the fused low-frequency component;

1) low frequency component fusion

And (4) for the corresponding low-frequency component obtained in the step (4), obtaining a fused low-frequency component by adopting an adaptive threshold fusion criterion:

wherein,for the fused low-frequency component, I_THIs a brightness threshold value, w_thAs the weight coefficient,corresponding low-frequency components of the brightness image and the infrared image respectively;

adopting a fusion criterion of self-adaptive threshold values for fusing low-frequency components of the brightness image and the infrared image, selecting the low-frequency component of the brightness image as the fused low-frequency component when the low-frequency component of the brightness image is greater than the threshold value, carrying out arithmetic mean on the low-frequency components of the brightness image and the infrared image when the low-frequency component of the brightness image is less than or equal to the threshold value, and calculating the fused low-frequency component;

2) high frequency component fusion

And (3) for the corresponding high-frequency component obtained in the step (4), obtaining a fused high-frequency component by adopting a fusion criterion of selecting the ignition times of the pulse coupling neural network, wherein the pulse coupling neural network is formed by connecting a feedback network and a plurality of neurons, each neuron mainly comprises a receiving part, a modulating part and a pulse generator, and the high-frequency component coefficient is the feedback input of the triggering pulse coupling neural network, and is shown as a formula (7):

wherein,is the output of the feedback section and is,is an input signal, (i, j) represents a pixel position, k represents the direction number of the d-th layer high-frequency component, and n' is the current iteration number;

the connection part of the pulse coupled neural network can be obtained from equation (8).

WhereinIs the output of the connecting portion, m and n are the ranges of connecting neurons, V_LTo normalize the coefficients, W_ij,mnWeight coefficients for other neurons connected;

the internal operation process of the pulse coupling neural network is obtained by calculation of a formula (9) and a formula (10);

wherein,is the internal state, β, a_θAnd V_θIn order to be a fixed factor,is a threshold value;

each iteration is as follows:

wherein X represents an original luminance image or an infrared image; n is the total iteration number;represents the total number of ignitions;

and selecting the corresponding high-frequency component with large ignition frequency as the fused high-frequency component by comparing the ignition frequency:

for the high-frequency components of the brightness image and the infrared image, fusing by adopting a criterion of taking a large number of ignition times based on a pulse coupling neural network, triggering the pulse coupling neural network by using a formula (7) according to the corresponding high-frequency components, calculating according to formulas (9) to (12) to obtain the corresponding ignition times, and selecting the high-frequency components corresponding to the large ignition times as the fused high-frequency components by comparing the ignition times by using a formula (13);

(6) and carrying out NSCT inverse transformation on the fused low-frequency component and high-frequency component to obtain a fused gray image I_Gray-F；

For the gray level fusion image obtained by carrying out multi-scale inverse transformation on the low-illumination gray level visible light image and the infrared image serving as the original image, the infrared image information of a low-illumination area is kept, meanwhile, the texture information of the low-illumination gray level image is kept, the contrast and the definition of the original visible light gray level image are improved, and people or other target objects in a scene can be judged and identified more accurately by human eyes;

(7) and (4) carrying out weight summation on the fused gray-scale image obtained in the step (6) and the registered color visible light image to obtain a final color fused image I_F；

Wherein C represents R, G, B color channels;original visible light pixel values representing the C color channel;the C-channel final fused image pixel value is represented, w is a weighting coefficient, and w is generally 0.5.

Under the condition of low illumination, the average brightness value of the color visible light image is lower, in order to ensure that more color information of the visible light image is reserved for the color fusion image, the gray level fusion image obtained in the step 6 and the original color visible light image are subjected to weight summation through a formula (14), and a final color fusion image is obtained.

The invention has the following beneficial effects:

(1) the invention relates to a method for improving the contrast and definition of a new image by performing frequency domain fusion on a low-illumination visible light image and an infrared image with the resolution of 640 x 480.

(2) The image obtained by the visible light sensor generally has rich color information and detail information, but under low-light conditions or other severe weather (such as haze and dust weather), a lot of scene information can be lost, and the all-weather work cannot be carried out. The infrared sensor can capture a hidden hot target through target heat radiation, is less influenced by scene brightness and severe weather, but has low contrast of a general infrared image and no color information, and is difficult to completely meet actual engineering application due to a single sensor. The invention adopts NSCT transformation, firstly carries out multi-scale decomposition on an original image, and extracts corresponding high-frequency components and low-frequency components. The low frequency components are then fused by a fusion criterion that employs an adaptive threshold. And fusing the high-frequency components by utilizing a fusion criterion based on the fact that the absolute value of the ignition times of the pulse coupling neural network is large. And then, a new fusion image is obtained by NSCT inverse transformation and weight summation, so that the blocking effect is effectively removed, the detail information of a scene target is highlighted, more detail information and color information are reserved, the contrast and the definition of the fusion image are improved, the observation and the judgment of human eyes are facilitated, and the fusion image can be applied to the conditions of low illumination and severe weather and has wide social application value.

(3) The invention is not only suitable for the fusion of gray level images, but also suitable for the fusion of color images, and the resolution ratio of the original image is more than 640 x 480.

(4) The invention has wide application value in the aspects of video monitoring, intelligent transportation, medical diagnosis, machine vision, national defense safety and the like.

Drawings

FIG. 1 is a flow chart of an algorithm in an application example of the present invention;

FIG. 2 is a schematic view of a pan/tilt head of an image capture system in an application example of the present invention;

FIG. 3(a) is an original low-luminance color visible image of scene 1 in an exemplary embodiment of the present invention;

FIG. 3(b) is an original infrared image of scene 1 in an application example of the present invention;

fig. 4(a) is a luminance image of a scene 1 after feature registration in an application example of the present invention;

fig. 4(b) is an infrared image of a scene 1 after feature registration in an application example of the present invention;

FIG. 5 is a color-blended image after scene 1 blending in an application example;

FIG. 6(a) is an original low-luminance color visible image of scene 2 in an exemplary embodiment of the present invention;

FIG. 6(b) is an original low-illumination infrared image of scene 2 in an application example of the present invention;

fig. 7(a) is a luminance image of a scene 2 after feature registration in an application example of the present invention;

fig. 7(b) is an infrared image of a scene 2 after feature registration in an application example of the present invention;

fig. 8 is a color fusion image after scene 2 fusion in the application example of the present invention.

Detailed Description

Comprises the following steps:

(1) acquiring an original low-illumination visible light image and an infrared image:

as shown in fig. 2, in the pan-tilt system for image acquisition according to the present invention, under a low illumination condition, an infrared camera and a visible light camera on the pan-tilt respectively acquire an original color visible light image and an infrared image, and the resolutions of the images are 640 × 480;

(2) and matching the obtained original infrared image and the obtained color visible light image with scale-invariant feature points:

because the camera positions are different, the focal lengths of the lenses are different and other external environments are influenced, images at the same position in the scene are at different positions in the color visible light image and the infrared image collected in the step (1), and therefore image registration is carried out according to the scale invariant features of the original infrared image and the color visible light image, and images at any same position in the scene are also at the same position in the collected images;

(3) extracting a brightness image of the registered color visible light image by using HIS transformation; according to the characteristics of the infrared sensor, enhancing the contrast of the infrared image after registration:

after registration, the color visible light image is a three-dimensional image, the infrared image is a two-dimensional image, and fusion errors can be caused by different image space dimensions, so that the brightness image I of the color visible light image is extracted by adopting color space HIS transformation_visibleThe calculation formula is shown as (1):

wherein, I_r、I_g、I_bPixel values of RGB three channels respectively expressed as a low-illumination visible light image;

the infrared sensor performs imaging by infrared radiation of an object in a scene, and the object (such as a person) is often high in brightness value in the image, but the object observed by human eyes is often low in brightness under a low-illumination condition at night. Therefore, the pixel brightness value of the infrared image is inverted, and the contrast of the infrared image is enhanced.

I_IR＝L-I_ir(2)

Wherein, I_IRFor the infrared image after reflection removal, I_irFor the registered infrared image, L is the gray level of the infrared image, and when the pixel of the infrared image is 8bit, L is 2^8 ^ 256;

(4) adopting NSCT transformation to convert the brightness image I_visibleAnd an infrared image I_IRPerforming multi-scale decomposition to obtain corresponding low-frequency components and high-frequency components respectively:

the NSCT transformation not only has the characteristics of Contourlet transformation such as multiresolution, localization and multi-directionality, but also has translation invariance, can eliminate the Gibbs phenomenon, and comprises two parts: a non-subsampled pyramid filter and a non-subsampled directional filter bank, the non-subsampled pyramid filter bank being used to implement a multi-scale decomposition process. The non-subsampling direction filter bank is used for realizing the decomposition in the frequency domain direction;

in the non-subsampled pyramid filter bank, the kth-level non-subsampled pyramid filter can be obtained by the following formula:

the non-subsampled pyramid filter also needs to satisfy the Bezout identity:

H₀(z)G₀(z)+H₁(z)G₁(z)＝1 (4)

wherein H₀(z)、G₀Low pass decomposition filter and synthesis with (z) non-subsampled pyramid filterA filter; h₁(z)、G₁(z) high pass decomposition and synthesis filters that are non-subsampled pyramid filters:

the non-subsampling directional filter bank is composed of a single fan-shaped filter, a sampling link is removed in order to achieve directional translation invariance of high-frequency components, the directional responses at lower and higher frequencies on the pyramid high-layer subbands easily cause a frequency spectrum aliasing phenomenon, and the non-subsampling directional filter is subjected to upsampling operation, so that the frequency spectrum aliasing phenomenon can be effectively eliminated.

According to the non-subsampling pyramid and the non-subsampling direction filter bank, multi-scale transformation is carried out on the brightness image and the infrared image, and corresponding high-frequency components are extractedAnd low frequency componentsWherein J is more than or equal to d and more than or equal to 1, and J represents the total decomposition layer number of the image. The larger the total decomposition layer number is, the longer the algorithm runs, and J is 2, k is [2,16 ] to ensure the multi-scale and multi-directional characteristics of image decomposition]；

(5) Using different fusion criteria for high-frequency componentsAnd low frequency componentsRespectively fusing to obtain fused low-frequency components and fused high-frequency components;

the low-frequency component reflects the general picture information of the original image, the high-frequency component represents the texture detail information of the image, and different criteria are adopted for fusion respectively according to the characteristics of the high-frequency component and the low-frequency component to obtain the fused high-frequency component and the fused low-frequency component;

1) low frequency component fusion

And (4) for the corresponding low-frequency component obtained in the step (4), obtaining a fused low-frequency component by adopting an adaptive threshold fusion criterion:

wherein,for the fused low-frequency component, I_THIs a brightness threshold value, w_thAs the weight coefficient,corresponding low-frequency components of the brightness image and the infrared image respectively;

for low-frequency components of the brightness image and the infrared image, fusion is carried out by adopting a fusion criterion of self-adaptive threshold, the overall average brightness value of the low-illumination visible light image is lower, wherein most of high-brightness pixel points are derived from background light, such as: car lights, street lights or other lighting devices, etc. Experimentally, the maximum corresponding to a low frequency component difference of 0.13% is derived from the background light, and the threshold is determined by equation (5), where w is 0.75 in the example. As can be seen from the formula (6), when the low-frequency component of the luminance image is greater than the threshold value, the low-frequency component of the luminance image is selected as the fused low-frequency component, and when the low-frequency component of the luminance image is less than or equal to the threshold value, the low-frequency components of the luminance image and the infrared image are arithmetically averaged to calculate the fused low-frequency component;

2) high frequency component fusion

And (3) for the corresponding high-frequency component obtained in the step (4), obtaining a fused high-frequency component by adopting a fusion criterion of selecting the ignition times of the pulse coupling neural network, wherein the pulse coupling neural network is formed by connecting a feedback network and a plurality of neurons, each neuron mainly comprises a receiving part, a modulating part and a pulse generator, and the high-frequency component coefficient is the feedback input of the triggering pulse coupling neural network, and is shown as a formula (7):

wherein,is the output of the feedback section and is,is an input signal, (i, j) represents a pixel position, k represents the direction number of the d-th layer high-frequency component, and n' is the current iteration number;

the connection part of the pulse coupled neural network can be obtained from equation (8).

WhereinIs the output of the connecting portion, m and n are the ranges of connecting neurons, V_LTo normalize the coefficients, W_ij,mnWeight coefficients for other neurons connected;

the internal operation process of the pulse coupling neural network is obtained by calculation of a formula (9) and a formula (10);

wherein,is the internal state, β, a_θAnd V_θIn order to be a fixed factor,is a threshold value;

each iteration is as follows:

wherein X represents an original luminance image or an infrared image; n is the total iteration number;represents the total number of ignitions;

and selecting the corresponding high-frequency component with large ignition frequency as the fused high-frequency component by comparing the ignition frequency:

for the high-frequency components of the brightness image and the infrared image, fusing by adopting a criterion of taking a large number of ignition times based on a pulse coupling neural network, triggering the pulse coupling neural network by using a formula (7) according to the corresponding high-frequency components, calculating according to formulas (9) to (12) to obtain the corresponding ignition times, and selecting the high-frequency components corresponding to the large ignition times as the fused high-frequency components by comparing the ignition times by using a formula (13);

(6) and carrying out NSCT inverse transformation on the fused low-frequency component and high-frequency component to obtain a fused gray image I_Gray-F；

For the gray level fusion image obtained by carrying out multi-scale inverse transformation on the low-illumination gray level visible light image and the infrared image serving as the original image, the infrared image information of a low-illumination area is kept, meanwhile, the texture information of the low-illumination gray level image is kept, the contrast and the definition of the original visible light gray level image are improved, and people or other target objects in a scene can be judged and identified more accurately by human eyes;

(7) and (4) carrying out weight summation on the fused gray-scale image obtained in the step (6) and the registered color visible light image to obtain a final color fused image I_F；

Wherein C represents R, G, B color channels;original visible light pixel values representing the C color channel;the C-channel final fused image pixel value is represented, w is a weighting coefficient, and w is generally 0.5.

Under the low illumination condition, the average brightness value of the color visible light image is low. In order to ensure that more color information of the visible light image is reserved in the color fusion image, the gray level fusion image obtained in the step 6 and the original color visible light image are subjected to weight summation through a formula (14) to obtain a final color fusion image.

The invention is further described below with reference to specific application examples and the accompanying drawings.

(1) And acquiring an original low-illumination visible light image and an infrared image.

In a low-illumination environment, an original low-illumination visible light image and an original infrared image are acquired through an infrared camera and a visible light camera of the holder system shown in fig. 2, and the resolution is 640 x 480. As shown in FIG. 2;

(2) matching the obtained original infrared image and the color visible light image with the scale-invariant feature points;

(3) extracting a brightness image of the registered color visible light image by using HIS transformation; enhancing the contrast of the infrared image after registration according to the characteristics of the infrared sensor;

(4) using NSCT transform, the luminance image I_visibleAnd an infrared image I_IRPerforming multi-scale decomposition to obtain corresponding low-frequency components and high-frequency components respectively;

(5) using different fusion criteria for high-frequency componentsAnd low frequency componentsRespectively fusing to obtain fused low-frequency components and fused high-frequency components;

the low-frequency component reflects the general picture information of the original image, the high-frequency component represents the texture detail information of the image, and different criteria are adopted for fusion respectively according to the characteristics of the high-frequency component and the low-frequency component to obtain the fused high-frequency component and the fused low-frequency component;

for the corresponding low-frequency components of the brightness image and the infrared image, adopting a fusion criterion of a self-adaptive threshold value to perform fusion, wherein the low-illumination visible light image has a low overall average brightness value which can be obtained by a formula (5) and a formula (6);

for the corresponding high-frequency components of the brightness image and the infrared image, fusing by adopting a criterion of taking a large number of ignition times based on a pulse coupling neural network, triggering the pulse coupling neural network by utilizing a formula (7), calculating according to formulas (8) to (12) to obtain the corresponding ignition times, and selecting the high-frequency component corresponding to the large ignition times as the fused high-frequency component by comparing the ignition times through a formula (13);

(6) NSCT inverse transformation is carried out on the fused low-frequency component and the fused high-frequency component, and then a fused gray image I is obtained_Gray-F；、

(7) And (4) carrying out weight summation on the fused gray image obtained in the step (6) and the registered color visible light image to obtain a final color fused image I_F；

Under the low illumination condition, the average brightness value of the color visible light image is low, the dynamic range is small, and the scene target information is unclear, as shown in fig. 3(a) and fig. 6(a), it is difficult to observe the outlines, trunks, etc. of pedestrians and vehicles in the scene; the infrared image collected by the infrared radiation of the scene target can clearly observe the target information which cannot be seen by the visible light sensor, but lacks color information, as shown in fig. 3(b) and fig. 6 (b);

as shown in fig. 4, the fusion image effectively retains color information of a street lamp, a tail lamp of an automobile and a sky area in the visible light image, and retains texture detail information of outlines (such as wheel parts) of the automobile, pedestrians, support bars of the street lamp and the like in the infrared image, as shown in fig. 6, the fusion image effectively retains color information of the automobile, lights of the street lamp, buildings and the like, and retains texture detail information of pedestrians, tree poles and the like in the infrared image; therefore, the method effectively fuses the infrared image and the color visible light image, not only can retain the color information of the visible light image, but also can obtain the texture detail information of the infrared image scene target, and can be widely applied to the fields of video monitoring, intelligent transportation, national defense safety and the like.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment, and all technical solutions belonging to the principle of the present invention belong to the protection scope of the present invention.

Claims (6)

1. A self-adaptive low-illumination visible light image and infrared image fusion method is characterized by comprising the following steps:

(1) acquiring an original low-illumination visible light image and an infrared image:

under the condition of low illumination, an infrared camera and a visible camera on a holder are used for respectively collecting an original color visible image and an original infrared image,

(2) and matching the obtained original infrared image and the obtained color visible light image with scale-invariant feature points:

according to the scale invariant features of the original infrared image and the color visible light image, image registration is carried out, and images at any same position in a scene are ensured to be at the same position in the acquired image;

(3) extracting a brightness image of the registered color visible light image by using HIS transformation; according to the characteristics of the infrared sensor, enhancing the contrast of the infrared image after registration:

(4) adopting NSCT transformation to convert the brightness image I_visibleAnd an infrared image I_IRPerforming multi-scale decomposition to obtain corresponding low-frequency components and high-frequency components respectively:

the device comprises two parts: the system comprises a non-subsampling pyramid filter and a non-subsampling direction filter bank, wherein the non-subsampling pyramid filter bank is used for realizing a multi-scale decomposition process, and the non-subsampling direction filter bank is used for realizing the decomposition of a frequency domain direction;

(5) using different fusion criteria for high-frequency componentsAnd low frequency componentsRespectively fusing to obtain fused low-frequency components and fused high-frequency components;

the low-frequency component reflects the general picture information of the original image, the high-frequency component represents the texture detail information of the image, and different criteria are adopted for fusion respectively according to the characteristics of the high-frequency component and the low-frequency component to obtain the fused high-frequency component and the fused low-frequency component;

(6) and carrying out NSCT inverse transformation on the fused low-frequency component and high-frequency component to obtain a fused gray image I_Gray-F；

For the gray level fusion image obtained by carrying out multi-scale inverse transformation on the low-illumination gray level visible light image and the infrared image serving as the original image, the infrared image information of a low-illumination area is kept, meanwhile, the texture information of the low-illumination gray level image is kept, the contrast and the definition of the original visible light gray level image are improved, and people or other target objects in a scene can be judged and identified more accurately by human eyes;

(7) and (4) carrying out weight summation on the fused gray-scale image obtained in the step (6) and the registered color visible light image to obtain a final color fused image I_F；

$I_F^C=I_{Gray-F}+w\×I_{Color-F}^C---\left(14\right)$

Wherein C represents R, G, B color channels;original visible light pixel values representing the C color channel;representing the final fused image pixel value of the C channel, wherein w is a weighting coefficient;

under the condition of low illumination, the average brightness value of the color visible light image is lower, in order to ensure that more color information of the visible light image is reserved for the color fusion image, the gray level fusion image obtained in the step 6 and the original color visible light image are subjected to weight summation through a formula (14), and a final color fusion image is obtained.

2. The adaptive low-illumination visible light image and infrared image fusion method according to claim 1, characterized in that: the resolution of the original color visible light image and the infrared image collected in the step (1) are 640 x 480.

3. The adaptive low-illumination visible light image and infrared image fusion method according to claim 1, characterized in that: in the step (3), after registration, the color visible light image is a three-dimensional image, the infrared image is a two-dimensional image, and the image space dimensions are different, so that the brightness image I of the color visible light image is extracted by adopting color space HIS transformation_visibleThe calculation formula is shown as (1):

$L_{visible}=\frac{I_r+I_g+I_b}{3}---\left(1\right)$

wherein, I_r、I_g、I_bPixel values of RGB three channels respectively expressed as a low-illumination visible light image;

the pixel brightness value of the infrared image is inverted, so that the contrast of the infrared image is enhanced;

I_IR＝L-I_ir(2)

wherein, I_IRFor the infrared image after reflection removal, I_irFor the registered infrared image, L is the gray level of the infrared image, and when the pixel of the infrared image is 8bit, L ^ 2^8 ^ 256.

4. The adaptive low-illumination visible light image and infrared image fusion method according to claim 1, wherein in the step (4):

in the non-subsampled pyramid filter bank, the kth-level non-subsampled pyramid filter can be obtained by the following formula:

$H_s^{eq}\left(z\right)=\left\{\begin{array}{cc}{H}_1\left(z^{2^{s-1}}\right)\underset{j=0}{\overset{s-2}{\Π}}{H}_0\left(z^{2^j}\right)& 2^k>s\≥1\\ \underset{j=0}{\overset{s-2}{\Π}}{H}_0\left(z^{2^j}\right)& s=2^k\end{array}\right.---\left(3\right)$

the non-subsampled pyramid filter also needs to satisfy the Bezout identity:

H₀(z)G₀(z)+H₁(z)G₁(z)＝1 (4)

wherein H₀(z)、G₀(z) low pass decomposition and synthesis filters that are non-subsampled pyramid filters; h₁(z)、G₁(z) high pass decomposition and synthesis filters that are non-subsampled pyramid filters:

the non-subsampling direction filter bank is composed of a single sector filter, and performs upsampling operation on the non-subsampling direction filter, so that the frequency spectrum aliasing phenomenon can be effectively eliminated.

According to the non-subsampling pyramid and the non-subsampling direction filter bank, multi-scale transformation is carried out on the brightness image and the infrared image, and corresponding high-frequency components are extractedAnd low frequency componentsWherein J is more than or equal to d and more than or equal to 1, and J represents the total decomposition layer number of the image. The larger the total decomposition layer number is, the longer the algorithm runs, and J is 2, k is [2,16 ] to ensure the multi-scale and multi-directional characteristics of image decomposition]。

5. The adaptive low-illumination visible light image and infrared image fusion method according to claim 1, wherein the low-frequency component fusion method in step (5) is as follows:

and (4) for the corresponding low-frequency component obtained in the step (4), obtaining a fused low-frequency component by adopting an adaptive threshold fusion criterion:

$I_{TH}=w_{th}\×max(I_{Visible}^J-I_{IR}^J)---\left(5\right)$

$I_F^{low}=\left\{\begin{array}{cc}{I}_{Visible}^J& I_{Visible}^J\≥I_{TH}\\ (I_{IR}^J+I_{Visible}^J)/2& otherwise\end{array}\right.---\left(6\right)$

wherein,for the fused low-frequency component, I_THIs a brightness threshold value, w_thAs the weight coefficient,corresponding low-frequency components of the brightness image and the infrared image respectively;

and for the low-frequency components of the brightness image and the infrared image, adopting a fusion criterion of a self-adaptive threshold value to perform fusion, selecting the low-frequency components of the brightness image as the fused low-frequency components when the low-frequency components of the brightness image are larger than the threshold value, performing arithmetic mean on the low-frequency components of the brightness image and the infrared image when the low-frequency components of the brightness image are smaller than or equal to the threshold value, and calculating the fused low-frequency components.

6. The adaptive low-illumination visible light image and infrared image fusion method according to claim 1, wherein the high-frequency component fusion method in step (5) is as follows:

and (3) for the corresponding high-frequency component obtained in the step (4), obtaining a fused high-frequency component by adopting a fusion criterion of selecting the ignition times of the pulse coupling neural network, wherein the pulse coupling neural network is formed by connecting a feedback network and a plurality of neurons, each neuron mainly comprises a receiving part, a modulating part and a pulse generator, and the high-frequency component coefficient is the feedback input of the triggering pulse coupling neural network, and is shown as a formula (7):

$F_{i,j}^{d,k}\left(n^{\′}\right)=I_{i,j}^{d,k}---\left(7\right)$

wherein,is the output of the feedback section and is,is an input signal, (i, j) represents a pixel position, k represents the direction number of the d-th layer high-frequency component, and n' is the current iteration number;

the connection part of the pulse coupled neural network can be obtained from equation (8).

$L_{i,j}^{d,k}\left(n^{\′}\right)=\exp (-a_L)L_{i,j}^{d,k}\left(n^{\′}\right)+V_L\underset{m,n}{\Σ}{W}_{ij,mn}{Y}_{ij,mn}^{d,k}(n^{\′}-1)---\left(8\right)$

WhereinIs the output of the connecting portion, m and n are the ranges of connecting neurons, V_LTo normalize the coefficients, W_ij,mnWeight coefficients for other neurons connected;

the internal operation process of the pulse coupling neural network is obtained by calculation of a formula (9) and a formula (10);

$U_{i,j}^{d,k}\left(n^{\′}\right)=E_{i,j}^{d,k}\left(n^{\′}\right)(1+{\mathrm{\βL}}_{i,j}^{d,k}(n^{\′}\left)\right)---\left(9\right)$

${\mathrm{\θ}}_{i,j}^{d,k}\left(n^{\′}\right)=\exp (-a_{\mathrm{\θ}}){\mathrm{\θ}}_{i,j}^{d,k}(n^{\′}-1)+V_{\mathrm{\θ}}{Y}_{i,j}^{d,k}(n^{\′}-1)---\left(10\right)$

wherein,is the internal state, β, a_θAnd V_θIn order to be a fixed factor,is a threshold value;

each iteration is as follows:

$Y_{i,j}^{d,k}\left(n^{\′}\right)=\left\{\begin{array}{cc}1& U_{i,j}^{d,k}\left(n^{\′}\right)>{\mathrm{\θ}}_{i,j}^{d,k}\left(n^{\′}\right)\\ 0& otherwise\end{array}\right.---\left(11\right)$

$\begin{array}{cccc}{T}_{X,ij}^{d,k}=\underset{n^{\′}=1}{\overset{N}{\Σ}}{Y}_{i,j}^{d,k}\left(n^{\′}\right)& X=Visible& or& IR\end{array}---\left(12\right)$

wherein X represents an original luminance image or an infrared image; n is the total iteration number;represents the total number of ignitions;

and selecting the corresponding high-frequency component with large ignition frequency as the fused high-frequency component by comparing the ignition frequency:

$I_F^{high}=\left\{\begin{array}{cc}{I}_{Visible}^{d,k}& T_{Visible,ij}^{d,k}\≥T_{IR,ij}^{d,k}\\ I_{IR}^{d,k}& otherwise\end{array}\right.---\left(13\right)$

and for the high-frequency components of the brightness image and the infrared image, fusing by adopting a criterion that the ignition frequency based on the pulse coupling neural network is large, triggering the pulse coupling neural network by using a formula (7) according to the corresponding high-frequency components, calculating according to formulas (9) to (12) to obtain the corresponding ignition frequency, and selecting the high-frequency component corresponding to the large ignition frequency as the fused high-frequency component by comparing the ignition frequency by using a formula (13).