JP4798558B2 - Moving object detection apparatus and moving object detection method - Google Patents

Description

  The present invention relates to a moving object detection apparatus and a moving object detection method for detecting a moving direction and a moving speed of a moving object based on a change in luminance of each pixel in a plurality of frames.

A method using a block matching method is known as a method for detecting a movement such as a moving direction and a moving speed of an object. In such a system, an object is detected by detecting the movement of an image in a block unit obtained by dividing a frame into an appropriate size between a plurality of frames captured at a predetermined frame rate. (For example, Patent Document 1).
Japanese Patent Laid-Open No. 2001-25021

  Thus, when detecting a moving body using a block matching method, control of a block size becomes important. For example, if the block size is too large, a plurality of movements are mixed in the block, and it becomes difficult to detect the movement in units of blocks. For example, if the block size is too small, the possibility of matching with an irrelevant block increases, and the motion detection accuracy decreases. Since an appropriate block size differs depending on the moving object to be detected, a complicated process for adaptively controlling the block size is required.

  In order to increase the detection accuracy of the moving object, it is effective to increase the frame rate. However, by increasing the frame rate, the moving distance of the object between one frame is reduced, and the detectable moving speed is increased. The lower limit becomes higher. That is, for example, when the moving distance between one frame is less than one pixel, the moving speed of the object cannot be obtained. In such a case, it is conceivable to lower the frame rate according to the moving object, but the control becomes complicated and the detection accuracy of an object moving at high speed is lowered.

  The present invention has been made in view of the above problems, and an object thereof is to detect a moving object with high accuracy at a fixed frame rate without requiring complicated processing such as block size control.

In order to achieve the above object, an animal body detection device of the present invention is composed of a plurality of photodiodes arranged in an array, and obtains a frame image at a predetermined time interval, and the plurality of photodiodes. A memory array configured by a plurality of storage elements provided in an array corresponding to the diode, storing the frame image acquired by the photodiode array, the frame image acquired by the photodiode array, A processing circuit that generates and outputs a motion signal indicating a moving direction and a moving speed of the moving object using the frame image stored in the memory array, and the processing circuit includes: The difference between the luminance of one pixel and the luminance of one pixel of the second frame, the first frame and the second frame The moving direction and moving speed of the moving object are detected based on the time interval between and the brightness of one pixel at the first position of the first frame and the vicinity of the first position of the second frame. A moving object detection unit that further detects a moving direction for each pixel based on a difference from the luminance of each pixel, and a depth direction of the moving object based on the moving direction for each pixel detected by the moving object detection unit A depth direction detection unit that detects a moving direction of the first frame, and a predicted luminance generation unit that generates a predicted luminance that is a luminance when the one pixel at the first position of the first frame is blurred by a blur filter. The depth direction detection unit determines whether the moving object is expanding or contracting in a two-dimensional direction based on the moving direction of each pixel, and when the moving object is contracting , Determines that the serial animal body is moving from the front to the back, when the moving object is enlarged, the animal body with determined to be moving from the back to the front, of the first frame The first difference obtained by subtracting the luminance of the one pixel at the first position from the predicted luminance and the luminance of the one pixel at the first position of the first frame An evaluation value is generated by multiplying the second difference subtracted from the luminance of one pixel at the first position, and when the evaluation value is a negative value, the moving object approaches the in-focus position. When it is determined that the object is moving in the direction, and the evaluation value is a positive value, it is determined that the moving object is moving in a direction away from the in-focus position .

  A moving object can be detected with high accuracy at a fixed frame rate without requiring complicated processing such as block size control.

== System configuration ==
FIG. 1 is a diagram showing a configuration of a moving object detection system including an image sensor according to an embodiment of the present invention. The moving object detection system 1 includes an image sensor (moving object detection apparatus) 10 and an optical lens 15. The image sensor 10 is a circuit that captures an image received from the optical lens 15 at a predetermined frame rate and detects the moving direction, moving speed, and the like of an object based on information of a plurality of frames. The focal point of the optical lens 15 is fixed.

== Hardware configuration ==
FIG. 2 is a diagram illustrating an example of a hardware configuration of the image sensor 10. The image sensor 10 includes a control circuit 21, a photodiode array (hereinafter referred to as “PD array”) 22, a memory array 23, a processing circuit 24, a horizontal shift register 25, and vertical shift registers 26 and 27.

  The control circuit 21 is a circuit that controls the entire image sensor 10 and outputs control signals for controlling various processes. The PD array 22 includes a plurality of photodiodes provided in an array, and converts an image received from the optical lens 15 into an analog image signal for each pixel and outputs the analog image signal. One image captured by the PD array 22 is referred to as a frame, and the frame generation interval is referred to as a frame rate. In the present embodiment, it is assumed that the frame rate is very high, for example, about 1000 frames / second.

  Under the control of the control circuit 21, data designating the columns of the PD array 22 is set in the horizontal shift register 25, and data designating the rows of the PD array 22 is set in the vertical shift register 26. That is, among the analog image signals of the frames captured by the PD array 22, the analog image signals of the pixels specified by the horizontal shift register 25 and the vertical shift register 26 are read and transmitted to the processing circuit 24.

  The memory array 23 includes a plurality of storage elements provided in an array, and stores the past several frames of image signals acquired by the PD array 22. Then, the processing circuit 24 uses the latest frame output from the PD array 22 and the past several frames stored in the memory array 23 to generate a motion signal indicating the moving direction and moving speed of the object. Output. Note that the processing circuit 24 reads frames from the PD array 22 and the memory array 23 in units of rows, and performs high-speed processing by performing processing on a plurality of columns of pixels in parallel.

  The PD array 22 corresponds to the frame acquisition unit of the present invention, and the memory array 23 corresponds to the frame storage unit of the present invention.

== Functional configuration ==
FIG. 3 is a diagram illustrating a configuration of functions realized by the processing circuit 24 of the image sensor 10. The processing circuit 24 includes a moving object detection unit 31, an intermediate luminance generation unit 32, a depth direction detection unit 33, and a predicted luminance generation unit 34.

  The moving object detection unit 31 detects the moving direction and moving speed of the object in the vertical and horizontal directions viewed from the optical lens 15. The intermediate luminance generation unit 32 generates intermediate luminance that is luminance between these two pixels based on the luminance of two adjacent pixels. That is, the intermediate luminance generated by the intermediate luminance generation unit 32 indicates the luminance of a virtual pixel located between two adjacent pixels. Note that the moving object detection unit 31 can detect the moving speed of the object whose moving distance between one frame is less than one pixel by using the intermediate luminance generated by the intermediate luminance generating unit 32.

  The depth direction detection unit 33 detects the moving direction of the object in the depth direction viewed from the optical lens 15. Further, the predicted luminance generation unit 34 predicts a change in luminance of each pixel caused by the action of the object.

== Up / down / left / right motion detection ==
First, detection of the movement of the object in the vertical and horizontal directions (two-dimensional direction) in the image sensor 10 will be described.

(1) Detection of presence / absence of motion First, an operation for detecting whether there is motion in the vertical and horizontal directions will be described. FIG. 4 is a diagram showing an outline of an operation for detecting whether there is a movement in the vertical and horizontal directions. In the figure, the x-axis and the y-axis are coordinates indicating the pixel position in the frame, the positive direction of the x-axis is the right direction, the negative direction of the x-axis is the left direction, the positive direction of the y-axis is the upward direction, The negative direction indicates the downward direction. The f-axis shows the time series of frames, with the positive direction indicating a new frame and the negative direction indicating an old frame. Note that the pixel at the position (x, y) in the frame f is represented as a pixel (x, y), and the luminance of the pixel (x, y) is represented as l _{f; x, y} .

The following expression (1) is an expression for obtaining an evaluation value d (x, y) for determining whether or not the pixel (x, y) has a motion.

As shown in Expression (1), the moving object detection unit 31 of the image sensor 10 has the luminance l _{f; x, y} of the pixel (x, y) of the frame f and the luminance l _{fk; x, y of the} preceding and following N frames. An evaluation value d is obtained based on the difference between Then, the moving object detection unit 31 determines that the pixel (x, y) has a motion if the evaluation value d is equal to or greater than a predetermined threshold value.

In addition, in order to reduce the memory size required for the image sensor 10 and the circuit scale, the number of frames necessary for obtaining the evaluation value d can be two. FIG. 5 is a diagram showing an outline of an operation for detecting whether there is a movement in the vertical and horizontal directions using two frames. The following equation (2) is an equation for obtaining the evaluation value d at that time.

As shown in Expression (2), the moving object detection unit 31 calculates the luminance l _{f; x, y} of the pixel (x, y) of the frame f and the pixel (x, y) of the previous frame f−1. ) Of the luminance l _{f-1; x, y} may be used to obtain the evaluation value d.

(2) Detection of moving direction Next, an operation for detecting the moving direction of the object in the vertical and horizontal directions will be described. FIG. 6 is a diagram showing an outline of an operation for detecting the left and right moving directions. As shown in FIG. 6 (a), rightward movement can be detected by shifting the pixels one frame at a time to the right to obtain a luminance difference. Further, as shown in FIG. 6B, the leftward movement can be detected by shifting the pixels one frame at a time to the left to obtain the luminance difference. Similarly, it is possible to detect the upward movement by shifting the pixel up one frame at a time to obtain the luminance difference, and shifting the pixel down one frame at a time. Thus, the downward movement can be detected by obtaining the luminance difference.

The following equations (3) to (6) are equations for _obtaining evaluation values m _up , m _down , m _right , and m _left for detecting such movement in the vertical and horizontal directions.

As shown in Expression (3) _{, the moving} object detection unit 31 of the image sensor 10 calculates the luminance l _{f; x, y} of the pixel (x, y) of the frame f and the luminance l _{fk; x, yk of the} preceding and following N frames. An evaluation value m _up for detecting upward movement is generated based on the difference between the brightness and the brightness. Similarly, as shown in the equations (4) to (6), the moving object detection unit 31 has an evaluation value m _down for detecting a downward movement and an evaluation value for detecting a rightward movement. m _right , an evaluation value m _left for detecting leftward movement is generated.

Here, if the object moves in a certain direction, the evaluation value in that direction is smaller than the evaluation value in the other direction. Therefore, the moving object detection unit 31 moves the object in the direction in which the evaluation value is minimum based on the evaluation values m _up , m _down , m _right , m _left and the evaluation value d described above. Can be determined.

Note that the moving object detection unit 31 may determine the vertical movement and the horizontal movement separately. For example, as shown in the following expression (7), the moving object detection unit 31 creates a matrix M _vertical with the evaluation values m _up and m _down and the evaluation value d, and the element of M _vertical is the minimum in the _vertical direction. It can be determined that it is moving in the direction of. Further, as shown in the following equation (8), the moving object detection unit 31 creates a matrix M _horizontal with the evaluation values m _left and m _right and the evaluation value d, and the element of M _horizontal is the smallest in the left-right direction. It can be determined that it is moving in the direction of.

In addition, in order to reduce the memory size required for the image sensor 10 and the circuit scale, the number of frames necessary for _obtaining the evaluation values m _up , m _down , m _right , m _left is set to two. You can also. FIG. 7 is a diagram showing an outline of an operation for detecting the vertical and horizontal movement directions using two frames. The following formulas (9) to (12) are formulas for obtaining evaluation values m _up , m _down , m _right , and m _left at that time.

As shown in FIG. 7A and Expression (9), the moving object detection unit 31 performs luminance l _{f−1; x, y} of the pixel (x, y) of the frame f−1 and the pixel ( The evaluation value m _up can be obtained from the difference from the luminance l _{f; x, y + 1} of the pixel (x, y + 1) adjacent to the upper side of x, y). Similarly, the moving object detection unit 31 can obtain the evaluation values m _down , m _right , and m _left as shown in FIGS. 7B to 7D and Equations 10 to 12.

== Moving speed detection ==
Next, detection of the moving speed of the object in the vertical and horizontal directions in the image sensor 10 will be described. The moving speed of the object can be obtained by dividing the moving distance of the object by the time required for the movement. The frame rate of the image sensor 10 of the present embodiment is very high at about 1000 frames per second. For this reason, the distance that the object moves during one frame may be less than one pixel. In such a case, if the moving speed is simply detected based on the moving distance in units of pixels between two consecutive frames, the detection accuracy of the moving speed is lowered. Therefore, the image sensor 10 detects the moving speed of the object by using two methods of “decimal precision determination” and “detection interval control”.

(1) Decimal accuracy determination First, detection of the moving speed of an object by decimal accuracy determination will be described. FIG. 8 is a diagram showing an outline of an operation for detecting the moving speed of an object with decimal precision. As described above, the object may not move by one pixel during one frame. Therefore, as shown in FIG. 8A, based on the luminance of the pixel (x, y) of the frame f and the luminance of the pixel (x, y + 1) of the frame f, the pixel (x, y) and the pixel ( The luminance of the virtual pixel between x, y + 1), that is, the luminance of the pixel with decimal precision is obtained. Then, by detecting the difference between the brightness of the pixel with decimal precision and the brightness of the pixel (x, y) of the previous frame f−1, it is detected that the pixel has moved 1/2 pixel in the upward direction. Can do. Similarly, as shown in FIGS. 8B to 8D, it can be detected that the pixel has moved 1/2 pixel in the downward direction, 1/2 pixel in the right direction, and 1/2 pixel in the left direction. it can.

Evaluation values m _{up1 / 2} , m _{down1 / 2} , m _{right1 / 2} , m _{left1 / 2} for detecting that the following expressions (13) to (16) are moved by 1/2 pixel in the vertical and horizontal directions _: Is an expression for obtaining.

As shown in Expression (13), the intermediate luminance generation unit 32 of the image sensor 10 is a pixel at the position (x, y) (Claim 1: first position) of the frame f (Claim 1: first frame). Luminance l _{f; x, y} and the luminance l _{f; x, y + 1} of the pixel at the position (x, y + 1) adjacent to the upper side of the position (x, y) (Claim 1: the second position). Thus, the luminance (l _{f; x, y + 1} + l _{f; x, y} ) / 2 of the decimal precision pixel between the pixel (x, y) and the pixel (x, y + 1) is obtained. Then, the moving object detection unit 31 detects the luminance of the pixel with decimal precision and the luminance l of the pixel at the position (x, y) of the frame f-1 (claim 1: the second frame) immediately before the frame f. _The evaluation value m _{up1 / 2} is generated based on the difference from _{f−1; x, y} . Similarly, as shown in Expressions (14) to (16), the intermediate luminance generation unit 32 generates the luminance of the lower half pixel, the luminance of the right half pixel, and the left 1 / left pixel (x, y). The luminance of two pixels is generated, and the _moving object detection unit 31 generates evaluation values m _{down1 / 2} , m _{right1 / 2} , and m _{left1 / 2} .

The _moving object detection unit 31 then sets the evaluation values m _{up1 / 2} , m _{down1 / 2} , m _{right1 / 2} , and m _{left1 / 2} in the direction in which the evaluation values m _{up1 / 2} , m _{down1 / 2} , and m _{left1 / 2} are minimum between the frames f−1 and f. It is determined that the pixel has moved by 1/2 pixel, and the moving speed of the object is calculated. Thus, by using the luminance of the pixel with decimal precision, it is possible to improve the detection accuracy of the moving speed of an object whose moving distance between one frame is less than one pixel.

  In the present embodiment, as shown in equations (13) to (16), the average brightness at the position of 1/2 pixel is obtained by averaging the brightness of adjacent pixels. The position is not limited to a half pixel position, and may be a position between adjacent pixels. That is, by changing the weighting for the luminance of the adjacent pixels, it is possible to generate intermediate luminance at various positions between the adjacent pixels. For example, by adding a value obtained by multiplying the brightness of one pixel of adjacent pixels by 1/3 and a value obtained by multiplying the brightness of the other pixel by 2/3, an intermediate brightness at the position of 1/3 pixel is added. Can be obtained.

  Further, in the present embodiment, the vertical luminance of the pixel (x, y) in the frame f is generated, and evaluation is performed based on the difference between the intermediate luminance and the luminance of the pixel (x, y) in the frame f-1. Although the values are to be obtained, the frame order relationship may be reversed. In other words, it is also possible to generate an intermediate luminance of the pixel (x, y) in the frame f-1, and obtain an evaluation value based on a difference between the intermediate luminance and the luminance of the pixel (x, y) in the frame f. Good.

(2) Detection interval control Next, the detection of the moving speed of the object by detection interval control is demonstrated. FIG. 9 is a diagram showing an outline of an operation for detecting the moving speed of the object by the detection interval control. As described above, the moving object detection unit 31 detects the luminance l _{f; x, y} of the pixel at the position (x, y) (claim 5: first position) of the frame f (claim 5: first frame). And m _up , m _down , m _right , m _left on the basis of the difference between the brightness of the pixel at the position adjacent to the top / bottom / left / right of the position (x, y) of the next frame f + 1, It is possible to detect the moving direction of an object that moves one pixel in between. That is, the moving object detection unit 31 can detect the moving speed on the assumption that the object has moved one pixel between the frame f and the frame f + 1.

Furthermore, the moving object detection unit 31 is a frame f + 2 that is the frame next to the frame f + 1 and is not adjacent to the luminance l _{f; x, y} of the pixel at the position (x, y) of the frame f. 5: second frame) based on the difference from the brightness of the pixel at the position (x, y) adjacent to the top, bottom, left, and right (claim 5: second position), m _up , m _down , m _right , m _{By obtaining left} , it is possible to detect the moving direction of an object that moves by one pixel between two frames. The moving object detection unit 31 can detect the moving speed on the assumption that the object has moved one pixel between the frame f and the frame f + 2.

  Similarly, the moving object detection unit 31 can detect the moving speed of an object that moves by one pixel in three or more frames. In this way, it is possible to detect the moving speed of an object whose moving distance between one frame is less than one pixel by changing the frame interval used when detecting the moving speed without changing the frame rate. It becomes.

  In the present embodiment, the evaluation value is obtained from the difference between the luminance of the pixel (x, y) of the frame f and the luminance of the adjacent pixel of the pixel (x, y) of the frame f + 1. These order relationships may be reversed. That is, the evaluation value may be obtained from the difference between the luminance of the pixel (x, y) of the frame f + 1 and the luminance of the adjacent pixel of the pixel (x, y) of the frame f. The same applies to the frame f + 2 and the frame f + 3.

  Further, even when the decimal accuracy determination shown in FIG. 8 and the equations (13) to (16) is performed, the detection interval control can be performed. For example, in the frame f + 2, it is possible to generate an intermediate luminance in the upper, lower, left, and right directions of the pixel (x, y), and obtain an evaluation value based on a difference between the intermediate luminance and the luminance of the pixel (x, y) in the frame f.

== Depth motion detection ==
Next, object depth motion detection in the image sensor 10 will be described.

(1) Depth Distance Detection Since the optical lens 15 in the moving object detection system 1 has a fixed focus, the distance to the in-focus position is fixed. Therefore, in this embodiment, the distance to the object is detected by determining the degree of defocus.

First, the predicted luminance generation unit 34 of the image sensor 10 changes from a current frame f (Claim 5: first frame) to a future frame f + k (Claim 5: second frame) for a certain pixel (x, y). Assuming that the degree of defocus increases in the meantime, the predicted luminance l _{p; x, y} of the pixel (x, y) in the future frame f + k is generated. Note that the predicted luminance l _{p; x, y} can be generated by using a blur filter such as a low-pass filter. For example, as shown in the following equation (17), it can be obtained by using a filter coefficient d that takes an average of the upper, lower, left and right pixels of the target pixel (x, y).

Then, the depth direction detection unit 33 generates the evaluation value δ _{x, y} by the following equation (18) when the future frame f + k is reached.

If the prediction is increased the degree of correct defocus, brightness l _{f; x,} luminance with respect to _y l _{p; x, y} and luminance l _{f + k; x,} since the increase or decrease direction of the _y is the same [delta] _{x, y} are a positive value, if increased degree of focus off the prediction, luminance l _f; brightness l _{p x,} for _{y; x, y} and luminance l _{f + k; x,} the increase or decrease direction of the _y-reversed Therefore _, δ _{x, y} is a negative value. The depth direction detection unit 33 can determine the value of k according to the moving speed of the object at the pixel (x, y) detected by the moving object detection unit 31. For example, the value of k can be 3 if the moving speed of the object is a speed at which one pixel moves during three frames. Thus, the detection accuracy can be increased by determining the value of k.

FIG. 10 is a diagram showing an outline of an operation for detecting the distance in the depth direction of the object. As shown in FIG. 10 (a), when the object moves away from the optical lens 15, the evaluation value δ _{x, y} gradually increases as the degree of in-focus increases while it is in front of the in-focus position. Is negative. When the object passes the in-focus position, the degree of defocusing gradually increases, and thus the evaluation value δ _{x, y} becomes a positive value. Therefore, when the evaluation value δ _{x, y} changes from negative to positive (at the time of zero crossing), the depth direction detection unit 33 determines that the object is at the in-focus position, and obtains the distance to the object. it can.

As shown in FIG. 10B, even when the object approaches from the optical lens 15, the depth direction detection unit 33 changes the evaluation value δ _{x, y} from negative to positive (when the zero crossing occurs). ), It can be determined that the object is at the in-focus position, and the distance to the object can be obtained.

(2) Depth direction detection The depth direction detection unit 33 detects the depth movement direction, that is, whether the object is moving from the front to the back or the front from the back as viewed from the optical lens 15. be able to. FIG. 11 is a diagram illustrating an outline of an operation for detecting the moving direction of the depth based on the enlargement / reduction of the object. The depth direction detection unit 33 determines whether the object is enlarged or reduced in the two-dimensional direction based on the moving direction of each pixel detected by the moving object detection unit 31. As illustrated in FIG. 11A, when the object is enlarged, the depth direction detection unit 33 determines that the object is approaching the optical lens 15. Further, as illustrated in FIG. 11B, when the object is reduced, the depth direction detection unit 33 determines that the object is moving away from the optical lens 15.

Furthermore, the relationship with the in-focus position can also be detected by using the evaluation value δ _{x, y} described above. FIG. 12 is a diagram illustrating the relationship between the moving direction of the depth and the evaluation value δ _{x, y} . That is, as shown in FIG. 12A, when the object approaches the in-focus position, the evaluation value δ _{x, y} is a negative value. Therefore, when the object is enlarged and δ _{x, y} is a negative value, the depth direction detection unit 33 can determine that an object located behind the in-focus position is approaching. . When the object is reduced and δ _{x, y} is a negative value, the depth direction detection unit 33 can determine that the object in front of the in-focus position is moving away. .

Further, as shown in FIG. 12B, when the object moves away from the in-focus position, the evaluation value δ _{x, y} is a positive value. Therefore, when the object is enlarged and δ _{x, y} is a positive value, the depth direction detection unit 33 can determine that the object in front of the in-focus position is approaching. . In addition, when the object is reduced and δ _{x, y} is a positive value, the depth direction detection unit 33 can determine that the object located behind the in-focus position is moving away. .

== Description of processing ==
Next, a processing flow in the image sensor 10 will be described. FIG. 13 is a flowchart showing the flow of processing in the image sensor 10. First, the frame rate in the image sensor 10 is determined (S1301), and imaging at the determined frame rate is started. Then, the moving object detection unit 31 selects pixels at the same position (x, y) in a plurality of frames (S1302), obtains an absolute difference in luminance between the frames, and generates an evaluation value d (x, y) ( S1303).

In the image sensor 10, one of the following four processes is selected in order to obtain an evaluation value for detecting the moving direction and moving speed of the object. In the first process, the moving object detection unit 31 selects the upper, lower, left, and right pixels of the target pixel (x, y) as described above (S1304), and formulas (3) to (6) or (9) ) To (12), absolute differences in luminance between frames are obtained, and evaluation values m _up , m _down , m _right , and m _left are generated (S1305).

In the second process, the decimal precision determination process described above is performed (S1306). FIG. 14 is a flowchart showing the flow of the decimal precision determination process. First, the intermediate luminance generating unit 32 generates the luminance of the upper, lower, left and right half pixels of the target pixel (x, y) (S1401). Then, the moving object detection unit 31 obtains an absolute difference in luminance between frames as shown in the equations (13) to (16), and evaluates values m _{up1 / 2} , m _{down1 / 2} , m _{right1 / 2} , m _{left1 / 2} is generated (S1402).

In the third process, the above-described detection interval control process is performed (S1307). FIG. 15 is a flowchart showing the flow of the detection interval control process. First, the moving object detection unit 31 selects the frame f and the frame f + 1 as the selection frames, and selects pixels at positions adjacent to the top and bottom, left and right of the position (x, y) of the frame f + 1 (S1501). Then, the moving object detection unit 31 obtains the absolute difference between the luminance of the pixel at the position (x, y) of the frame f and the pixel at the position adjacent to the top and bottom, left and right of the position (x, y) of the frame f + 1, Evaluation values m _up , m _down , m _right , m _left are generated (S1502).

Then, the moving object detection unit 31 selects a frame f + 2 that is a frame subsequent to the frame f + 1 as a selection frame (S1503). Then, the moving object detection unit 31 generates evaluation values m _up , m _down , m _right , m _left using the newly selected frame f + 2 and the frame f (S1501 to S1503). Note that the number of frames ahead of the frame f is determined in advance, and the moving object detection unit 31 repeatedly performs the processing until the processing of the target frame is completed (S1501 to S1504).

  In the fourth process, a process combining the decimal precision process (S1306) and the detection interval control process (S1307) is executed (S1308).

In this way, when the evaluation value d and the evaluation values m _up , m _down , m _right , m _left or the evaluation values m _{up1 / 2} , m _{down1 / 2} , m _{right1 / 2} , m _{left1 / 2} are generated. The moving object detection unit 31 detects in which direction the object is moving up, down, left, or right (S1309), and detects the moving speed of the object (S1310).

Then, in the image sensor 10, in order to detect the distance or the moving direction in the depth direction of the object, one of the following three processes is selected. In the first process, the above-described depth distance detection process is performed (S1311). FIG. 16 is a flowchart showing a flow of processing for detecting a depth distance. First, the predicted luminance generation unit 34 generates predicted luminance l _{p; x, y} using a blur filter (S1601). Then, the depth direction detection unit 33 generates an evaluation value δ _{x, y} based on the predicted luminance l _{p; x, y} and the actual luminance l _{f + k; x, y} (S1602). Then, the distance in the depth direction is detected by determining whether the evaluation value δ _{x, y} is in the in-focus position by positive / negative determination (S1603).

  In the second process, the process of detecting the depth moving direction is performed (S1312). FIG. 17 is a flowchart showing a flow of processing for detecting the moving direction of the depth. First, the depth direction detection unit 33 calculates the enlargement ratio of the object based on the movement direction of each pixel in the vertical and horizontal directions detected by the moving object detection unit 31 (S1701). The depth direction detection unit 33 detects that the object is approaching the optical lens 15 when the object is enlarged, and moves away from the optical lens 15 when the object is reduced. (S1702).

In the third process, a process combining the depth distance detection process (S1313) and the depth direction detection process (S1314) is executed. In this case, the depth direction detection unit 33 detects whether the object is in front of or behind the in-focus position, along with the moving direction of the depth of the object, using the magnification factor of the object and the evaluation values δ _{x, y.} can do.

  The image sensor 10 according to the present embodiment has been described above. As described above, the image sensor 10 can detect the moving direction and moving speed of an object whose moving distance between one frame is less than one pixel by generating intermediate luminance that is the luminance of the pixel with decimal precision. .

  The image sensor 10 uses the luminance value difference d (x, y) of the same pixel (x, y) as an evaluation value when detecting the moving direction and moving speed using the intermediate luminance. Can be detected as not moving.

  Further, the image sensor 10 can detect the moving direction and moving speed of an object whose moving distance between one frame is less than one pixel by controlling the detection interval of the moving direction and moving speed.

  Further, in the image sensor 10, even when the detection interval of the moving direction and the moving speed is controlled, the difference between the luminance values d (x, y) of the same pixel (x, y) is used as the evaluation value, so that the object is detected. It can be detected that it is not moving.

  Further, in the image sensor 10, even when the moving direction and the moving speed are detected using the intermediate luminance, the detection range of the moving direction and the moving speed can be widened by controlling the detection interval.

  Further, the image sensor 10 can detect the moving direction of the object in the depth direction by determining the enlargement / reduction of the object based on the moving direction of each pixel.

Further, the image sensor 10 obtains an evaluation value δ _{x, y} based on the predicted luminance l _{p; x, y} and the actual luminance l _{f + k; x, y} , whereby the object approaches the in-focus position. It is possible to detect whether it is far away from the in-focus position.

  As described above, the image sensor 10 detects the moving direction and moving speed of the object based on the luminance of each pixel, so that complicated processing such as block size control is unnecessary. Further, the image sensor 10 can detect the moving object with high accuracy without changing the frame rate.

  In addition, the said embodiment is for making an understanding of this invention easy, and is not for limiting and interpreting this invention. The present invention can be changed and improved without departing from the gist thereof, and the present invention includes equivalents thereof.

  For example, in the present embodiment, each unit 31 to 35 that performs detection processing in the image sensor 10 is realized by a circuit, but may be realized by a processor executing a program stored in a memory. Good.

It is a figure which shows the structure of the moving body detection system comprised including the image sensor which is one Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of an image sensor. It is a figure which shows the structure of the function implement | achieved by the processing circuit of an image sensor. It is a figure which shows the outline of the operation | movement which detects whether there exists a motion in an up-down-left-right direction. It is a figure which shows the outline of the operation | movement which detects whether there exists a motion in an up-down and left-right direction using two frames. It is a figure which shows the outline of the operation | movement which detects the left-right moving direction. It is a figure which shows the outline of the operation | movement which detects the up / down / left / right moving direction using two frames. It is a figure which shows the outline of the operation | movement which detects the moving speed of an object by decimal precision. It is a figure which shows the outline of the operation | movement which detects the moving speed of an object by detection interval control. It is a figure which shows the outline of the operation | movement which detects the distance in the depth direction of an object. It is a figure which shows the outline of the operation | movement which detects the moving direction of a depth based on expansion / contraction of an object. It is a figure which shows the relationship between the moving direction of depth, and evaluation value (delta) _{x, y} . It is a flowchart which shows the flow of a process in an image sensor. It is a flowchart which shows the flow of a process of decimal precision determination. It is a flowchart which shows the flow of a detection interval control process. It is a flowchart which shows the flow of the process which detects the distance of a depth. It is a flowchart which shows the flow of the process which detects the moving direction of depth.

Explanation of symbols

1 moving object detection system 10 image sensor (moving object detection device)
DESCRIPTION OF SYMBOLS 15 Optical lens 21 Control circuit 22 Photodiode array 23 Memory array 24 Processing circuit 25 Horizontal shift register 26,27 Vertical shift register 31 Moving object detection part 32 Intermediate luminance generation part 33 Depth direction detection part 34 Predictive luminance generation part

Claims (1)

A plurality of photodiodes arranged in an array, and a photodiode array that acquires frame images at predetermined time intervals;
A memory array configured by a plurality of storage elements provided in an array corresponding to the plurality of photodiodes, and storing the frame image acquired by the photodiode array;
A processing circuit that generates and outputs a motion signal indicating a moving direction and a moving speed of a moving object using the frame image acquired by the photodiode array and the frame image stored in the memory array; ,
With
The processing circuit includes:
Based on the difference between the luminance of one pixel of the first frame and the luminance of one pixel of the second frame and the time interval between the first frame and the second frame, the moving direction and moving speed of the moving object are determined. And detecting the moving direction of each pixel based on the difference between the luminance of one pixel at the first position of the first frame and the luminance of the pixel near the first position of the second frame. Furthermore, a moving body detection unit for detecting,
A depth direction detection unit that detects a movement direction in the depth direction of the moving object based on the moving direction of each pixel detected by the moving object detection unit;
A predicted luminance generation unit that generates a predicted luminance that is a luminance when the one pixel at the first position of the first frame is blurred by a blur filter;
With
The depth direction detection unit,
Based on the moving direction of each pixel, it is determined whether the moving object is expanding or contracting in a two-dimensional direction, and when the moving object is contracting, the moving object moves from the front to the back. When it is determined that the moving object is moving and the moving object is expanding, it is determined that the moving object is moving from the back to the front ,
The first difference obtained by subtracting the luminance of the one pixel at the first position of the first frame from the predicted luminance, and the luminance of the one pixel at the first position of the first frame. Generating an evaluation value multiplied by a second difference subtracted from the luminance of one pixel at the first position of the second frame;
When the evaluation value is a negative value, it is determined that the moving object is moving in a direction approaching the in-focus position, and when the evaluation value is a positive value, the moving object is aligned. Determining that it is moving away from the focal position ,
A moving object detecting device characterized by the above.

Images