JP3312562B2 - Dozing state detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a dozing state detecting apparatus for detecting a dozing state of a driver of a vehicle, an operator of a ship, an operator of a plant, or the like, by arithmetically processing face image data.

[0002]

2. Description of the Related Art As a conventional dozing state detecting apparatus based on image processing, there is, for example, an apparatus described in Japanese Patent Laid-Open No. 6-17921. This is achieved by setting a rectangular local area A in the upper right part of the face screen of the subject as shown in FIG. 24A, and setting the binarization threshold based on the average density information of the local area A. Performs binarization processing of the grayscale image data, performs density projection, labeling processing, and the like on the obtained binarized image, detects the position of the eye, and detects the dozing state from the state change of the open / closed eye. I have to.
Thus, the dozing state is detected in a stable state even when the light environment changes.

[0003]

However, in the above-mentioned conventional dozing state detection device, since the local area A is fixed at the upper right portion of the screen, FIG.
If the position of the face is shifted from the local area A as shown in FIG. 4B, an appropriate binarization threshold cannot be set,
There is a problem that the detection of the eye position and the detection of the open / closed eyes become difficult. In particular, in a vehicle, the light environment changes particularly drastically, and since an unspecified number of people drive, the range of changes in the face position is large even in individual differences.

Therefore, the present invention further improves the above-mentioned dozing state detection device, and can cope with a change in light environment.
It is an object of the present invention to provide a dozing state detection device that can always stably detect a dozing state in response to a difference in face position due to individual differences.

[0005]

Therefore, according to the present invention, as shown in FIG. 1, in a dozing state detecting apparatus which processes face image data to detect a dozing state, an image input means for inputting a face image is provided. 1, density detecting means 2 for detecting the density of pixels along a pixel row in the vertical direction of the face, and one pixel for each local increase in density in scanning along the pixel row. A first point extracting means 3 for extracting the extraction points of the above, and a curve group for extracting a curve group extending in the lateral direction of the face by connecting the first extraction points adjacent to each other in the pixel column direction. Extraction means 4, eye position detection means 5 for identifying an eye curve from the curve group and detecting an eye position,
A single pixel is determined for each local increase in density along a vertical pixel row in a predetermined area including the eye, and a second extraction point is extracted, and a density including the second extraction point is determined. A second point extracting means 6 for extracting a third extraction point indicating a boundary point of the local height of the face, and a horizontal direction of the face based on the presence or absence of a second extraction point in the pixel column direction of an adjacent pixel column. Continuity determining means 7 for determining continuity to the eye, eye opening detecting means 8 for detecting the degree of eye opening based on the data of the continuous third extraction points, and eye opening based on the degree of eye opening. And a wakefulness determination means 9 for determining the wakefulness from the change in the open / closed state.

[0006] The density detecting means 1 is used for detecting 1
It is preferable to average the density values of a plurality of continuous pixels including one pixel to obtain the density value of the one pixel. Further, the first point extracting means 3 or the second point extracting means 6 specifies pixels whose differential values before and after the one-way peak of the density on the pixel row exceed a predetermined value, and performs the first extraction. It can be a point or a second extraction point. Then, it is preferable that the curve group extraction means 4 selects only a group in which the number of consecutive first extraction points exceeds a predetermined value.

[0007] The second point extracting means 6 preferably sets the maximum differential value before and after the second extraction point as the third extraction point. Further, the eye opening detecting means 8 can set the maximum interval of the intervals between the third extraction points as the eye opening value.

Further, the second point extraction means 6 detects two second extraction points along the vertical pixel row, and the continuity judgment means 7 determines whether the two second extraction points are continuous conditions. The eye opening detection means 8
Can detect an eye opening value based on the second extraction point data in a predetermined detection order based on the scanning direction. In this case, the eye opening detecting means 8 switches the detection order and switches the detection target of the second extraction point data when the eye opening value does not change by a predetermined value or more for a predetermined time or more. can do.

[0009]

First, the density of the face image input by the image input means 1 is detected by the density detection means 2 along the vertical pixel row. Then, the first point extracting means 3 extracts a first extraction point at which the density locally increases. The curve group extracting means 4 connects the adjacent pixel rows to adjacent ones, and extracts a curve group extending in the lateral direction of the face. Each curve is representative of a dark portion transversely traversing a bright portion of the image and represents eyes, eyebrows, mouth, and the like. The eye position detecting means 5 identifies the curve representing the eye from the above-mentioned curve group by utilizing the feature that the eye is on both sides of the face and the eye and the eyebrow are near the upper and lower positions, for example. Detect the position.

Next, a predetermined area including the position of the eye is set by the second point extracting means 6, and the first area is set in the predetermined area.
The second extraction point is extracted in the same manner as the extraction point, and a third extraction point indicating a boundary point of a local increase in density is extracted before and after the second extraction point. The continuity determining means 7 determines that the third extraction point is close to adjacent pixel columns and is continuous in the lateral direction of the face.
After confirming the above, the opening detecting means 8 detects the opening of the eye based on the data of the third extraction point. Then, the wakefulness determination means 9 determines the wakefulness from the change in the open / closed state of the eyes based on the degree of opening of the eyes. Extracting a group of curves extending in the horizontal direction of the face, and using the characteristics of the general positional relationship between the eyes and other parts of the face such as eyebrows, identifying the curves representing the eyes and detecting the position of the eyes Therefore, even if there is a difference in the physique or posture of the subject, the position of the eye can be easily and reliably detected.

By averaging the density values of a plurality of consecutive pixels including one pixel in detecting the density of the pixels in the pixel row, it is possible to eliminate the small variation in the density value at the time of photographing the image data and to reduce the density value. We can catch global change. In addition, the first extraction point and the second extraction point can be easily extracted by selecting a pixel whose differential value before and after the one-way density peak on the pixel row exceeds a predetermined value. These correspond to the approximate center positions of dark parts such as eyes, eyebrows, and mouth.

Further, if the curve group extracting means 4 selects only a group in which the number of consecutive first extraction points exceeds a predetermined value, extraction of moles and nostrils is avoided. The processing target in the next eye position detecting means 5 is narrowed down, and the detection becomes easy. In addition, by obtaining the point of the maximum differential value before and after the second extraction point as the third extraction point, extraction of wrinkles and irregularities on the face can be avoided.

Further, the second point extraction means 6 allows the detection of two second extraction points, selects the second extraction point data in a predetermined detection order based on the scanning direction, and selects the eye opening degree value. Is detected, the predetermined area can be enlarged, and tracking of the position of the eye becomes easy. Then, when the change of the eye opening value does not exceed a predetermined value for a predetermined time or more, the second
By switching the detection target of the extraction point data, the data can be switched to the correct data even if the selected second extraction point is incorrect.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment in which the present invention is applied to a drowsiness alarm for a driver of an automobile will be described with reference to the drawings. FIG. 2 is a diagram illustrating the configuration of the embodiment.
A TV camera 21 for photographing a driver's face from the front is installed on an instrument panel of a car. T
The input image captured by the V camera 21 is in the horizontal (X) direction 5.
It is composed of 12 pixels and 480 pixels in the vertical (Y) direction.
The A / D converter 22 converts the analog voltage for each pixel into a digital value of 256 levels and stores the digital value in the image memory 23.

An image data operation circuit 24 is connected to the image memory 23. The image data calculation circuit 24 detects the density of the pixel row in the vertical direction of the face based on the input image data stored in the image memory 23, and performs a first extraction described later on the basis of the increase in the density of the pixel row and its change state. Extract points. Then, the first extraction points adjacent to each other in the pixel column direction of the adjacent pixel column are connected to extract a curve group in the horizontal direction of the face.

An eye position detection circuit 25 is connected to the image data calculation circuit 24, and subsequently, an open / closed eye detection circuit 26 is connected. The eye position detection circuit 25 detects an eye position by selecting an eye from a group of curves. The open / closed eye detection circuit 26 detects a vertical density increase and a change state thereof in a predetermined region including the position-detected eye, based on the second and later described,
A third extraction point is extracted, the continuity of the second extraction point in the horizontal direction of the face is determined, and an eye opening value is detected based on the third extraction point of the continuous data.

The open / closed eye detection circuit 26 includes an alertness determination circuit 2
7 is connected, and the awakening degree determination circuit 27
The arousal level is determined based on the detection result of the open / closed eye sent from 6. An alarm device 28 is connected to the arousal level determination circuit 27, and generates an alarm in response to a determination signal of a dozing state in which the arousal level has decreased.

Next, the flow of the control operation in the above configuration will be described with reference to the flowchart of FIG. In step 31, the face of the driver is photographed by the TV camera 21. In step 32, after the pixel density is converted into a digital value by the A / D converter 22, the image memory 23
Stores image data corresponding to one frame image.

In step 33, the image data calculation circuit 2
In step 4, it is checked whether or not the eye existence area has been set. If the area has been set, the process proceeds to step 36; otherwise, the process proceeds to step 34. Here, the eye existence area, which will be described later with reference to FIG. 19, indicates a predetermined area 100 including the eye, and is an area where the processing for detecting the opening degree of the eye is performed. The predetermined area 100 including the eye is also used as an eye tracking area. In step 34, the eye position is detected by the eye position detection circuit 25. For details of eye position detection, see the flowchart of FIG. 4 and FIGS.
0 will be described later.

In step 35, the predetermined area 10 including the eye
The width in the horizontal direction (X direction) and the width in the vertical direction (Y direction) of 0 are determined, the eye existence region (predetermined region including the eye) 100 is set, and the process proceeds to step 36. In step 36, the open / closed eye detection circuit 26 detects the degree of eye opening. For details of the eye opening detection, see the flowchart of FIG.
This will be described later with reference to FIGS.

At step 37, it is checked whether or not the eye is correctly tracked based on the eye opening value detected at step 36. If the eye is correctly tracked, the process proceeds to step 38, and it is determined that the eye is not correctly tracked. If so, the process proceeds to step 311. In step 38, it is determined whether the eye is open or closed based on the eye opening value. Then step 3
In step 9, the eye is tracked, that is, the region where the eye is present is changed.
The details of the eye tracking will be described later with reference to FIG.

In step 310, the arousal level is determined by the arousal level determination circuit 27. If the awake state is reached, the process returns to step 31 and proceeds to the processing of the next frame. If the driver is not in the awake state, the process proceeds to step 312, in which the alarm device 28 is operated to wake the driver to generate a dozing warning, and thereafter, the process returns to step 31. In step 311, the eye opening value and the eye existence area are cleared, and the process returns to step 31 to proceed to the processing of the next frame. Steps 31 to 32 of the above constitute image input means in the present invention,
Steps 38 to 310 constitute arousal level determination means.

FIG. 4 shows details of the processing operation for detecting the position of the eye in step 34. Here, as shown in FIG. 5, the data on the 480-pixel line (pixel row) in the Y-axis direction is subjected to arithmetic processing of the density value of each pixel row, and the pixel (Y Coordinates). That is, a large change in the density value on the pixel row is obtained, and the local peak is set as a first extraction point. When the first extraction point extraction process is performed as described above and one line is completed, the process proceeds to the next line. Therefore, first, in step 41, it is checked whether the extraction of the first extraction point has been completed for all the lines in the predetermined direction (Y-axis direction).

If it is determined in step 41 that the first extraction point extraction for all lines has not been completed, step 4
Proceed to 2. In step 42, an arithmetic average of the density values is performed on one line in a predetermined direction, and an arithmetic average density value in which a high frequency is removed from a density value change curve on the pixel row is obtained. This processing is intended to capture a global change in the density value while eliminating a small variation in the change in the density value at the time of capturing the image data. For example, the pixel column X shown in FIG.
An arithmetic average density value as shown in FIG. 6A is calculated from a, and an arithmetic average density value as shown in FIG. 7A is calculated from the pixel row Xb.

In step 43, the one-line arithmetic mean density value in the predetermined direction calculated in step 42 is differentiated. FIG.
The differential value shown in FIG. 7B is obtained from the arithmetic average density value shown in FIG. 7A, and the differential value shown in FIG. 7A is obtained from the arithmetic average density value shown in FIG. The derivative value is obtained. In the next step 44, the arithmetic mean density in FIG. 6A and FIG. 7A is calculated based on the differential value of FIG. 6B and FIG. A point (Y coordinate value) corresponding to the peak of darkness that is convex downward is extracted. The Y coordinate value at which the differential value is inverted from negative to positive corresponds to the peak of darkness, and is one Y coordinate value representing each of the eyes, eyebrows, mouth, and the like.

The arithmetic mean density value shown in FIG. 6A is a differential value shown in FIG. 6B, and points p1, p2, p3, p4, and p5 at which the differential value reverses from negative to positive. Is extracted. Next, whether or not the peak values q1, q2, q3, q4, q5 of the differential values up to the respective points are equal to or less than a predetermined value, that is, whether or not the hatched portion in FIG. Then, the Y coordinate values A1, A2, and A3 are extracted for points p1, p2, and p5 having a change in density value equal to or less than a predetermined value, and these are set as first extraction points. By determining the peak of the differential value before the point where the differential value reverses from negative to positive, it is possible to avoid extraction of wrinkles and irregularities on the face with a small change in density.

In the arithmetic mean density value shown in FIG.
The points p1, p2, and p3 at which the differential value shown in (b) of FIG. 13 is inverted from negative to positive are extracted, and the peaks q1, q2, and q3 of the preceding differential value are below a predetermined value at any point. There is no first extraction point because it is not in the hatched area. In step 45, when the point extraction processing for extracting the first extraction point is completed for one line, the processing is switched to the processing for the next line, the processing returns to step 41, and the processing is repeated.

If it is determined in step 41 that point extraction for all lines has been completed, the first point as shown in FIG.
Are extracted. That is, on the line Xc in FIG. 8, two first extraction points A1 and A2 are extracted, and on the line Xd, A1, A2, A3,
Four first extraction points of A4 have been extracted. Step 4
Step 2 and steps 43 and 44 constitute the first point extracting means, while 2 constitutes the concentration detecting means of the invention.

If it is determined in step 41 that the extraction of the first extraction points has been completed for all the lines, in step 46, the Y coordinate values A1, A2 of the first extraction points (extraction points) of the adjacent lines are determined. , A3,... If the difference between the Y coordinate values is within a predetermined value (for example, 10 pixels), the data is connected in the X-axis direction to form a group, and the group number of continuous data, the continuous start line number, and the number of continuous data are stored.

The specific processing content of step 46 will be described with reference to FIG. Here, for the sake of simplicity, the description is made on the assumption that there are 11 lines (pixel columns). The first extraction points respectively specified by the lines 1 to 11 are provided with a group number (G), a Y coordinate value (Y value), and a continuous number (N) for each extraction point to form connected data. I have. The 34 extraction points shown in FIG. 9 are grouped into 10 groups. Line 1 has two first extraction points, Y coordinate values 192 and 229, which are extraction points. Since there is no line to the left of the point where the first Y coordinate value is 192, there is no other continuous data at this stage, so the group number of the continuous data is 1. Also, the Y coordinate value 22
For the same reason, there is no continuous data at this stage other than 1 at the point 9 as well, so the group number of the continuous data is set to the following 2.

Next, in line 2 on the right, the Y coordinate value is 1
There are two extraction points, 91 and 224. Since the point of the Y coordinate value 191 of the line 2 is a point within 10 pixels from the Y coordinate value 192 of the line 1 on the left, the group number of the continuous data is set to 1 as an extraction point connected to the extraction point 192. At this time, the number of continuous data is 2. The determination is similarly performed for the point of the Y coordinate value 224 of the line 2, and the group number of the continuous data is 2 and the number of the continuous data is 2.

At the point of the Y coordinate value 360 of the next line 3, 360 and the number of pixels 10 in the Y-axis direction
Since there are no points within the range, the group number of the continuous data is 3, and the number of continuous data is 1. Note that the continuous start line number in step 46 refers to a line number having a point at which the number of continuous data items is determined to be one. Therefore, for example, the value is 3 at a point having a Y coordinate value of 360. In step 46,
In this way, determination of the continuity of points on each line is performed until the end of all lines, and a curve group is obtained. Upon completion, the flow shifts to step 47.

In step 47, first, the number of continuous data in the continuous data group is evaluated. In this embodiment, it is assumed that an effective continuous data group is formed when the number of continuous data is 5 or more. Next, candidate point data of an eye position is formed for the effective continuous data group. The average of the Y coordinate values of the points having the same continuous data group number is calculated and stored as the average value of the continuous points. This value is used as a representative Y coordinate value of the group. Further, a continuous end line is obtained from the continuous start line and the number of continuous data, and an average value of the X coordinate values of the continuous start line and the continuous end line is calculated and stored. This value becomes the representative X coordinate value of the group.

In step 48, based on each continuous group data obtained in step 47, the position of the eye is detected from the length of each continuous group and the positional relationship of the (X, Y) coordinates. In other words, considering the features of the eye,
It can be defined as an upwardly convex bow shape.
When narrowing down the concatenated data based on this definition, the number of consecutive points is 5 due to the condition that the eye is long horizontally.
Continuous data having a small difference between the Y coordinate values of the continuous start point and the continuous end point can be narrowed down as eye candidates, based on the condition that the data is continuous for more than points and has an arcuate shape. When narrowing down continuous data based on this,
Groups G1 to G6 as shown in FIG. 10A are extracted as valid data groups.

Next, considering the positions of the X and Y representative coordinate values of each group calculated in step 47, as shown in FIG. 10B, the ZONE: L, ZONE: C, and ZONE: R can be classified. This is because the left eye and the right eye are largely separated in the X coordinate direction, that is, the face is almost symmetrically divided into left and right, the left eye and the left eyebrow are not largely separated in the X coordinate direction, and the right eye and the right eyebrow are also separated. They do not greatly separate from each other in the X coordinate direction. Also,
The data connected by the shadow under the nose and the data connected to the mouth are located near the center.

As described above, the data is further classified according to the degree of approach in the X coordinate direction, and the position of the eye can be easily detected by narrowing down the data. The linked data included in ZONE: L are the left eye and the left eyebrow, and ZONE: R
Is determined to be the right eye and the right eyebrow, the eye positions are G3 and G4 in FIG. 10A, and their coordinate values can be specified. Step 46 constitutes a curve group extracting means of the invention, and steps 47 and 48 constitute eye position detecting means.

FIG. 11 is a flowchart showing the operation of step 36 shown in FIG.
3 shows details of the processing operation for detecting the degree of opening of the eye in FIG. Here, as shown in FIG. 12A, the second and third extraction point extraction processes are performed on the data on the line (pixel column) in the Y-axis direction, and after the completion of one line, the next adjacent line Move on to processing. Therefore, first, in step 51, it is checked whether or not the extraction processing of the second and third extraction points has been completed for the data on the line in the Y-axis direction in the predetermined area 100 including the eyes.

If it is determined in step 51 that the extraction of the second and third extraction points has not been completed for all the lines, the process proceeds to step 52. In step 52, the arithmetic mean of the density values of one line in the vertical direction is calculated.
This processing is performed to eliminate a small variation in the change in the density value at the time of capturing the image data, and to perform a global change in the density value. FIG. 12B shows FIG.
The processing result of the arithmetic averaging operation of the line data of the pixel column Xa of FIG. Subsequently, in step 53, a differential operation is performed on the arithmetic mean value calculated in step 52. The processing result is shown in FIG.

In the next step 54, first, a second extraction point P is extracted from the arithmetic mean density value calculated in step 52 and the differential value calculated in step 53. The method of extracting the second extraction point is the same as the method of extracting the first extraction point described above. The point P in FIG. 12C where the differential value changes from negative to positive becomes the point where the graph becomes convex to the left in FIG. 12B and becomes the peak point of the density. The peak point of this density is extracted as a second extraction point, and whether or not the change (differential value) of the density value before and after that point is equal to or less than a predetermined value, that is, the hatching in FIG. It is determined whether or not the part is included.

Then, points having density value changes satisfying this condition are extracted as third extraction points Q and R.
That is, the point R is a point having a positive maximum value of the differential value, and the point Q is a point having a negative maximum value. The third extraction points R and Q correspond to the dark portions along the vertical pixel row including the second extraction points in FIG.
Represents a boundary point with a bright portion on the screen. Steps 52 to 54 constitute the second point extracting means of the present invention.

In step 55, it is checked whether or not the second extraction point exists. If so, the process proceeds to step 56, and if not, the process proceeds to step 511. Step 5
6, the distance H in the Y coordinate direction between the third extraction points Q and R, which is the maximum positive / negative value of the differential values before and after the second extraction point P,
Find a. That is, the length of the dark part along the pixel row including the second extraction point is obtained. Next, when the third extraction points are continuously present in the vertical pixel line as shown in FIGS. 12 and 13, the intervals between the third extraction points Q and R are compared for each line, and the maximum is determined. Update the value Hb.

In step 57, Xa in FIG.
As shown by Xb in FIG. 13A, when the second extraction points are continuously present, the number of consecutive second extraction points in the X-axis direction is counted up. In step 58, it is determined whether or not the number of consecutive second extraction points has exceeded a predetermined value. If it has, the process proceeds to step 59, and if not, the process proceeds to step 510. In step 510, the process is switched to the next line and the process returns to step 51, and the above process is repeated for the switched line. Thus, the same processing is continued until the number of consecutive second extraction points reaches a predetermined value.

During this process, if the continuity of the extraction points is interrupted and it is determined in the step 55 that the second extraction point does not exist, the flow shifts to the step 511. In step 511, the third
, The maximum value of the interval between the extraction points Q and R and the counter are cleared, and the process proceeds to step 510. In step 510, switching to the next line is performed, and the process returns to step 51. As shown in FIG. 15, the processing when the continuity of the extraction points is interrupted is as follows.
As shown in FIG. 6 (a), when the data enters the region 100 including the eye and becomes the connected data, the maximum value H1max of the interval between the third extraction points Q and R of the frame is not regarded as the eye opening. In this way, the processing in step 58 is to determine the length of L1 in FIG.

If the continuity of the second extraction point exceeds the predetermined value in step 58, the process proceeds to step 59. In step 59, the maximum value of the interval between the third extraction points Q and R is updated and stored. In step 59, the memory of the maximum value in the continuous data is updated, and the update is performed between other continuous data exceeding a predetermined value.

That is, when a bright spot enters the center of the eye, as shown in FIG.
1 and P2, and the degree of the density change becomes small. Therefore, the peak of the differential value on one side of each of the peak points P1 and P2, that is, R1 at the peak P1, and Q2 at the peak P2 do not enter the hatched portions in FIG. Sometimes. As described above, since the condition for the establishment of the second extraction point is not satisfied, the continuity of the second extraction point may be interrupted, and the eye data is changed to L2 and L3 as illustrated in FIG. May be split. Therefore, at this time, by setting the predetermined number of consecutive numbers in step 58 to a little less than half the length of the eye, both L2 and L3 are extracted as continuous data exceeding the predetermined value, and the third extraction in step 59 is performed. Updating of the maximum value of the interval between the points Q and R is performed by H2max and H3max.

In FIG. 16, section D indicates a place where there is no extraction point satisfying the extraction conditions of the second and third extraction points in step 54, and section L indicates a place where an extraction point exists. On the other hand, when it is determined in step 51 that the above-described processing has been completed, the process proceeds to step 512, in which the third extraction indicated by H3max in FIG. 16A when the eyes are opened and the H2max in FIG. The maximum value of the interval between the points Q and R is output as the eye opening value.

Next, a description will be given of a case where hair is applied to the eyes as shown in FIG. Also in this case, the portion D5 where the hair is applied in FIG. 18A showing the state when the eye is open and the portion D6 where the hair is applied in FIG.
Since the section does not satisfy the extraction condition of the second extraction point in No. 4 and does not have the second extraction point, the eye opening can be detected in the same manner as in the case where the hair is not applied. Here, steps 57 and 58 constitute continuity determination means of the present invention, and steps 56, 59, 511 and 512 constitute eye opening degree detection means.

Next, step 3 in the flowchart of FIG.
The methods of tracking the eyes and recovering from the tracking error in 7, 39, and 311 will be described with reference to FIGS. FIG.
Immediately after the system is started based on the flowchart in (1), the region where the eye is present (the predetermined region including the eye) is not set in the first frame. Therefore, it is determined that there is no eye existing area in the check in step 33, and after the position of the eye is detected in step 34, step 35
Sets the eye existence region 100. At this time, the center coordinates of the eye and the center coordinates of the eye existence region 100 match as shown in FIG.

After a series of processing for the first frame is completed, the processing shifts to the processing for the second frame.
When the process proceeds to, it is determined that there is an eye existing region because the eye existing region 100 has already been set. Accordingly, the process proceeds to step 36 to detect the degree of eye opening, and then proceeds to step 37. At this time, when the eye is correctly captured, the state becomes as shown in FIG. 19B, for example. Here, the eye existence region 100 is at the position set in the first frame, whereas the eye position is the eye position in the image data of the second frame captured in the second frame. Due to movement or the like, the center point of the eye is at a position shifted from the center of the eye existence region 100.
However, as long as the eye does not touch the boundary of the eye existence region 100, the eye opening detection described above can be performed.

Next, in step 39, the reference point of the eye existence area 100 is changed to the center coordinates of the eye in FIG. That is, step 36
As shown in FIG. 16A, when the eye opening is detected and the interval Hmax between the third extraction points Q and R is a large value H3max at which the eye is determined to be open, as shown in FIG. That H3ma
the X coordinate of the point where the pixel row that became x is located, and H3
The Y coordinate of the median value of max is set as a new reference point of the eye existence region 100. Further, as shown in FIG.
If max is a small value H2max determined to be closed, the eye existence area 100 is set using the X coordinate of the median of the number of continuous data and the Y coordinate of the median of H2max as reference points. Thereby, it is possible to correspond to the movement of the driver's face.

FIGS. 19C and 19D show the positional relationship between the eye position and the eye existence area 100 in the face image data captured in the third and fourth frames, respectively. . Since the change of the eye existence area 100 is performed in the processing of the previous frame, the deviation between the center coordinates of the eye and the center coordinates of the eye existence area 100 is small.

The confirmation as to whether or not the eye tracking has been correctly performed in step 37 is performed by using the eye opening value. In other words, once the driver who is the subject is specified, the eye opening value only changes in the range from when the eye is open to when the eye is closed, so when a value outside this range is output, It is determined that an eye tracking error has occurred. In this case, the process shifts to step 311 to clear the output value of the eye opening and the region where the eye exists, and shifts to the next frame processing. Then, in step 33, the process starts again from the entire face.

Next, the reference value for judging the open / closed eye in step 38 will be described. As described above, when the driver is specified, the output value of the degree of opening of the eye changes between the open state and the closed state. Will be within that range. Here, the threshold value is set as the median value between the open and closed eyes because the dozing person is not in the deep sleep state and the eyes may not completely close in some cases.

Next, the determination of the arousal level in step 310 of the flowchart of FIG. 3 will be described. FIG.
Is an open / closed eye pattern indicating the determination result output in step 38. The arousal level is determined by checking whether or not the integrated value of the closed eyes output in the arousal level determination section (for example, about one minute) on this open / closed eye pattern is equal to or greater than a predetermined value (for example, 5). In the example of FIG. 20, since the integrated value of the closed eyes is 7, it is determined to be dozing. The dozing state can be detected based on the degree of arousal obtained in this manner, and the alarm device can be operated accurately, the dozing state can be eliminated, and a dozing accident can be prevented.

Since the present embodiment is configured as described above, it is possible to always stably detect the dozing state in response to the difference in the face position due to the individual difference. , 39, and 311 can omit the eye position detection from the entire face by the steps 34 and 35 in the processing of a new image frame, and speed up the eye opening detection. It has the advantage that it can be done.

Next, a second embodiment will be described. In this embodiment, the tracking performance of the eye is further improved. For this reason, a predetermined region including the eye (the region where the eye exists) is set relatively large. In the first embodiment, since the height (Y-axis direction) of the predetermined area 100 including the eye is set to be small, the second extraction point is narrowed to one. By the way, in order to improve the tracking performance of the eyes, it is better to set a predetermined area including the eyes wide. However, when the predetermined area including the eye is enlarged, as shown in FIG. 21, both the eye and the eyebrow enter the predetermined area 200, and one line (pixel row) is formed.
May be detected as two second extraction points. In this embodiment, the processing is performed according to the flowcharts shown in FIGS.

By the processing of steps 61 to 64, the second extraction point P, which is the darkest position of the dark part along the line (pixel row), and the third extraction point Q, which is the boundary point between the dark part and the light part, R is extracted. Steps 61 to 64 are the same as steps 51 to 54 in the flowchart of FIG. At step 65, it is checked whether the number of the second extraction points on the line is one. If the number of the second extraction points is not one on the line, the process proceeds to step 612. If there is one, the process proceeds from step 66 to steps 67 and 68, and the same processing is repeated until the number of consecutive second extraction points exceeds a predetermined value. Steps 66 to 68 are the same as steps 56 to 58 in the flowchart of FIG.

If it is determined in step 68 that the number of continuations has exceeded a predetermined value, the flow advances to step 69. In step 69, it is checked whether or not a two-point extraction flag indicating that there are two second extraction points, that is, two detection targets, is set. If so, the process proceeds to step 611. In step 610, the maximum value of the interval between the third extraction points Q and R for one point, that is, when the number of detection targets is one is updated and stored, and the process proceeds to step 611. In step 611, the process is switched to the next line, the process returns to step 61, and the process from the beginning is repeated.

At step 612, it is checked whether or not the number of the second extraction points is two.
Proceed to 3 and if it is 3 or more, proceed to step 619. That is, as shown in FIG. 21, the detection target A appears during the extraction of the second extraction point for one detection target B of one point, and two points, that is, two detected extraction points are detected. If it is determined that the number has reached one, the process proceeds to step 613.
In step 613, the counter of the maximum value of the interval between the third extraction points Q and R for one point (detection target B), which has been updated, and the counter of the continuous number thereof are cleared. Then, in step 614, instead of the three detection targets,
The maximum value of the interval between the extraction points Q and R is updated.

In the next step 615, the number of continuations of the two second extraction points in the X-axis direction is counted.
At 16, it is checked whether the number of consecutive two second extraction points exceeds a predetermined value. If the number of continuations exceeds a predetermined value, the flow advances to step 617 to set a two-point extraction flag. If the maximum value of the interval between the third extraction points Q and R when one point, that is, one detection target is stored, the value is simultaneously cleared at this time.
This process is performed when two extraction points are detected and they are consecutive, because the detection target is an eye and an eyebrow at a high rate.

In step 618, the memory is continued while updating the maximum value of the interval between the third extraction points Q and R of the second detection target. Here, since the density detection is scanned from top to bottom on the Y-axis, the order is from eyebrows to eyes, and the second corresponds to eyes. Thereafter, the process proceeds, and the second extraction point of the detection target B is extracted again. Even if the continuity is detected, since the two-point flag is set in the check in step 69, one is extracted in step 610. The maximum value of the interval between the third extraction points Q and R of the eye (detection target B) is never stored.

On the other hand, in step 619, if there is a maximum value of the interval between the extraction points Q and R updated so far, it is cleared, the counter is also cleared, and the routine proceeds to step 611. This is because if there are three or more second extraction points, it is determined that the detection target is neither the eye nor the eyebrow. Step 6
In step 20, when the processing of all the lines in the vertical direction is completed, the maximum value of the interval between the third extraction points Q and R is output as the eye opening value.

If the continuity of one second extraction point is confirmed, in step 610, the third extraction point Q and R intervals of one point are updated and stored. Also, two second
If the continuity of the extraction points is confirmed, step 618
Then, the second third extraction points Q and R intervals are updated and stored. Therefore, when the processing for all the lines is completed, one of the values is output as the eye opening value in step 620.

As described above, in the present embodiment, when two second extraction points are detected, one of them is selected to determine the third extraction point Q, R interval. The predetermined area can be enlarged, and the eye tracking performance is further improved.

In the above description, it is assumed that the detection target entering the predetermined area 200 including the eye is the eye and the eyebrow having the highest probability, and the second extraction point in the step 618 is the second extraction point. The maximum value of the interval between Q and R is stored in memory.
In rare cases, when the eye and the lower frame of the glasses enter, the positional relationship is reversed. In this case, even if an erroneous eyeglass frame is selected as a detection target, if the output value is monitored in a time series, there is no change in the value unlike the eye. A determination can be made. As a result, the first detection target in step 618 is switched. Such processing can be applied to the case where the eyebrows are erroneously detected. Also, in the above description, the case where continuous data appears for three or more second extraction points is excluded from the selection targets. In this case, what is the detection target (eyebrows, eyes,
If the frame of the glasses can be specified, the processing can be expanded similarly.

[0066]

As described above, according to the present invention, in the dozing state detecting apparatus based on face image data processing, a first extraction point is extracted for each local increase in density in a vertical pixel row of a face. A curve group extending in the horizontal direction of the face is continuously extracted from adjacent first extraction points of adjacent pixel columns, an eye curve is specified from the curve group, and an eye position is detected. Similarly, a second extraction point is extracted along a vertical pixel row in a predetermined area including the extracted eye, and a third point indicating a boundary point of a local increase in density including the second extraction point
Since the extraction point is extracted, the opening degree of the eye is detected based on the data of the third extraction point that is continuous in the horizontal direction, and the degree of arousal is determined from the state change of the opening degree of the eye. Even if there is a difference in the driver's physique or driving posture, the position of the eyes can be easily and reliably detected.

In detecting the density of the pixels in the pixel row, the average value of the density values of a plurality of pixels including one pixel is averaged to obtain a small variation in the density value at the time of capturing the image data. Thus, a global change in the density value can be grasped. In addition, the first extraction point and the second extraction point are selected by selecting a pixel whose differential value before and after the one-way peak of the density on the pixel row exceeds a predetermined value, thereby darkening the eyes, eyebrows, and mouth. It can be easily extracted corresponding to the approximate center position of the part. Furthermore, in extracting a curve group, by selecting only a group in which the number of consecutive first extraction points exceeds a predetermined value, extraction of moles, nostrils, and the like can be avoided, and the position of the eye can be avoided. Detection becomes easy. In addition, by obtaining the point of the maximum differential value before and after the second extraction point as the third extraction point, extraction of wrinkles and irregularities on the face can be avoided.

Further, when two second extraction points are detected, one of the extraction point data in a predetermined detection order is selected and the eye opening value is detected, so that the predetermined area can be enlarged. Thus, an effect of facilitating eye position tracking can be obtained. In this case, if there is no change in the eye opening value equal to or more than the predetermined value for the predetermined time or more, the detection target is switched to use the other extraction point data, so that even if the set detection order is incorrect, the detection order is correct. Switch to data.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of the present invention.

FIG. 2 is a block diagram illustrating a configuration of a first exemplary embodiment of the present invention.

FIG. 3 is a flowchart showing a processing operation of dozing state detection.

FIG. 4 is a flowchart showing a processing operation of eye position detection.

FIG. 5 is a diagram illustrating a pixel array for detecting the position of an eye.

FIG. 6 is a diagram illustrating a procedure for extracting a first extraction point.

FIG. 7 is a diagram illustrating an example in which a first extraction point does not exist.

FIG. 8 is a diagram illustrating extraction of a curve group.

FIG. 9 is a diagram illustrating a first extraction point grouping process.

FIG. 10 is a diagram for explaining a point of detecting an eye position from a curve group.

FIG. 11 is a flowchart showing a processing operation of eye opening detection.

FIG. 12 is a diagram illustrating a procedure for extracting second and third extraction points.

FIG. 13 is a diagram for explaining detection of an opening degree from an interval between third extraction points.

FIG. 14 is a diagram illustrating opening degree detection when there is a bright spot at the center of the eye.

FIG. 15 is a diagram showing a face image when wearing glasses.

FIG. 16 is a diagram illustrating processing when a frame of glasses is extracted.

FIG. 17 is a diagram showing a face image in a case where hair is hung on eyes.

FIG. 18 is a diagram illustrating processing when hair is on the eyes.

FIG. 19 is a diagram illustrating an eye tracking state.

FIG. 20 is a diagram for explaining a procedure for determining a degree of arousal.

FIG. 21 is a diagram illustrating a second embodiment of the present invention.

FIG. 22 is a flowchart showing a processing operation of dozing state detection.

FIG. 23 is a flowchart showing a processing operation of dozing state detection.

FIG. 24 is a diagram showing a conventional example.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 image input means 2 density detection means 3 first point extraction means 4 curve group extraction means 5 eye position detection means 6 second point extraction means 7 continuity determination means 8 eye opening detection means 9 arousal level determination Means 21 TV camera 22 A / D converter 23 Image memory 24 Image data calculation circuit 25 Eye position detection circuit 26 Open / closed eye detection circuit 27 Arousal level determination circuit 28 Alarm device A1, A2, A3, A4 First extraction point P Second extraction point Q, R Third extraction point

──────────────────────────────────────────────────続 き Continuation of front page (56) References JP-A-4-25706 (JP, A) JP-A-8-300978 (JP, A) JP-A-9-270010 (JP, A) JP-A-10-1998 44824 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 1/00 340 G06T 1/00 330 G06T 7/60 250

Claims (9)

(57) [Claims] 1. A dozing state detection device for processing a face image data to detect a dozing state, an image input means for inputting a face image, and detecting a density of pixels along a vertical pixel row of the face. Density detection means, and a first extraction point for extracting a first extraction point by defining one pixel for each local increase in density in scanning along the pixel row
Point extraction means, and a curve group extraction means for connecting the first extraction points adjacent to each other in the pixel column direction to extract a curve group extending in the horizontal direction of the face; Eye position detection means for detecting the position of the eye by identifying the curve of the eye, and defining one pixel for each local increase in density along a vertical pixel row in a predetermined area including the eye A second point extracting means for extracting a second extraction point and extracting a third extraction point indicating a boundary point of a local increase in density including the second extraction point; Continuity determining means for determining the continuity of the face in the horizontal direction based on the presence or absence of a second extraction point close to the pixel column direction, and detecting the degree of eye opening based on the data of the continuous third extraction point An eye opening detection unit, and a degree of arousal based on a change in an eye open / closed state based on the eye opening. Doze state detecting apparatus characterized by having a constant to awareness determination means. 2. The method according to claim 1, wherein the density detecting unit is configured to output 1
2. The dozing state detection device according to claim 1, wherein the density values of a plurality of continuous pixels including one pixel are averaged to obtain the density value of the one pixel. 3. The method according to claim 1, wherein the first point extracting unit specifies a pixel whose differential value before and after the one-way peak of the density on the pixel row exceeds a predetermined value and determines the pixel as the first extraction point. The dozing state detection device according to claim 1 or 2, wherein: 4. The dozing state detection according to claim 1, wherein the curve group extracting means selects only a group in which the number of consecutive first extraction points exceeds a predetermined value. apparatus. 5. The method according to claim 5, wherein the second point extracting unit specifies a pixel whose differential value before and after a one-way peak of the density on the pixel row exceeds a predetermined value and determines the pixel as the second extraction point. 5. The dozing state detection device according to claim 1, 2, 3, or 4. 6. The method according to claim 1, wherein the second point extracting means sets a maximum differential value before and after a second extraction point as the third extraction point. The dozing state detection device according to the above. 7. The eye opening detecting means according to claim 1, wherein a maximum interval between the third extraction points is an eye opening value. Or the dozing state detection device according to 6. 8. The second point extraction means detects two second extraction points along a vertical pixel row,
When the two second extraction points satisfy the continuity condition in the continuity determination means, the eye opening degree detection means sets the eye based on the second extraction point data in a predetermined detection order based on the scanning direction. 2. The dozing state detection device according to claim 1, wherein an opening degree value of the drowsiness is detected. 9. The eye opening detection means, when the eye opening value does not change by a predetermined value or more for a predetermined time or more, replaces the detection order and determines a detection target of the second extraction point data. 9. The dozing state detection device according to claim 8, wherein the switching is performed.