KR20200012355A - Online lecture monitoring method using constrained local model and Gabor wavelets-based face verification process - Google Patents

Abstract

Disclosed is a face authentication method using a constrained local model (CLM) and Gabor wavelets. The method comprises: a process of extracting main components of a face by CLM-based face feature point extraction; and a process of generating Gabor feature vectors by Gabor wavelet conversion on extracted feature points, and then using a nearest neighbor distance ratio and a correlation between the Gabor feature vectors to authenticate an individual. If the present invention is applied to application areas such as an online lecture monitoring system, online evaluation, and mandatory repair education, satisfactory performance with a practical level is expected to be provided.

Description

Online lecture monitoring method using constrained local model and Gabor wavelets-based face verification process}

The present invention relates to an online lecture monitoring method having a face authentication process using CLM and Gabor wavelets.

Today's online courses are emerging as a useful means of providing a variety of knowledge and learning experiences in remote locations through a network-based online learning environment. Online lectures allow learners to overcome the physical constraints of time and space, allowing them to learn at the time and place they want.

Therefore, students can voluntarily learn according to their own learning level, purpose, and individual conditions, and can reduce social and personal education costs at the macro level. And there is a social net function to increase the utility of cost through the efficient distribution of limited educational resources. In particular, recent online education has been evaluated as a high value-added industry that can create social value by reducing educational inequality and distributing educational resources. The birth of this trend is the Massive Open Online Courses (MOOC) platform represented by Cousera and edX.

Korea is also participating in this trend that is happening globally, and has made efforts to open Korean-style MOOC (K-MOOC) under the leadership of governments and universities, and to upload online lecture contents to each expert.

However, in spite of the above advantages, the current online lecture system needs to compensate for the deficiencies in various aspects. This includes the lack of a way to determine whether or not participants have taken the online course content in good faith. For example, in the case of inverted learning using online lectures, if the attendees attend the in-class without viewing the online lecture contents provided for pre-class use, There is a problem that hinders and consequently lowers the learning efficiency of others.

Therefore, it is necessary to devise a method for effectively monitoring whether or not the online lecture is normally viewed without disturbing the learners. In order to meet this need, the inventor of the present invention has proposed an online lecture monitoring system using gaze tracking technology.

However, even if this system is introduced, there is a limit to blocking the cherry picker who is recognized for attending the online lecture by increasing the rate of application by false viewing using pictures, dolls, sculptures, etc., rather than the actual viewing. We also proposed an auditory deception method that can block fraudulent users by detecting fluctuations in eye movement at the end of lecture monitoring systems.

However, the existing online lecture monitoring system includes fine false tremors caused by changes in lighting or camera noise when false viewing using photographs, figurines, sculptures, etc., and therefore, needs to be effectively suppressed to improve viewing deception detection performance.

The proposed online lecture monitoring system performs Kalman filtering on the coordinates of the left and right eye center points between adjacent frames obtained using the gradient-based eye detection method proposed by the researchers. Find the variation in the movement. It is intended to effectively use the operating characteristics of the Kalman filter to suppress the changes caused by the abnormal variables.

The gradient-based eye detection method detects the face area from the input image acquired through the PC cam, and then expands or converges in two concentric directions originating from the center and the outside of the eye search area, and bidirectionally by a limited number of pixels. The pupil area can be labeled to improve eye detection accuracy and computation speed, and Kalman filtering can be performed on the corrected cumulative mean to determine whether the eye is open or closed.

The proposed online lecture monitoring system is expected to improve the reliability and learning efficiency of the online lecture system by monitoring whether the online lectures are faithfully watched and preventing false attendance by the viewers.

An object of the present invention is to provide an online lecture monitoring method that can monitor whether or not the attendees sincerely watch using a low-cost PC cam to efficiently and reliably manage online attendance.

Another object of the present invention is to provide an online lecture monitoring method that provides feedback to learners in real time by displaying whether or not online attendance is displayed on the screen during the playback of lecture contents.

Another object of the present invention is to provide an online lecture monitoring method that can be easily combined with the existing online lecture system to improve the reliability and learning efficiency of the online lecture system by monitoring the integrity of the online lecture and preventing false attendance. To provide.

According to an aspect of the present invention, there is provided a computer program comprising: operating software on a computer and selecting and playing lecture contents through file opening; Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames; Determining whether the learner stares at the screen; If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner; When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture content screen; The inner cumulative mean is calculated by dividing the inner cumulative mean by the number of gradient vectors, and the Kalman filtering is performed by inputting the corrected inner cumulative mean values of the previous frame and the current frame to perform the corrected inner cumulative mean value of the current frame. Determining whether to open or close the eyes by comparing the updated internal cumulative average and the eye opening and closing threshold value; And in real time, the number of full screens played so far, the number of gaze screens checked by eye tracking, the screen gaze ratio that is the ratio of the number of screens and the number of screens, the viewing dazzle rate and the number of screens that are the ratio of dazzle screens and the total screens. An online lecture monitoring method is provided, comprising the step of displaying a final learning rate, which is a ratio of a subtracted number of screens and a total number of screens.

Preferably, the face authentication step includes: generating a Gabor kernel having a plurality of direction and frequency components; A sub-step of extracting major facial feature points from a gallery face image in advance; Extracting and storing a gallery feature vector using a Gabor kernel at each facial feature point in advance offline; Measuring a similarity between the Gabor feature vector of the online extracted probe face image and a previously stored gallery feature vector; Determining that the gallery face image having the highest similarity with the probe face image includes a face of the same person as the probe face, and determines a misclassification otherwise; A substep of obtaining a nearest neighbor distance ratio only when the gallery face having the most similarity to the probe face is classified as the same face as the probe face; And a substep of determining that face authentication succeeds if the nearest distance ratio is less than an authentication threshold, and otherwise deems unauthorized.

Preferably, the main facial feature points are extracted 68 by using Constrained Local Models (CLM), the Gabor kernel is generated 40 using 8 direction components and 4 frequency components, the authentication threshold value can be 0.95 have.

Preferably, the eye opening / closing threshold value is obtained by accumulating the updated internal cumulative average value of the previous frames and the current frame by a predetermined number of frames and dividing the interval by the number of frames. Only the 70 frames determined as the opened eye state excluding the frame may be counted, and the eye open / close threshold value may be a value obtained by multiplying the section average value by a proportional coefficient of 0.8.

According to another aspect of the present invention, there is provided a computer program comprising the steps of: running software on a computer and selecting and playing lecture content through file opening; Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames; Determining whether the learner stares at the screen; If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner; When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture contents screen; And in real time, the number of full screens played so far, the number of gaze screens checked by eye tracking, the screen gaze ratio that is the ratio of the number of screens and the number of screens, the viewing dazzle rate and the number of screens that are the ratio of dazzle screens and the total screens. And displaying a final learning rate, which is a ratio of the total number of screens minus the number of deceptive screens, wherein detecting the viewing deception comprises: a difference between vertical and horizontal coordinates of left and right eye center points between a previous frame and a current frame. Substeps of substituting the eye center point of the current frame as a false tremor if the predetermined threshold value is less than the predetermined reference value; After performing Kalman filtering on the horizontal and vertical coordinates of the eye center point from which the false tremor is removed between successive frames, the difference between the vertical and horizontal coordinates of the left and right eye center points between the current frame and the previous frame is the predetermined reference value. A sub-step of determining that the eye's center point is not changed to increase the number of eye stops; And determining that the number of eye stoppages is greater than or equal to a predetermined number of stops during a predetermined frame period, and determining that the viewing is deceptive. Otherwise, the subblock is configured to continuously monitor whether or not viewing is deceptive while initializing the number of eye stops every predetermined frame period. An online lecture monitoring method is provided comprising the steps.

Preferably, the detecting of the viewing deception is to warn the outside that it is determined that the viewing deception state by displaying a red warning light when the attendees normally watch the video, but the red warning light does not come in. Substep; If it is determined that the viewing deception state in more than 85% of the entire frame, the final determination to deliberate viewing deception intention, and may further include a sub-step of processing the final learning rate to 0%.

Preferably, the predetermined reference value is 3 pixels, the predetermined frame section is 30 frames, and the predetermined number of stops may be 29 times.

According to another aspect of the present invention, there is provided a computer program comprising the steps of: running software on a computer and selecting and playing lecture content through file opening; Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames; Determining whether the learner stares at the screen; If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner; When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture contents screen; And in real time, the number of full screens played so far, the number of gaze screens checked by eye tracking, the screen gaze ratio that is the ratio of the number of screens and the number of screens, the viewing dazzle rate and the number of screens that are the ratio of dazzle screens and the total screens. And displaying a final learning rate, which is a ratio of a subtracted number of screens to a total number of screens, and determining whether to stare at the screen comprises: detecting a face region in an input image; Specifying a left and right eye search region using the detected face region; Detecting a pupil candidate region from each of the designated eye searching regions; Labeling the pupil area of the detected pupil candidate area as only the limited number of pixels in both directions while converging in two concentric directions originating from each of the center and the outside of the eye searching area; A substep of determining an edge by obtaining an edge map using horizontal and vertical differential masks; And a sub-step of performing an inner cumulative calculation in the labeled pupil area to calculate an inner cumulative value and detecting the inner cumulative value, and in the labeling of the pupil area, concentric diffusion from a center of the eye search area. After labeling the pupil area within a predetermined pixel area limit while proceeding in a direction, if the labeled pupil area is in contact with the outer periphery of the eye search area, the labeled pupil area proceeds in the concentric contraction direction from the outer part. In the step of removing the outer adjacent region from the labeled pupil region within the ratio limit and designating it as the final labeled pupil region, and performing the dot accumulation operation, the normal displacement vector is taken only in the final labeled pupil region to search for each eye. All grady in the zone After calculating the dot product with the vector, the cumulative operation is performed to obtain the dot product cumulative value, multiply the inverse brightness weight corresponding to each of the dot product cumulative values to update the dot product cumulative value, and then determine the position of the maximum dot product cumulative value. An online lecture monitoring method is provided that detects at the center of the eye.

Preferably, when the pupil of the participant is detected and when it is not detected, it may be determined whether the eyes are opened or closed by displaying different colors on the eye region of the input image, and when the face is not detected from the input image. If the video is stopped and a face is detected, the video may be played again from that point on.

Preferably, the detecting of the face region comprises: a sub-step of overlapping an initial face template with a starting point of an upper left end of an input image and obtaining a cross correlation with an overlapped portion; A sub-step of detecting a position having the highest cross-correlation as a face region while moving by one pixel from the starting point; And if the cross-correlation is equal to or greater than a predetermined value, the template matching may be continuously performed. Otherwise, the template matching may be performed by re-detecting the face region and updating the new face template.

Preferably, if the face region is not detected, the template matching may be performed by lowering the cross correlation, and if the face region is not detected even in the lower cross correlation, it may be determined that there is no face region in the image.

According to the above configuration, the eye detection accuracy can be improved by labeling the pupil region in both directions while converging in two concentric circles originating from the center and the outer edge of the eye search region, respectively.

In addition, Kalman filtering is performed on the corrected cumulative inner cumulative average to suppress performance degradation due to unspecified external conditions, thereby improving performance of the conventional gradient vector field-based eye open / closed judgment method.

Also, by applying the deception function of viewing, it analyzes the loophole of the gaze determination algorithm and prevents the false attendance of online lectures from being deceived as if the learner gazes at the contents of the lecture using human face photos, figurines, and sculptures. can do.

In addition, by using a camera to monitor the attendee's sincerity to watch the online attendance can be managed and display the on-screen during the course content playback can provide feedback to the learner in real time.

를 대상으로 서로 다른 값으로 임계처리해 구한 이진 영상이다.
도 4는 동공 영역 레이블링을 적용한 예를 보여준다.
도 5는 동공 중심 영역의 국부 블록 마스크를 나타낸다.
도 6은 그레이디언트 벡터의 개수로 보정한 내적 누적 평균을 이용해 눈 개폐 판단 여부를 나타낸 그래프이다.
도 7은 본 발명의 시스템의 시청 현혹 검출 실험 상황을 예시한 것이다.
도 8은 시청 현혹 검출 시의 최종학습률을 표시하는 예를 보여준다.
도 9는 본 발명의 방법을 적용한 눈 검출 결과를 예시한 것이다.
도 10은 본 발명의 눈 개폐 판단 방법을 이용한 판단 결과 영상을 나타낸 것이다.
도 11은 얼굴 인증 동작 과정을 나타낸 순서도이다.
도 12는 CLM을 이용한 주요 얼굴 특징점의 추출 결과를 나타낸 것이다.
도 13은 FEI Face Database의 1인당 14장씩 촬영한 얼굴 영상 세트를 예시한 것이다. Figure 1 (a) shows the user interface of the administrator mode of the online lecture monitoring system, Figure 1 (b) shows the normal operating state of the online lecture monitoring system.
2 is a flowchart illustrating a method for monitoring online lectures according to the present invention.
3 is a normalized image

It is a binary image obtained by thresholding with different values.
4 shows an example in which pupil area labeling is applied.
5 shows a local block mask of the pupil center region.
6 is a graph showing whether the eye is opened or closed by using an internal cumulative average corrected by the number of gradient vectors.
Figure 7 illustrates the situation of deception detection experiment of the system of the present invention.
8 shows an example of displaying a final learning rate at the time of detecting deception of watching.
9 illustrates an eye detection result to which the method of the present invention is applied.
10 is a view showing a determination result image using the eye opening and closing determination method of the present invention.
11 is a flowchart illustrating a face authentication operation process.
12 shows extraction results of major facial feature points using CLM.
FIG. 13 exemplifies a set of face images captured by 14 people in the FEI Face Database.

Technical terms used in the present invention are merely used to describe particular embodiments, it should be noted that it is not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as meanings generally understood by those skilled in the art unless the present invention is defined in any other meaning in the present invention, and is excessively comprehensive. It should not be interpreted in the sense of or in the sense of being excessively reduced. In addition, when the technical terminology used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be understood as being replaced by a technical term that can be properly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced sense.

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of an online lecture system according to the present invention.

Figure 1 (a) shows the user interface of the administrator mode of the online lecture monitoring system, Figure 1 (b) shows the normal operating state of the online lecture monitoring system.

The screen interface may include an area for displaying an input image of a participant according to an administrator mode and a user mode.

Figure 1 (a) shows the screen interface in the administrator mode, pressing the User Mode button (user mode switch button), the operation mode is switched from the administrator mode to the user mode while the User Mode button is the Admin Mode button (administrator) Mode switch button). Conversely, if the Admin Mode button is pressed in the user mode, the operation mode is changed to the administrator mode and the Admin Mode button is changed to the User Mode button at the same time.

In the user mode, the right screen displaying the face image of the participant and the list box displaying the screen gaze information are removed and only the content screen and the operation control button remain.

When the desired lecture content is selected and played on the user interface as shown in FIG. 1 (a), the face region is detected from the input image acquired through the PC cam, and then the screen staring rate can be calculated by monitoring the eye position and the open / closed state. By examining the number of screens with a change in pupil position between adjacent screens in a certain sequence section, the deceptive state of viewing can also be identified.

Referring to FIG. 1 (b), the entire screen is divided into four regions having different sizes to display a content screen on the upper left, an input image on the upper right, and various function buttons are located on the lower left, and the lower right Display a list box.

An input image of the participant photographed through the front camera of the computer is displayed on the input image, and the detection result of the face region, facial feature elements, and whether the eyes are opened or closed is displayed on the input image.

Various function buttons can be arranged according to the characteristics of the function. For example, there are Open button, Play button, Stop button and Pause button for basic video play on the left side, Eye State button, Cherry Picker operation button, Smart Stay operation on the right side. The button and the mode switch button are located.

Play button (play button) and Pause button (pause button) can control whether or not the course content is played, press the Stop button (stop button) to stop the course content playback, the number of full screen, the number of screens The screen rate, viewing rate and final learning rate are initialized.

When the Eye State button (eye open / close button) is pressed, the result of the eye open / closed judgment is displayed on the right screen by using the gradient-based high speed eye detection method and the adaptive eye open / closed judgment algorithm. If the display is closed and the eyes are closed, it is indicated by a red ellipse and the red warning light is displayed on the upper left of the content screen in consideration of not staring at the screen. The list box outputs Open Eye if the eyes are opened and Closed Eye if the eyes are closed.

The Cherry Picker button is used to detect cherry pickers, and analyzes the loopholes in the gaze judgment algorithm and uses the face of a person's face or a doll to deceive the lectures as if the participant is staring at the video content. This function is to prevent recognition. If there is no movement of the pupil within a predetermined frame section of the input image sequence, it is determined that the viewing is deceptive and the attendance is not recognized.

When the Smart Stay button is pressed, it judges whether the input image is open or closed and whether there is a face. When the learner on the right screen is watching the lecture contents, the normal playback is continued. When the player enters the stopped state and watches again, the user changes from the pause state to the play state.

In the list box, the number of screens that have been played so far, the number of screens that have been checked by eye tracking, the screen viewing rate that is the ratio of the number of screens to the number of screens, and the viewing dazzle rate that is the ratio of the number of screens to the screen The final learning rate, the eye open state, and the eye detection time, which are the ratio of the number of screens minus the number of dazzle screens and the total screens, are displayed.

In addition, when finally viewing all the contents of a lecture, for example, by displaying the final learning rate of the learner through a pop-up window, the learner can determine the validity or invalidity of the attendance of the online course attended.

2 is a flowchart illustrating a method for monitoring online lectures according to the present invention.

When the software is operated on the computer, the lecture contents are selected and played by opening the file (step S21).

When lecture content is selected with the Open button and played with the Play button, the lecture content is played on the left screen, and the face image of the participant photographed through the front camera is displayed on the right screen.

A face is detected from an input image of a participant acquired through a camera, a gaze is tracked, and an amount of change in eye movement between frames is examined to detect viewing deception (step S22).

Then, it is determined whether the student stares at the screen (step S23).

As a result, if the learner's face and pupil are detected while the lecture contents are being played, the student's face authentication is prevented through the verification of the learner's face (step S24), and then judged whether the viewing is deceptive (step S25). The number of screens is increased (step S26).

On the other hand, if the face and pupil of the participant are not detected, it is determined that the user does not stare at the screen and the smart stay function is activated (step S27), and the number of non-stare screens is increased and a red warning light is displayed at the upper left of the content screen (step S28).

Thereafter, it is checked whether the lecture content reproduction is finished (step S29), and if not, the process returns to step S22, and if it is finished, the screen gaze rate is calculated (step S30).

Hereinafter, eye detection, eye open / closed judgment and viewing deception detection in step S22 will be described in detail.

<Eye detection>

Conventional methods have been proposed to reduce the computational burden and improve the detection accuracy of conventional methods through the method using the dual edge map, pupil candidate region, non-pupillary region labeling, and pupil candidate region labeling. In order to provide a more accurate method, it is necessary to develop a method to further improve eye detection accuracy even in a more extreme situation.

In order to meet this need, the present invention proposes a gradient-based eye detection method that improves eye detection accuracy by labeling the pupil area in both directions while converging in two concentric directions originating from the center and the outside of the eye search area, respectively. do.

First, the method of the present invention detects a face region from an input image using a Haar-like feature, an AdaBoost algorithm, and adaptive template matching, and designates left and right eye searching regions as shown in FIG. .

Next, the gaussian filtered inverse brightness image of the eye search region is normalized by histogram smoothing, and the pupil candidate region is extracted through threshold processing.

의 영상

을 구한 후, 가우시안 필터링을 취한다.

를 식 (2)와 같이 히스토그램 평활화를 취한 후, 식 (3)처럼

의 최대값을 이용해 정규화 영상

을 구한다.First, the eye navigation area as shown in equation (1)

Video of

Then, Gaussian filtering is performed.

After histogram smoothing as in Eq. (2),

Normalized image using maximum of

Obtain

Thereafter, a predetermined threshold value obtained experimentally is used to classify the pupil candidate region including the center of the pupil and the non-pupillary region, which is another region.

를 대상으로 서로 다른 값으로 임계처리해 구한 이진 영상으로, 상부는 안경을 착용한 경우이고 하부는 안경을 착용하지 않은 경우이다.3 is a normalized image

This is a binary image obtained by thresholding different values with respect to the upper part when wearing glasses and the lower part when wearing glasses.

Here, the bright place is the pupil candidate area including the center of the pupil, and the dark place is the non-pupillary area. It can be seen that as the threshold is smaller, the area of the non-pupillary region gradually increases.

Then, after accumulating the inner product between the gradient vector and the normal displacement vector of the edge pixels in the eye search region, the position of the maximum cumulative value is detected as the center of the left and right eyes. At this time, the center of the pupil is located in the vicinity of the low-brightness region of the left and right eye search center region, respectively, and proceeds in the direction of concentric circles from the center of the eye search region to label the pupil area within a predetermined pixel area limit.

4 shows an example in which pupil area labeling is applied.

Next, when the labeled pupil area is in contact with the edge of the eye search area, the border adjacent area is designated as a non-porous area within the predetermined ratio limit of the labeled pupil area while proceeding in the concentric contraction direction from the border.

Finally, the inner cumulative calculation is limited to only the labeled pupil area of FIG. 4 (b), not the entire eye searching area, and then the position of the maximum inner cumulative value is detected as the center of the left and right eyes.

The eye open / close determination method of the present invention can improve the performance of the conventional gradient vector field-based eye open / close determination method by performing Kalman filtering on the corrected cumulative internal mean to suppress performance degradation due to unspecified external disturbances.

5 shows a local block mask of the pupil center region.

In order to determine whether the eyes are opened or closed, a local block mask having a predetermined pixel size (for example, 7 × 7 pixel size) is specified around the positions of the maximum inner cumulative values in the left and right eye search areas, and then the pixels included in the block mask. Calculate the mean by summing the inner cumulative values of and dividing by the size of the block mask.

Thereafter, the inner cumulative average is divided by the number of gradient vectors and the eye open / close threshold is determined to determine whether the eye is open or closed.

Statistically, the magnitude of the internal cumulative mean obtained in the pupil region of the same person is larger when the eyes are opened and smaller when the eyes are closed as shown in FIG. 5. Also, under the same conditions, the value increases when the glasses are worn and increases in proportion to the size of the eyes and the pupils. However, since the internal cumulative average is severely different among individuals, it is difficult to reduce the probability of misclassification below a certain level simply by determining whether eyes are opened or closed based on this value.

Since the inner cumulative value is calculated by multiplying each normal displacement vector of the pupil candidate area with all the gradient vectors in the eye search area, the larger the number of the gradient vectors, the larger the inner cumulative value becomes. That is, the number of gradient vectors is the number of operations that accumulate the dot product.

을 그레이디언트 벡터의 개수

로 나눠 보정된 내적 누적 평균

을 구함으로써 이러한 편차를 효과적으로 줄일 수 있다. Therefore, the cumulative inner product mean as in equation (4)

Number of gradient vectors

Cumulative internal mean corrected by

This variation can be effectively reduced by

In this case, the internal cumulative average corrected for each frame unit may be used as it is. However, if Kalman filtering is performed on the corrected cumulative cumulative mean values of the previous frame and the current frame in a certain section as shown in Equation (5), the corrected cumulative inner cumulative mean of the current frame is updated. The adverse effects can be properly suppressed to induce stable operation. The method of the present invention performs Kalman filtering in units of five frames including the current frame to update the corrected cumulative mean of the current frame.

On the other hand, in order to determine whether the eyes open or close it is necessary to obtain the eye open and close threshold. Each time a new frame is input, a relatively long previous frame period and a corrected inner cumulative mean value of the current frame are summed by performing Kalman filtering, and the frame period average is obtained by dividing by the number of frames. The method of the present invention calculates the average frame interval of the current frame by performing Kalman filtering and averaging on the basis of 70 frames including the current frame.

In this case, when the average of the frame interval is obtained, the frame interval average of the open state may be obtained by excluding the corrected internal cumulative average value of the frame determined to be closed eyes. The frame section average obtained as described above can be adaptively responded to the detachment of glasses, the change of the person, or the change of the lighting by updating the value by overlapping the frame sections each time the next frame is input.

은 식 (6)과 같이 이 프레임 구간 평균에 비례 계수

를 곱해 각 프레임 단위로 결정된다. 본 발명에서는 이 비례 계수로서 실험적으로 구한 0.8을 사용하고 있다. Eye open and close threshold

Is proportional to the mean of this frame interval as

Multiply by to determine each frame unit. In the present invention, 0.8 obtained experimentally is used as this proportional coefficient.

Finally, as shown in equation (7), when the corrected cumulative mean value of each frame is larger than the eye open / close threshold value, it is determined that the eyes are opened, otherwise it is determined that the eyes are closed.

6 is a graph showing whether the eye is opened or closed by using an internal cumulative average corrected by the number of gradient vectors.

When the corrected inner cumulative mean value is larger than the eye open / close threshold value, it is determined that the eyes are opened, and it can be confirmed that the eye open / close judgment performance is provided.

<Viewing deception detection>

The deception function of viewing is to analyze the loopholes of the gaze determination algorithm and to prevent the attendees from being falsely admitted to the online lectures as if the learners are staring at the contents of the lecture using human photos, figurines, and sculptures.

Figure 7 illustrates the situation of deception detection experiment of the system of the present invention.

When a real person watches a video content, eye movements, distortions, winks, and blinks are caused by changes in muscles and eye movements around the eyes. Accordingly, the detected final eye position also fluctuates slightly. However, in general, an object having a characteristic of a human face such as a photograph or a doll does not cause eye movement.

In the system of the present invention, if a face region or an eye center point does not change in consecutive frames of a predetermined period or more, it is determined that a participant does not stare at the corresponding lecture content.

However, when the actual system is implemented, even a doll or a picture without motion may include minute false vibration when detecting the eye center point in the captured image due to a change in ambient illumination or photoelectric conversion noise of the camera.

To suppress this, if the difference between the vertical and horizontal coordinates of the left and right eye center points between the previous frame and the current frame is less than a predetermined reference value, the eye center point of the current frame is regarded as false tremor and replaced with the value of the previous frame.

The system of the present invention has an eye position stabilization measure, which is experimentally set to 2 pixels. If the reference value is set to a too large value, the normal judgment is incorrectly judged to be a deceptive state of viewing, and vice versa.

Next, after performing Kalman filtering on the horizontal and vertical coordinates of the eye center point from which the false blur is eliminated between successive frames, the difference between the vertical and horizontal coordinates of the left and right eye center points between the current frame and the previous frame is constant. If it is below the threshold, it is judged that the center point of the eye has not changed and the number of eye stops is increased.

If the number of eyeball stops is equal to or greater than the predetermined number of stops during a predetermined frame period, it is determined as a viewing delusion state. Otherwise, it is determined to be normal viewing, and the watchdog is continuously monitored while initializing the number of eyeball stops for each predetermined frame period.

For example, the predetermined reference value may be 3 pixels, the predetermined frame section may be 30 frames, and the number of predetermined stops may be 29 times.

Therefore, when the difference between the vertical and horizontal coordinates of the left and right eye center points between the current frame and the previous frame is 3 pixels or less, the value of the eye stop count α is increased. If this condition is satisfied, it is determined that the eye's center point does not change.

At this time, even if a real person stares at the screen, the eye's center point often does not change temporarily when the eyes are focused, so if the α value increases more than 29 times in 30 frames, it is judged to be a deceptive state of viewing. We discriminate by city hall. The α value is initialized every 30 frames to continuously monitor whether or not viewing is deceptive.

If the attendees normally watch the video, the red warning light does not come in. However, if the student is a fake student such as a photograph or a doll, the red warning light is displayed to warn the outside that it is judged to be a deceptive state.

Based on the entire frame, if it is determined that the viewing is deceptive in the section of 85% or more, it is finally determined as the deliberate viewing deception intention, and as shown in FIG. 8, the final learning rate is treated as 0% to induce the viewer to watch the lecture again.

Through this process, it is possible to block the deceptive viewers who are subtly recognized by using the system loophole without actually watching the lecture.

Simulation of eye detection, eye open / closed judgment and gaze deception detection

For performance evaluation, the simulation was performed using Microsoft Visual C ++ 2013, OpenCV 2.4.9 in Intel Core i7-7500U CPU, 8GB DDR4 RAM, Geforce 940MX (2GB).

First, a test image sequence for eye detection is a face image obtained through a CMOS webcam, which is taken under normal illumination (about 400 lux) indoor lighting at a distance of about 25 to 75 cm. A total of 4,436 640 × 480 images consisted of 2,093 and 2,343 images of glasses without wearing and glasses wearing, respectively.

Table 1 compares the eye detection performance of the method of the present invention and the existing methods. According to the simulation results, it has the advantage of maintaining a similar computational speed as the improved conventional method while providing an excellent reproducibility compared to the conventional method and the improved conventional method.

Way
division Conventional
Way Conventional Improved Method Method of the invention Without glasses (2,093 pieces) % Recall 81.12 99.56 99.85 Wearing glasses (2,343) % Recall 90.01 97.05 99.57

9 illustrates an eye detection result to which the method of the present invention is applied. It can be seen that the pupils on the left and right sides are detected relatively accurately without distinguishing between wearing glasses and not wearing glasses. In most eye detection methods, wearing glasses is a major cause of poor detection accuracy. The method of the present invention has the advantage that the non-pupillary areas such as eyeglass frames, eyebrows and hair are naturally removed in the pupil area labeling process, which simultaneously improves the detection accuracy and the calculation speed. Overall, the method of the present invention has the advantage of providing excellent eye detection accuracy while requiring a low calculation amount.

On the other hand, the test image sequence for eye open / closed determination is 9,909 total of 640 × 480 sizes of 4,534 and 5,375 images of glasses-free and glasses-wearing face images taken under the same conditions as eye detection.

Table 2 compares the eye open / closed determination performance of the method of the present invention and the conventional method, and FIG. 10 shows an image of the determination result using the eye open / closed determination method of the present invention.

Way
division Conventional
Way Of the present invention
Way Wearing glasses
(4,534) F1-Measure (%) 92.5 96.2 Processing time per frame (msec) 25 26 Wearing glasses
(5,375) F1-Measure (%) 94.8 96.8 Processing time per frame (msec) 22 25

When the eye open / closed judgment performance was measured by applying the eye open / closed judgment method of the present invention as shown in Table 2, the F ₁ -Measure value was 96.2% as a precision and 96.3% recall when the glasses were not worn. In addition, when wearing glasses, the accuracy of 97.1% and 96.6 recall were F ₁ -Measure value of 96.8%.

Thus, when not wearing glasses and wearing glasses, it was confirmed that they provide superior performance by 3.7% p and 2.0% p, respectively, compared to the conventional eye open / close determination method. In addition, the method of the present invention was found to provide performance similar to the conventional method in terms of processing time per frame.

Table 3 compares the deceptive detection performance of the method of the present invention and the conventional method. The method of the present invention is 99.40% recall and 98.82% precision, with a F ₁ -Measure value of 99.12%. .

Way
division Conventional
Way Of the present invention
Way Dazzle pictures
(5,110)
+
Real people
(5,360 photos)
% Recall
85.11
99.40 % Accurate 99.25 98.82 F1-Measure (%) 91.64 99.12

The deceptive detection method of the present invention was able to confirm that the F ₁ -Measure value is superior to the conventional method by 7.48% p, the reason that the accuracy is slightly lower than the conventional method is because the horizontal and vertical coordinates of the eye center point This is because when the Kalman filtering is performed on the object, the change in the characteristics of the Kalman filter tends to be suppressed, thereby slightly increasing the rate of misjudged the normal viewing to the deceptive state.

As described above, the gaze tracking technology adopted in the online lecture monitoring method of the present invention expands or converges in two concentric directions originating from the center and the outer portion of the eye search area, respectively, and the pupil area is bidirectionally limited by the limited number of pixels. The detection accuracy and the computation speed are improved by labeling and performing internal cumulative calculation only in the corresponding area.

In addition, as the number of gradient vectors increases as the number of gradient vectors increases from the simulation results of the eye open / close judgment method, the cumulative inner cumulative average is divided by the number of gradient vectors to obtain a corrected inner cumulative mean, and performance deteriorates due to unspecified external warming. In order to avoid the Kalman filtering was performed to determine the superiority of the method of determining whether the eyes open or closed.

According to the simulation results, the eye detection method of the present invention has an advantage of maintaining the operation speed similar to the conventional method improved when wearing glasses and non-wear while providing excellent accuracy compared to the existing method and the improved conventional method.

In addition, the eye detection and eye open and close determination method of the present invention is more robust to changes in the distance between the camera and the face than the conventional method, and provides a relatively high detection rate even in a low light image, and is particularly good even when wearing glasses. There was an advantage in providing performance.

Hereinafter, the face authentication operation process in step S24 will be described in detail.

11 is a flowchart illustrating a face authentication operation process.

A Gabor kernel having a plurality of directions and frequency components is generated (step S111).

Subsequently, a main facial feature point is extracted in advance on the gallery face image (step S112).

For example, 68 major facial landmarks exist around the eyes, nose, mouth and chin using the CLM (Constrained Local Model) proposed by D. Cristinacce and T. Cootes for offline gallery facial images. Is detected.

The gallery feature vector using the Gabor kernel is extracted and stored at each facial feature point in advance offline (step S113).

For example, a 40-dimensional Gabor feature vector consisting of eight directions and five frequencies is extracted and stored in the state of performing geometric normalization of rotation and magnitude in terms of the detected facial feature points.

Subsequently, the similarity between the Gabor feature vector of the Probe face image extracted online and the previously stored gallery feature vector is measured (step S114).

If the gallery face image that is most similar to the probe face image contains the face of the same person as the probe face image, it is determined as a normal classification. Otherwise, it is determined as a misclassification. A ratio (Nearest Neighbor Distance Ratio) is obtained, and when the nearest distance ratio NNDR is smaller than the authentication threshold value, it is determined that the face authentication succeeds, otherwise, it is determined to be unauthorized (step S116).

Hereinafter, each process is explained in full detail.

<CLM-based Major Facial Feature Detection>

Constrained Local Models (CLMs) are used to explore facial regions and to find differences in facial features such as eyes, nose, and mouth, or from others.

Briefly explaining the concept of the CLM model building and searching process, first, it is necessary to search the face region in a given input image, which is used by the Viola-Jones algorithm.

Subsequently, a process of searching for an appearance element of a face in the searched face area is performed. In this case, the concept of shape constraint, which is a relative arrangement of appearance elements, is applied. The shape constraints mean that the appearance elements such as eyes, nose, mouth, etc. are present in a certain part of the face, can not escape any more than this specific part, and must observe the relative placement relationship with other appearance elements. For example, the eye may refer to a protocol such that the eye cannot be moved beyond a certain range of the face, the nose and the mouth cannot be positioned above the eye, and the like.

It can be expressed as a local patch of each of the unique facial elements such as eyes, nose, mouth, and eyebrows. In order to detect major facial features based on CLM, two types of information related to this patch are shape model. ) And a patch model. The shape model represents position (or placement) information of each patch, and the patch model represents appearance (appearance, pattern, texture, etc.) information of each patch.

Naturally, first, CLM Model-building needs to be done in advance using training face data. When two models of the shape model and the patch model are prepared through the CLM model construction process, when a new input image is given, the location of each major facial landmark is searched using the pre-built CLM model. This goal is achieved by searching for patches around the most probable location points for each major facial feature that you want to find, using shape constraints to limit the search range and also to place them relative to other feature points. Control not to break the protocol.

Whether it's a training course or a practice course, CLM needs to remove the variance factors of face rotation, size, and translation before preprocessing to find the main facial features of the face. Procrustes analysis is performed for this purpose. A shape model indicating where the patches of each major facial feature point are located is constructed using Principle Component Analysis (PCA). In this process, the PCA learns how much shape variation the main facial feature points have from each average shape position from the training face data. The PCA is based on the fact that the types of variations can be expressed as eigen vectors, and their corresponding eigen values can represent the degree of the deviation as a value.

The patch model, on the other hand, is built using SVM (Support Vector Machine). When SVM classifies data using hyperplanes, the key idea is to find the plane with the largest margin, the distance between each hyperplane and each data. For example, if you have an eye patch and a nose patch, you can find the hyperplane with the largest margin between the two. In other words, a decision boundary is needed to classify whether each patch is a valid patch or not, and using SVM as a way of drawing this decision boundary. This is the process of finding the weight of each SVM in a given training data image and using these weights to classify the patches.

For example, whether a patch has a nose can be expressed as a pattern of each pixel brightness value (or color value) constituting the patch. The output of the SVM is formulated as a linear function of each pixel brightness value and each pixel SVM training is performed to find the appropriate weights of brightness values. The process of building a patch model with linear SVM is to find the weight of each SVM using the given training images. After finding the weights, the weights are used to search for patches of a predetermined size in a limited local area of the input image to search for an optimal patch containing the nose.

As shown in the previous example, after the CLM model is constructed, the search is possible. For the search, the initial position is initialized to the average shape position of each major feature point, and template matching is performed to search the local area around each feature point. The response characteristics are examined in the searched local area and the position having the highest value is designated as the main facial feature point. At this time, it is determined by excluding the feature point that violates the shape constraint. Therefore, if a feature point with the highest response characteristic violates the constraint, a rescan process is required. If the rescan process is repeated, the amount of calculation may increase. In order to reduce the amount of computation, the response characteristics can be modeled as quadratic or quadratic Gaussian functions, and the function can be optimized to quickly find the optimal location of each feature point. In this way, the location of all major facial features is searched.

12 shows extraction results of major facial feature points using CLM.

Gabor Wavelets

The Gabor function takes a triangular waveform with a Gaussian envelope of one signal. Since it is a filter that is optimally localized in both the spatial domain and the spatial frequency domain, it is known to have characteristics that are robust against noise and light changes. To establish a model applying the orientation of the selected simple cell, the 2D Gabor function is generalized as shown in Equation (8).

는 가우시안 함수에 의해 포락된 벡터

에 의해 특성을 갖는 평면파이고,

는 가우시안 포락선의 표준편차이다. 이때

은 서로 다른 공간 주파수 대역이 대략적으로 동일한 에너지를 갖도록 조정한다.

번째 Gabor 커널은

로 주어지는 주파수와 방향을 갖는 특성파 벡터로 표현된다.bracket

Is a vector enveloped by a Gaussian function

Plane pie having characteristics by

Is the standard deviation of the Gaussian envelope. At this time

Is adjusted so that different spatial frequency bands have approximately the same energy.

The first Gabor kernel

It is expressed as a characteristic wave vector with frequency and direction given by.

와

는 Gabor 커널의 방향과 주파수의 인덱스이고

이고

이다. 그리고

는 주파수 영역에서 커널 사이의 공간 팩터(spacing factor)이다. 가우시안의 폭은

인데, 비례 계수

를 적절히 정하는 것이 매우 중요하다. here

Wow

Is the index of the direction and frequency of the Gabor kernel

ego

to be. And

Is the spacing factor between kernels in the frequency domain. Gaussian width

Is a proportional factor

It is very important to set the appropriately.

In Equation (8), the first equation in parentheses determines the oscillation part of the kernel, and the second equation corrects the kernel's DC (Direct Current) value. By excluding the DC response, Gabor Wavelets become insensitive to light changes.

에서 주어진 특징점

의 위치에 다중 방향 및 주파수의 Gabor 커널로 컨벌루션(convolution)을 수행하는데, 통상

를 가버 제트(Gabor jet)라고 한다. 여기서

는

위치에서의 영상 밝기값이다.Image as shown in equation (10)

Feature points given by

Convolution is performed with a Gabor kernel of multiple directions and frequencies at the position of

Is called a Gabor jet. here

Is

Image brightness at the location.

,

,

이고 5개의 다른 주파수

와 8개의 다른 방향

을 갖는 Gabor 커널을 사용한다. 5개의 주파수 성분

과 8개의 방향 성분

의 조합에 의해

로 인덱스되는 40개의 파형 벡터를 구한다.In the present invention

,

,

And 5 different frequencies

And 8 different directions

Use a Gabor kernel with 5 frequency components

And eight direction components

By a combination of

Obtain 40 waveform vectors indexed by.

Gabor wavelets need to determine how many filters to apply to which reference points. Applying them to all the pixels in an image will give you complete information, but it will result in a heavy computational load and information duplication. In general, Gabor kernels of multiple directions and frequencies are applied only to the main features of rectangular grids or faces at appropriate intervals. In the present invention, eye, nose, and mouth obtained through the extraction of facial features based on well-known Constrained Local Models (CLM) 40-dimensional Gabor feature vectors are obtained for each of the 68 major facial feature points representing eyebrows, facial contours, and the like.

번째 특징점의 Gabor 제트

을 40개의 복소 계수(complex coefficients)의 집합

를 이용하여 표현하면,

가 된다. Each of these 40-dimensional Gabor feature vectors is called a Gabor jet.

Gabor jet at the first feature point

Is a set of 40 complex coefficients

If you express using

Becomes

는 가버 커널의 인덱스(

)이고

은 각 특징점(

)의 인덱스를 의미한다. 가버 제트의 각 요소는

로 표현된다. 이때

는 허수(imaginary number)를 의미하는 기호이다.At this time,

Is the index of the Gabor kernel.

)ego

Each feature point (

) Index. Each element of the Gabor Jet

It is expressed as At this time

Is a symbol for imaginary number.

번째 갤러리 얼굴 영상의 얼굴 특징 그래프에 대한 전체 가버 제트는 식(11)과 같이 각각

및

로 표현할 수 있다. A facial feature graph is composed of a set of all major facial feature points of one face image. Probe face imaging and

The total Gabor jets for the facial feature graph of the first gallery face image are shown in Eq. (11), respectively.

And

Can be expressed as

<Face Recognition>

For face authentication, it is necessary to obtain individual similarity by using cross correlation between corresponding specific points, and total similarity which is weighted combination of all individual similarities.

번째 Probe 얼굴 특징점과 갤러리 얼굴 특징점 간의 가버 제트 유사도

을 다음과 같이 정의한다.priority,

Gabor Jet Similarity Between the First Probe Face Feature and the Gallery Face Feature

Define as

번째 갤러리 얼굴 특징 그래프 간의 전체 가버 제트 유사도

를 식(13)과 같이 정의한다. 여기서

은 68개이고,

은

번째 특징점의 가중치로서 실험적으로 결정하였다.Next we have a Probe facial feature graph

Total Gabor Jet Similarity Between Second Gallery Facial Feature Graphs

Is defined as in Equation (13). here

Is 68,

silver

It was determined experimentally as the weight of the first feature point.

The total similarity between the probe face image and all gallery face images for face authentication is determined by Equation (13), and if the gallery face with the highest similarity to the probe face is the same face as the probe face, it is classified as a correct classification. Determine.

와 가장 잘 정합된 갤러리 얼굴 영상의 얼굴 특징 그래프

간의 전체 가버 제트 유사도가

이고, 프로브 얼굴 특징 그래프

와 두 번째로 잘 정합된 갤러리 얼굴 특징 그래프

간의 전체 가버 제트 유사도가

이라고 가정할 때, 최근접 거리비(NNDR)는 다음과 같이 정의할 수 있다.In addition, the method of the present invention performs final face authentication by using the nearest nearest distance ratio (NNDR) as a face authentication measure only when it is classified in order to increase the accuracy of face authentication. Facial feature graph of probe face image

Feature graph of gallery face images best matched with

Overall Gabor Jet Similarity Between

Probe face feature graph

Second best matched gallery facial features graph with

Overall Gabor Jet Similarity Between

In this case, the nearest distance ratio NNDR can be defined as follows.

)보다 작으면(

), 프로브 얼굴의 인증이 성공한 것으로 판정하고 그렇지 않으면 불인증으로 판정한다. 본 발명에서는 인증 임계값(

)으로 0.95를 사용한다. The closest distance ratio of these two faces in equation (14) is

Less than)

), It is determined that authentication of the probe face is successful, otherwise, it is determined as unauthorized. In the present invention, the authentication threshold (

0.95 is used.

Experiments with various researchers have shown that the NNDR scale provides relatively better performance than finding a nearest neighbor below the threshold or using a fixed threshold.

The above face authentication process is briefly described again, when the closest distance ratio NNDR is smaller than the authentication threshold value (0.95) only when the gallery face having the most similarity to the probe face is classified as the same face as the probe face. It is determined that face authentication succeeds, otherwise it is determined not to be authenticated.

Simulation Results and Discussion

To evaluate the performance of the system, simulations were performed using Microsoft Visual C ++ 2015 and OpenCV 3.0.0 in an Intel Core i7-7500U CPU, 8GB DDR4 RAM, and Geforce 940MX (2GB).

In the face recognition stage, the face recognition rate was evaluated based on the FEI database produced and published by FEI University in Brazil.

FIG. 13 exemplifies a set of face images captured by 14 people in the FEI Face Database.

o Paulo, Brazil)에 소재한 FEI 대학교(University Center of FEI(Faculty of Industrial Engineering))의 인공지능연구실(Artificial Intelligence Laboratory에서 2005년 6월~2006년 3월 사이에 200명(19세~40세의 남자 100명 및 여자 100명)의 브라질인을 대상으로 각 사람당 14장씩 촬영한 얼굴 데이터베이스로, 얼굴 인식 및 인증 관련 연구자들에게 널려 알려진 공개된 벤치마킹 데이터베이스이다. 남녀 혼합, 다양한 인종(백인, 흑인, 황인), 안경 착용 및 미착용, 정상 조도 및 저 조도 환경의 흰색 배경상에서 180도 회전하면서 왼쪽 및 오른쪽의 단계적 측면 얼굴과 정면 얼굴을 촬영한 640×480 크기의 상반신 컬러 얼굴로 구성되어 있다. 특히 각 사람당 4장의 정면 얼굴 영상은 정상 조도의 무표정 정면 얼굴(11번 영상) 및 미소진 정면 얼굴(12번 영상), 중간 조도의 정면 얼굴(13번 영상), 저 조도의 정면 얼굴(14번 영상)로 구성된다. 도 13은 FEI Face 데이터베이스의 2번 남자의 좌상부터 우하단 순으로 1번 영상~14번 얼굴 영상을 예시한 것이다.The FEI Face database is located in São Paulo, Brazil.

o 200 people (19-40 years old) between June 2005 and March 2006 at the Artificial Intelligence Laboratory at the University Center of Faculty of Industrial Engineering (FEI) in Paulo, Brazil. This is a database of 14 images taken for each person from 100 Brazilians and 100 females, and is an open benchmark database known to researchers involved in facial recognition and authentication. Yellow face), 640 × 480 upper torso color face with 180 degrees rotated to the left and right side face and front face on white background in normal and low light conditions. Four frontal face images per person include the normal faceless front face (image 11) and the unsharp front face (image 12), the front face with medium illumination (image 13), and the low It consists of a front face (image 14) Figure 13 illustrates images 1 to 14 of the face from the top left to the bottom right of the second man of the FEI Face database.

When the probe face image (image 11 or 13) is normal or medium illuminance, the gallery face image uses a total of 1,230 shots for each of 200 FEI Face Databases and 5 self shots. When the face image (image 14) was low, a total of 994 images were used, with 7 images per person for 137 FEI Face Databases and 5 self-images. In the situation where the illumination state of the probe face image varies from normal illumination to low illumination, it can be confirmed that the average face recognition rate is 98.72%.

Probe face image index People in gallery Classification Misclassification NNDR certification Facial recognition rate 11, 13, 14 546 people 544 people 2 people 539 people 98.72%

Probe face image index People in gallery Face per person
Number of images Classification Misclassification NNDR certification Facial recognition rate 11th 205 people Chapter 6 204 people 1 person 201 people 98.05%

Probe face image index People in gallery Face per person
Number of images Classification Misclassification NNDR certification Facial recognition rate 13th 205 people Chapter 6 205 people 0 people 203 people 99.02%

Probe face image index People in gallery Face per person
Number of images Classification Misclassification NNDR certification Facial recognition rate 14th 136 people Chapter seven 135 people 1 person 135 people 99.26%

Table 4 shows the average face authentication rate of the system applied to the present invention, Tables 5 to 7 shows the face authentication rate for the case where the probe face image is normal illumination, medium illumination and low illumination, respectively.

In the present invention, after extracting the main components of the face through the extraction of facial feature points based on Constrained Local Models (CLM) and generating the Gabor feature vector through Gabor wavelet transformation on the extracted feature points, We proposed a method of authenticating individuals using the correlation and the nearest distance ratio.

According to the simulation results, it was confirmed that the method of the present invention provides a good face authentication rate of 98.72% on average even when the illumination state of the probe face image varies from normal illumination to low illumination. In particular, it has an advantage of providing excellent face recognition rate of 99.26% when a low light probe probe image is input. On average, 98.72% of the face recognition rates proved that the online lecture monitoring system is useful at the practical level when applied to the online lecture monitoring system.

The face authentication method of the present invention checks whether a student or a reviewer has normally watched or learned online contents in a remote place such as remote screening or inspection, compulsory pay training, access management, and patient care in addition to online lecture monitoring. It is expected to be widely used in applications that monitor whether or not the subject behaves according to its original purpose. Of course, even if applied to the field of traditional face recognition or face authentication is expected to provide good results.

When the method of the present invention is applied to a gaze tracking application system such as drowsiness prevention, gaze control, online lecture monitoring, etc., it can provide a stable gaze tracking function and can also be usefully applied to an application that utilizes eye open / close judgment. It is expected to be.

As described above, the online lecture monitoring method of the present invention monitors whether the online lectures are faithfully watched, prevents the false attendance of the viewers, and automatically replays or stops the lectures according to the situation of viewing deception or gaze or return. It can improve the reliability, learning efficiency and user convenience of online lecture system.

In particular, it is useful to induce participants to participate faithfully in online lectures while not disturbing the overall learning flow.

In addition, in terms of producing online lecture contents, it is advantageous that no additional authoring time and cost are incurred due to monitoring.

The online lecture monitoring system to which the method of the present invention is applied is not only an online lecture system but also an application field for confirming whether a student or a reviewer has normally watched or learned online contents at a remote location such as a public procurement agency online examination and compulsory remuneration training. It is expected to be widely used in.

In the above description, the embodiment of the present invention has been described, but various changes can be made at the level of those skilled in the art. Therefore, the scope of the present invention should not be construed as being limited to the above embodiments, but should be construed by the claims described below.

Claims (17)

Operating the software on a computer and selecting and playing the lecture content by opening the file;
Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames;
Determining whether the learner stares at the screen;
If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner;
When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture content screen;
The inner cumulative mean is calculated by dividing the inner cumulative mean by the number of gradient vectors, and the Kalman filtering is performed by inputting the corrected inner cumulative mean values of the previous frame and the current frame and performing the Kalman filtering. Determining whether to open or close the eyes by comparing the updated internal cumulative average and the eye opening and closing threshold value; And
In real time, from the number of full screens played so far, the number of screens checked by eye tracking, the screen staring rate which is the ratio of the number of screens to the total number of screens, the viewing dazzle rate and the number of screens which are the ratio of the number of screens and the number of full screens On-line lecture monitoring method comprising the step of displaying the final learning rate that is the ratio of the subtracted number of screens and the total number of screens. In claim 1
The face authentication step,
Generating a Gabor kernel having a plurality of directions and frequency components;
A sub-step of extracting major facial feature points from a gallery face image in advance;
Extracting and storing a gallery feature vector using a Gabor kernel at each facial feature point in advance offline;
Measuring a similarity between the Gabor feature vector of the online extracted probe face image and a previously stored gallery feature vector;
Determining that the gallery face image having the highest similarity with the probe face image includes a face of the same person as the probe face, and determines a misclassification otherwise;
A substep of obtaining a nearest neighbor distance ratio only when the gallery face having the highest similarity to the probe face is classified as the same face as the probe face; And
And determining that the face authentication succeeds when the nearest distance ratio is smaller than the authentication threshold value, and otherwise, determines that the face authentication is successful. In claim 1
The main facial features are extracted 68 using CLM (Constrained Local Models),
40 Gabor kernels are generated using eight direction components and four frequency components,
The authentication threshold is 0.95 online lecture monitoring method characterized in that. In claim 1,
The eye opening / closing threshold value is obtained by accumulating the updated internal cumulative average value of the previous frames and the current frame by a predetermined number of frames, and then using the section average value divided by the number of frames. In claim 4,
The predetermined number of frames is counted as only 70 frames determined as the opened eye state except for the closed frames,
The eye opening and closing threshold value of the online lecture monitoring method, characterized in that for using the average value of the interval multiplied by a proportional coefficient 0.8. Operating the software on a computer and selecting and playing the lecture content by opening the file;
Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames;
Determining whether the learner stares at the screen;
If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner;
When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture contents screen; And
In real time, from the number of full screens played so far, the number of screens checked by eye tracking, the screen staring rate which is the ratio of the number of screens to the total number of screens, the viewing dazzle rate and the number of screens which are the ratio of the number of screens and the number of full screens And displaying the final learning rate, which is the ratio of the subtracted screen count to the total screen count.
Detecting the viewing deception,
A sub-step of substituting the eye center point of the current frame as the value of the previous frame if the difference between the vertical and horizontal coordinates of the left and right eye center points between the previous frame and the current frame is equal to or less than a predetermined reference value;
After performing Kalman filtering on the horizontal and vertical coordinates of the eye center point from which the false tremor is removed between successive frames, the difference between the vertical and horizontal coordinates of the left and right eye center points between the current frame and the previous frame is the predetermined reference value. A sub-step of determining that the eye's center point is not changed to increase the number of eye stops; And
A sub-step of determining whether the eye stop is deceptive if the number of eye stops is equal to or greater than a predetermined stop number during a predetermined frame period; otherwise, judging from normal viewing; Online lecture monitoring method comprising a. In claim 6
The face authentication step,
Generating a Gabor kernel having a plurality of directions and frequency components;
A sub-step of extracting major facial feature points from a gallery face image in advance;
Extracting and storing a gallery feature vector using a Gabor kernel at each facial feature point in advance offline;
Measuring a similarity between the Gabor feature vector of the online extracted probe face image and a previously stored gallery feature vector;
Determining that the gallery face image having the highest similarity with the probe face image includes a face of the same person as the probe face, and determines a misclassification otherwise;
A substep of obtaining a nearest neighbor distance ratio only when the gallery face having the most similarity to the probe face is classified as the same face as the probe face; And
And determining that the face authentication succeeds when the nearest distance ratio is smaller than the authentication threshold value, and otherwise, determines that the face authentication is successful. In claim 6
The main facial features are extracted 68 using CLM (Constrained Local Models),
40 Gabor kernels are generated using eight direction components and four frequency components,
The authentication threshold is 0.95 online lecture monitoring method characterized in that. In claim 6,
Detecting the viewing deception,
A sub-step, in which the red warning light does not come in when the attendees normally watch the video, but the external warning that the red warning light is determined by displaying a red warning light when it is determined that the viewing state is deceptive;
If it is determined that the viewing dazzle state in more than 85% of the entire frame, the final judgment as the deliberate viewing deception intention, and further comprising a sub-step of processing the final learning rate to 0%. In claim 6,
The predetermined reference value is 3 pixels,
The predetermined frame period is 30 frames,
The schedule number of stops online monitoring method characterized in that 29 times. Operating the software on a computer and selecting and playing the lecture content by opening the file;
Detecting a face-to-face detection by detecting a face from an input image of a participant acquired through a camera, tracking eyeballs, and examining changes in eye movement between frames;
Determining whether the learner stares at the screen;
If it is determined that the learner gazes at the screen, a face authentication step of preventing the attendance or the attendance of the surrogate through the face authentication of the learner;
When the lecture content is being reproduced, if the face and pupil of the student are detected, it is determined whether or not the audience is deceived, and the number of screens is increased. If the face and pupil of the learner are not detected, the screen is determined not to stare. Increasing the number of screens and displaying a warning light on the lecture contents screen; And
In real time, from the total number of screens played so far, the number of screens checked by eye tracking, the screen staring rate which is the ratio of the number of screens to the total screens, the viewing dazzle rate and the number of screens, which is the ratio of the number of screens to the deception And displaying the final learning rate, which is the ratio of the subtracted screen count to the total screen count.
Determining whether to stare at the screen,
Detecting a face region in the input image;
Specifying a left and right eye search region using the detected face region;
Detecting a pupil candidate region from each of the designated eye searching regions;
Labeling the pupil area of the detected pupil candidate area as only the limited number of pixels in both directions while converging in two concentric directions originating from each of the center and the outside of the eye searching area;
A substep of determining an edge by obtaining an edge map using horizontal and vertical differential masks; And
A sub-step of performing an inner cumulative operation in the labeled pupil area to calculate an inner cumulative value and detecting the inner cumulative value as the center of the left and right eyes;
In the step of labeling the pupil area, the pupil area is labeled within a predetermined pixel area limit while proceeding from the center of the eye search area to the concentric diffusion direction, and the thus labeled pupil area may be in contact with the outside of the eye search area. In this case, while advancing in the concentric contraction direction from the outer region, an outer adjacent region is removed from the labeled pupil region within a predetermined ratio of the labeled pupil area, and designated as the final labeled pupil region,
In the step of performing the dot product accumulation operation, the dot product is obtained by taking a normal displacement vector only in the final labeled pupil area to obtain a dot product with all the gradient vectors in each eye search area, and then performs a cumulative operation to obtain the dot product cumulative value. And updating the inner cumulative value by multiplying inverse brightness weights corresponding to each of the inner cumulative values, and then detecting the position of the maximum inner cumulative value as the center of the left and right eyes. In claim 11
The face authentication step,
Generating a Gabor kernel having a plurality of directions and frequency components;
A sub-step of extracting major facial feature points from a gallery face image in advance;
Extracting and storing a gallery feature vector using a Gabor kernel at each facial feature point in advance offline;
Measuring a similarity between the Gabor feature vector of the online extracted probe face image and a previously stored gallery feature vector;
Determining that the gallery face image having the highest similarity with the probe face image includes a face of the same person as the probe face, and determines a misclassification otherwise;
A substep of obtaining a nearest neighbor distance ratio only when the gallery face having the most similarity to the probe face is classified as the same face as the probe face; And
And determining that the face authentication succeeds when the nearest distance ratio is smaller than the authentication threshold value, and otherwise, determines that the face authentication is successful. In claim 11
The main facial features are extracted 68 using CLM (Constrained Local Models),
40 Gabor kernels are generated using eight direction components and four frequency components,
The authentication threshold is 0.95 online lecture monitoring method characterized in that. In claim 11,
And if the pupil of the participant is detected or not, it is displayed on the eye region of the input image in a different color to determine whether the eye is opened or closed. In claim 11,
If a face is not detected from the input image, the lecture video is stopped, and if a face is detected, the lecture is restarted from that point in time. In claim 11,
Detecting the face area,
A sub-step of superimposing an initial face template on an upper left start point of an input image and obtaining cross correlation with the overlapped portion;
A sub-step of detecting a position having the highest cross-correlation as a face region while moving by one pixel from the starting point; And
And if the cross-correlation is above a predetermined value, continue template matching, otherwise, the online lecture monitoring method comprises a sub-step of re-detecting a face region and updating the new face template to perform template matching. In claim 16,
And if the face region is not detected, template matching is performed by lowering the cross correlation, and if the face region is not detected even in the lower cross correlation, it is determined that there is no face region in the image.

Images