JP2012055428A - Visual line detector - Google Patents

Visual line detector Download PDF

Info

Publication number
JP2012055428A
JP2012055428A JP2010200059A JP2010200059A JP2012055428A JP 2012055428 A JP2012055428 A JP 2012055428A JP 2010200059 A JP2010200059 A JP 2010200059A JP 2010200059 A JP2010200059 A JP 2010200059A JP 2012055428 A JP2012055428 A JP 2012055428A
Authority
JP
Japan
Prior art keywords
detection
line
output
sight
pupil
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2010200059A
Other languages
Japanese (ja)
Other versions
JP5358540B2 (en
Inventor
Akio Takahashi
昭夫 高橋
Shinsuke Ueda
信介 植田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honda Motor Co Ltd
Original Assignee
Honda Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honda Motor Co Ltd filed Critical Honda Motor Co Ltd
Priority to JP2010200059A priority Critical patent/JP5358540B2/en
Publication of JP2012055428A publication Critical patent/JP2012055428A/en
Application granted granted Critical
Publication of JP5358540B2 publication Critical patent/JP5358540B2/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Eye Examination Apparatus (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

PROBLEM TO BE SOLVED: To improve detection accuracy of a visual line direction and reliability of a detection result.SOLUTION: When a first visual line direction and a second visual line direction are detected, the visual line direction selection part 30 of a visual line detector 10 selects the first visual line direction in the case that the absolute value of an angle difference between the first visual line direction and the second visual line direction is smaller than a first predetermined value, and selects the second visual line direction in the case that the absolute value of the angle difference is equal to or larger than the first predetermined value. When only the first visual line direction is detected, the visual line direction selection part 30 selects the first visual line direction of a present frame in the case that the angle difference between the first visual line direction (the first visual line direction of the present frame) and the first visual line direction of the frame one before (the first visual line direction of the previous frame) is smaller than a second predetermined value, and selects the visual line direction of the previous frame as the visual line direction of the crew of a vehicle in the case that the angle difference is equal to or larger than the second predetermined value. When only the second visual line direction is detected, the visual line direction selection part 30 selects the second visual line direction as the visual line direction of the crew of the vehicle.

Description

本発明は、視線検出装置に関する。   The present invention relates to a line-of-sight detection device.

従来、例えば予め運転者が注視する所定の注視点の距離を変化させたときの網膜反射像の輝度の変化を計測して計測結果を記憶しておき、運転者の注視点を検知するときに計測した網膜反射像の輝度と、予め記憶している計測結果とを用いて、運転者の注視点の位置を計測する注視点計測装置が知られている(例えば、特許文献1参照)。
また、従来、例えば運転者の顔画像から推定される顔向き方向と、複数の運転者毎に登録された顔向きの時系列変化と注視対象との対応関係を示すモデルとを用いて、運転者の視線方向を推定する視線方向推定装置が知られている(例えば、特許文献2参照)。
Conventionally, for example, when measuring the change in luminance of the retina reflection image when the distance of a predetermined gazing point that the driver gazes at is changed and storing the measurement result and detecting the gazing point of the driver A gaze point measuring apparatus that measures the position of a driver's gaze point using the measured luminance of the retinal reflection image and a measurement result stored in advance is known (for example, see Patent Document 1).
Conventionally, for example, driving using a face orientation direction estimated from a driver's face image, and a model indicating a correspondence relationship between a time-series change in face orientation and a gaze target registered for each of a plurality of drivers. A gaze direction estimation device that estimates the gaze direction of a person is known (see, for example, Patent Document 2).

特開平7−35543号公報JP-A-7-35543 特開2008−146356号公報JP 2008-146356 A

ところで、上記従来技術に係る注視点計測装置によれば、例えば外光などによって網膜反射像や角膜反射像を適切に検出することが出来ない場合には、注視点位置の計測が困難になるという問題が生じる。
また、上記従来技術に係る視線方向推定装置によれば、視線方向は運転者毎の視認行動の学習結果に基づく推定により得られるだけであって、視線方向の推定精度を向上させることが困難であるという問題が生じる。
By the way, according to the gaze point measuring apparatus according to the above-described prior art, it is difficult to measure the gaze point position when the retinal reflection image and the corneal reflection image cannot be appropriately detected by external light, for example. Problems arise.
Further, according to the gaze direction estimation device according to the above-described conventional technology, the gaze direction is only obtained by estimation based on the learning result of the visual behavior for each driver, and it is difficult to improve the gaze direction estimation accuracy. The problem that there is.

本発明は上記事情に鑑みてなされたもので、視線方向の検出精度および検出結果の信頼性を向上させることが可能な視線検出装置を提供することを目的とする。   The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a visual line detection device capable of improving the detection accuracy of the visual line direction and the reliability of the detection result.

上記課題を解決して係る目的を達成するために、本発明の第1態様に係る視線検出装置は、ひとの顔を撮像して顔画像を出力する撮像手段(例えば、実施の形態での乗員カメラ12)と、前記ひとが存在する方向に向かって光を照射可能に配置された光源(例えば、実施の形態での光源11)と、前記撮像手段から出力された前記顔画像から瞳孔の位置および前記光源から照射された前記光による角膜反射像の位置を検出して検出結果を出力する位置検出手段(例えば、実施の形態での瞳孔位置検出部24、角膜反射像検出部25)と、該位置検出手段から出力された前記検出結果に基づいて前記ひとの視線方向を検出して検出結果を出力する第1検出手段(例えば、実施の形態での第1視線方向検知部27)とを備える視線検出装置であって、前記位置検出手段から出力された前記検出結果の前記瞳孔の位置と、前記位置検出手段から出力された前記検出結果の前記角膜反射像を含まない前記顔画像での所定部位の位置(例えば、実施の形態での目頭の位置)とに基づいて前記ひとの視線方向を検出して検出結果を出力する第2検出手段(例えば、実施の形態での第2視線方向検知部28)と、前記第1検出手段から出力された前記検出結果と前記第2検出手段から出力された前記検出結果とを比較して前記検出結果同士の差を比較結果として出力する比較手段(例えば、実施の形態での比較部29)と、前記比較手段から出力された前記比較結果の前記差が所定値未満である場合には前記第1検出手段から出力された前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出し、前記比較手段から出力された前記比較結果の前記差が所定値以上である場合には前記第2検出手段から出力された前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出する視線方向検出手段(例えば、実施の形態での視線方向選択部30)とを備える。   In order to solve the above-described problems and achieve the object, the line-of-sight detection device according to the first aspect of the present invention is an image pickup means for picking up a human face and outputting a face image (for example, an occupant in the embodiment). The position of the pupil from the face image output from the camera 12), the light source (for example, the light source 11 in the embodiment) arranged so as to be able to irradiate light in the direction in which the person exists, and the imaging means And position detection means (for example, pupil position detection unit 24, corneal reflection image detection unit 25 in the embodiment) that detects the position of the corneal reflection image by the light emitted from the light source and outputs a detection result; First detection means (for example, the first gaze direction detection unit 27 in the embodiment) that detects the gaze direction of the person based on the detection result output from the position detection means and outputs the detection result; Gaze detection device The position of the pupil of the detection result output from the position detection means and the position of a predetermined part in the face image not including the corneal reflection image of the detection result output from the position detection means (for example, Second detection means (for example, the second gaze direction detection unit 28 in the embodiment) that detects the gaze direction of the person and outputs a detection result based on the position of the eye in the embodiment), and Comparison means for comparing the detection result output from the first detection means and the detection result output from the second detection means and outputting the difference between the detection results as a comparison result (for example, in the embodiment) And when the difference between the comparison results output from the comparison means is less than a predetermined value, the person is based on the line-of-sight direction of the detection results output from the first detection means. Gaze direction And when the difference between the comparison results output from the comparison means is greater than or equal to a predetermined value, the line-of-sight direction of the person is detected based on the line-of-sight direction of the detection results output from the second detection means. Gaze direction detection means (for example, gaze direction selection unit 30 in the embodiment).

さらに、本発明の第2態様に係る視線検出装置では、前記位置検出手段は、前記撮像手段から順次出力された前記顔画像を所定時間毎に区分した各フレーム毎に前記瞳孔の位置を前記顔画像全体から検出すると共に、前記各フレームのうち所定のフレームよりも1つ前のフレームにおいて前記瞳孔の位置を検出した場合には、前記所定のフレームにおいて、前記1つ前のフレームにおいて検出した前記瞳孔の位置の近傍における前記顔画像全体よりも小さい所定範囲内から前記瞳孔の位置を検出する。   Furthermore, in the line-of-sight detection device according to the second aspect of the present invention, the position detection means determines the position of the pupil for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals. When detecting the position of the pupil in the frame immediately before a predetermined frame among the frames, the detection is performed in the predetermined frame and the frame detected in the previous frame is detected from the entire image. The position of the pupil is detected from a predetermined range smaller than the entire face image in the vicinity of the position of the pupil.

さらに、本発明の第3態様に係る視線検出装置は、前記位置検出手段は、前記撮像手段から順次出力された前記顔画像を所定時間毎に区分した各フレーム毎に前記瞳孔の位置および前記角膜反射像の位置を検出して検出結果を出力しており、前記視線方向検出手段は、前記第1検出手段および前記第2検出手段の何れか一方のみから前記検出結果が出力されたときに、前記検出結果が前記第2検出手段から出力された場合、あるいは、前記各フレームのうち適宜のフレームにおいて前記検出結果が前記第1検出手段から出力されたときに当該適宜のフレームにおいて前記第1検出手段から出力された前記検出結果の前記視線方向と前記適宜のフレームの1つ前のフレームにおいて前記視線方向検出手段により検出された前記視線方向との差が所定値未満である場合に、前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出する。   Furthermore, in the visual axis detection device according to the third aspect of the present invention, the position detection means includes the position of the pupil and the cornea for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals. The position of the reflected image is detected and a detection result is output, and when the detection result is output from only one of the first detection means and the second detection means, When the detection result is output from the second detection unit, or when the detection result is output from the first detection unit in an appropriate frame among the frames, the first detection is performed in the appropriate frame. A difference between the line-of-sight direction of the detection result output from the means and the line-of-sight direction detected by the line-of-sight direction detection means in a frame immediately before the appropriate frame. If it is less than value, to detect the gaze direction of the person on the basis of the sight line direction of the detection result.

本発明の第1態様に係る視線検出装置によれば、各検出手段から逐次出力される検出結果に対して、瞳孔の位置および角膜反射像の位置に基づいて視線方向を検出する第1検出手段では、角膜反射像によらずに視線方向を検出する第2検出手段に比べて、各検出結果の検出精度が高くなる可能性は高いが各検出結果の信頼性や複数の検出結果の安定性は低くなる場合がある。
このため、各検出手段の検出結果の差が所定値未満になることで第2検出手段の検出結果によって第1検出手段の検出結果の信頼性および安定性が高いと判断することができる場合には、第1検出手段の検出結果に基づき視線方向を検出することによって、視線方向の検出精度を向上させることができる。
また、各検出手段の検出結果の差が所定値以上になることで第1検出手段の検出結果の信頼性および安定性が低いと判断することができる場合には、第2検出手段の検出結果に基づき視線方向を検出することによって、視線方向の検出結果に対して所望の信頼性および安定性を確保することができる。
According to the line-of-sight detection device according to the first aspect of the present invention, the first detection unit that detects the line-of-sight direction based on the position of the pupil and the position of the cornea reflection image with respect to the detection result sequentially output from each detection unit. Then, the detection accuracy of each detection result is likely to be higher than that of the second detection unit that detects the line-of-sight direction without using the cornea reflection image, but the reliability of each detection result and the stability of a plurality of detection results May be lower.
For this reason, when the difference between the detection results of the detection means is less than a predetermined value, it can be determined that the detection result of the first detection means is highly reliable and stable based on the detection result of the second detection means. Can improve the detection accuracy of the visual line direction by detecting the visual line direction based on the detection result of the first detection means.
In addition, when it can be determined that the reliability and stability of the detection result of the first detection means is low because the difference between the detection results of the detection means is equal to or greater than a predetermined value, the detection result of the second detection means By detecting the line-of-sight direction based on the above, desired reliability and stability can be ensured for the detection result of the line-of-sight direction.

さらに、本発明の第2態様に係る視線検出装置によれば、適宜のフレームにおいて、1つ前のフレームで瞳孔の位置を検出した場合には、この瞳孔の位置の近傍の所定範囲内を探索することから、顔画像の全体を探索する場合に比べて、処理負荷を低減することができる。   Furthermore, according to the visual axis detection device according to the second aspect of the present invention, when the position of the pupil is detected in the previous frame in an appropriate frame, a search is made within a predetermined range near the position of the pupil. Therefore, the processing load can be reduced as compared with the case where the entire face image is searched.

さらに、本発明の第3態様に係る視線検出装置によれば、第2検出手段のみから検出結果が出力された場合には、この検出結果に基づき視線方向を検出することによって、視線方向の検出結果に対して所望の信頼性および安定性を確保することができる。
また、第1検出手段のみから検出結果が出力された場合であっても、この検出結果と1つ前のフレームで視線方向検出手段により検出された視線方向との差が所定値未満であれば、第1検出手段の検出結果が所望の信頼性および安定性を有すると判断することができ、この検出結果に基づき所望の信頼性および安定性を確保して視線方向を検出することができる。
Furthermore, according to the visual line detection device according to the third aspect of the present invention, when the detection result is output only from the second detection means, the visual line direction is detected by detecting the visual line direction based on the detection result. Desired reliability and stability can be ensured for the result.
Even if the detection result is output only from the first detection means, if the difference between the detection result and the gaze direction detected by the gaze direction detection means in the previous frame is less than a predetermined value, Thus, it can be determined that the detection result of the first detection means has the desired reliability and stability, and the line-of-sight direction can be detected while ensuring the desired reliability and stability based on the detection result.

本発明の実施の形態に係る視線検出装置の構成図である。It is a lineblock diagram of a look detection device concerning an embodiment of the invention. 本発明の実施の形態に係る視線検出装置の瞳孔位置検出部により前フレームにおいて検出された瞳孔あるいは虹彩の中心位置の近傍における所定範囲の例を示す図である。It is a figure which shows the example of the predetermined range in the vicinity of the center position of the pupil or iris detected in the front frame by the pupil position detection part of the gaze detection apparatus which concerns on embodiment of this invention. 本発明の実施の形態に係る視線検出装置の動作を示すフローチャートである。It is a flowchart which shows operation | movement of the gaze detection apparatus which concerns on embodiment of this invention.

以下、本発明の視線検出装置の一実施形態について添付図面を参照しながら説明する。
本実施の形態による視線検出装置10は、例えば図1に示すように、光源11と、乗員カメラ12と、処理装置13とを備えて構成されている。
Hereinafter, an embodiment of a visual line detection device of the present invention will be described with reference to the accompanying drawings.
The line-of-sight detection device 10 according to the present embodiment includes, for example, a light source 11, an occupant camera 12, and a processing device 13, as shown in FIG.

光源11は、撮像対象(例えば、運転席に着座した運転者の顔など)に可視光線および赤外線などの光を照射する。
乗員カメラ12は、少なくとも座席に着座した乗員の顔を撮像対象として撮像領域内に含み、例えば可視光領域および赤外線領域にて撮像可能であって、順次撮像して得られる乗員の顔を含む顔画像を出力する。
The light source 11 irradiates a subject to be imaged (for example, a driver's face seated in the driver's seat) with light such as visible light and infrared light.
The occupant camera 12 includes at least an occupant's face seated on the seat in the imaging region as an imaging target, for example, a face that can be imaged in the visible light region and the infrared region and includes the occupant's face obtained by sequentially capturing images. Output an image.

処理装置13は、例えば、照射制御部21と、乗員撮像制御部22と、顔画像取得部23と、瞳孔位置検出部24と、角膜反射像検出部25と、目頭検出部26と、第1視線方向検知部27と、第2視線方向検知部28と、比較部29と、視線方向選択部30と、視線方向記憶部31とを備えて構成されている。   The processing device 13 includes, for example, an irradiation control unit 21, an occupant imaging control unit 22, a face image acquisition unit 23, a pupil position detection unit 24, a cornea reflection image detection unit 25, an eye-head detection unit 26, and a first A line-of-sight direction detection unit 27, a second line-of-sight direction detection unit 28, a comparison unit 29, a line-of-sight direction selection unit 30, and a line-of-sight direction storage unit 31 are configured.

乗員撮像制御部22は、乗員カメラ12による撮像を制御する。
照射制御部21は、光源11による光の照射を制御する。
顔画像取得部23は、乗員カメラ12から順次出力される顔画像を取得する。
The occupant imaging control unit 22 controls imaging by the occupant camera 12.
The irradiation control unit 21 controls light irradiation by the light source 11.
The face image acquisition unit 23 acquires face images sequentially output from the occupant camera 12.

瞳孔位置検出部24は、顔画像取得部23により順次取得された顔画像を所定時間毎に区分した複数のフレーム毎に乗員の左右の眼球を検知対象物とした特徴量算出および形状判別などの認識処理を行ない、この処理結果に基づき、眼の瞳孔あるいは虹彩の中心位置を検出する。   The pupil position detection unit 24 performs feature amount calculation and shape determination using the left and right eyeballs of the occupant as detection objects for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. Recognition processing is performed, and based on the processing result, the center position of the pupil of the eye or the iris is detected.

なお、瞳孔位置検出部24は、処理対象となるフレームに対して、1つ前のフレーム(前フレーム)において瞳孔あるいは虹彩の中心位置を検出した場合には、前フレームにおいて検出した瞳孔あるいは虹彩の中心位置の近傍における所定範囲内、つまり顔画像全体よりも小さい所定範囲(例えば、図2に示す左側範囲Lおよび右側範囲Rなど)内のみを処理対象として瞳孔あるいは虹彩の中心位置を検出する。
一方、1つ前のフレーム(前フレーム)において瞳孔あるいは虹彩の中心位置を検出していない場合には、顔画像全体を処理対象として瞳孔あるいは虹彩の中心位置を検出する。
Note that the pupil position detection unit 24 detects the pupil or iris center position in the previous frame when the pupil or iris center position is detected in the previous frame (previous frame) with respect to the frame to be processed. The center position of the pupil or iris is detected only within a predetermined range in the vicinity of the center position, that is, within a predetermined range smaller than the entire face image (for example, the left range L and the right range R shown in FIG. 2).
On the other hand, if the center position of the pupil or iris is not detected in the previous frame (previous frame), the center position of the pupil or iris is detected with the entire face image being processed.

角膜反射像検出部25は、顔画像取得部23により順次取得された顔画像を所定時間毎に区分した複数のフレーム毎に乗員の左右の眼球周辺において、光源11から照射された光の角膜表面における反射点(例えば、赤外線の反射像であるプルキニエ像の中心位置)を検出する。   The cornea reflection image detection unit 25 is a corneal surface of light emitted from the light source 11 around the left and right eyes of the occupant for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. The reflection point at (for example, the center position of the Purkinje image, which is an infrared reflection image) is detected.

目頭検出部26は、顔画像取得部23により順次取得された顔画像を所定時間毎に区分した複数のフレーム毎に乗員の左右の眼球を検知対象物とした特徴量算出および形状判別などの認識処理を行ない、この処理結果に基づき、眼の目頭の位置を検出する。   The eye detection unit 26 recognizes feature amount calculation and shape determination using the left and right eyeballs of the occupant as detection objects for each of a plurality of frames obtained by dividing the face images sequentially acquired by the face image acquisition unit 23 at predetermined time intervals. Processing is performed, and the position of the eye's eye is detected based on the processing result.

第1視線方向検知部27は、複数のフレーム毎に、瞳孔位置検出部24により検出される眼の瞳孔あるいは虹彩の中心位置と、角膜反射像検出部25により検出される角膜表面の反射点との相対距離に基づき視線方向(第1視線方向)を検知する。   The first line-of-sight direction detection unit 27 includes, for each of a plurality of frames, the eye pupil or iris center position detected by the pupil position detection unit 24, and the corneal surface reflection point detected by the corneal reflection image detection unit 25. The line-of-sight direction (first line-of-sight direction) is detected on the basis of the relative distance.

第2視線方向検知部28は、複数のフレーム毎に、瞳孔位置検出部24により検出される眼の瞳孔あるいは虹彩の中心位置と、目頭検出部26により検出される目頭の位置との相対距離に基づき視線方向(第2視線方向)を検知する。   The second line-of-sight direction detection unit 28 sets the relative distance between the center position of the eye pupil or iris detected by the pupil position detection unit 24 and the eye position detected by the eye detection unit 26 for each of a plurality of frames. Based on this, the gaze direction (second gaze direction) is detected.

比較部29は、各フレーム毎に、第1視線方向検知部27により検知された視線方向(第1視線方向)と、第2視線方向検知部28により検知された視線方向(第2視線方向)とを比較し、第1視線方向と第2視線方向との差の絶対値が第1所定値未満であるか否かを判定する。
なお、第1視線方向と第2視線方向との差は、所定方向(例えば、座席に着座した乗員の正面方向など)を基準とした各視線方向の角度の差である。
For each frame, the comparison unit 29 has a gaze direction (first gaze direction) detected by the first gaze direction detection unit 27 and a gaze direction (second gaze direction) detected by the second gaze direction detection unit 28. To determine whether or not the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is less than a first predetermined value.
Note that the difference between the first line-of-sight direction and the second line-of-sight direction is the difference in angle between the respective line-of-sight directions with reference to a predetermined direction (for example, the front direction of an occupant seated on the seat).

視線方向選択部30は、各フレーム毎に、第1視線方向検知部27および第2視線方向検知部28により各視線方向が検出された場合において、比較部29による比較結果の差の絶対値が第1所定値未満である場合には第1視線方向を車両の乗員の視線方向として選択し、比較部29による比較結果の差の絶対値が第1所定値以上である場合には第2視線方向を車両の乗員の視線方向として選択する。   The line-of-sight direction selection unit 30 calculates the absolute value of the difference between the comparison results by the comparison unit 29 when each line-of-sight direction is detected by the first line-of-sight direction detection unit 27 and the second line-of-sight direction detection unit 28 for each frame. When it is less than the first predetermined value, the first line-of-sight direction is selected as the line-of-sight direction of the vehicle occupant, and when the absolute value of the comparison result by the comparison unit 29 is greater than or equal to the first predetermined value, the second line of sight is selected. The direction is selected as the line-of-sight direction of the vehicle occupant.

また、視線方向選択部30は、各フレーム毎に、第1視線方向検知部27のみによって視線方向(第1視線方向)が検出された場合には、この第1視線方向(現フレームの第1視線方向)と、1つ前のフレーム(前フレーム)において視線方向選択部30により選択されて視線方向記憶部31に記憶された視線方向(前フレームの視線方向)との差が第2所定値未満である場合に、現フレームの第1視線方向を車両の乗員の視線方向として選択する。
一方、現フレームの第1視線方向と前フレームの視線方向との差が第2所定値以上である場合に、前フレームの視線方向を車両の乗員の視線方向として選択する。
Further, the line-of-sight direction selection unit 30 detects the first line-of-sight direction (the first line-of-sight direction) when the line-of-sight direction (first line-of-sight direction) is detected only by the first line-of-sight direction detection unit 27 for each frame. The difference between the gaze direction (gaze direction of the previous frame) selected by the gaze direction selection unit 30 and stored in the gaze direction storage unit 31 in the previous frame (previous frame) is a second predetermined value. If it is less, the first viewing direction of the current frame is selected as the viewing direction of the vehicle occupant.
On the other hand, when the difference between the first line-of-sight direction of the current frame and the line-of-sight direction of the previous frame is greater than or equal to a second predetermined value, the line-of-sight direction of the previous frame is selected as the line-of-sight direction of the vehicle occupant.

なお、現フレームの第1視線方向と前フレームの視線方向の差は、所定方向(例えば、座席に着座した乗員の正面方向など)を基準とした各視線方向の角度の差であり、例えば第2所定値は第1所定値よりも大きい値である。
また、視線方向選択部30は、各フレーム毎に、第2視線方向検知部28のみによって視線方向(第2視線方向)が検出された場合には、この第2視線方向を車両の乗員の視線方向として選択する。
The difference between the first line-of-sight direction of the current frame and the line-of-sight direction of the front frame is a difference in angle between the respective line-of-sight directions with reference to a predetermined direction (for example, the front direction of an occupant seated on the seat). The 2 predetermined value is a value larger than the first predetermined value.
Further, when the line-of-sight direction (second line-of-sight direction) is detected only by the second line-of-sight direction detection unit 28 for each frame, the line-of-sight direction selection unit 30 uses the second line-of-sight direction as the line of sight of the vehicle occupant. Select as direction.

視線方向記憶部31は、視線方向選択部30により選択された各フレーム毎の視線方向を記憶する。   The gaze direction storage unit 31 stores the gaze direction for each frame selected by the gaze direction selection unit 30.

この実施の形態による視線検出装置10は上記構成を備えており、次に、この視線検出装置10の動作について説明する。   The line-of-sight detection apparatus 10 according to this embodiment has the above-described configuration. Next, the operation of the line-of-sight detection apparatus 10 will be described.

先ず、例えば図3に示すステップS01においては、今回の処理対象であるフレーム(現フレーム)の一つ前のフレーム(前フレーム)において第1視線方向検知部27または第2視線方向検知部28によって視線方向が検知されたか否かを判定する。
この判定結果が「YES」の場合には、ステップS02に進み、このステップS02
においては、前フレームにおいて検出した瞳孔あるいは虹彩の中心位置の近傍における所定範囲内、つまり顔画像全体よりも小さい所定範囲(例えば、図2に示す左側範囲Lおよび右側範囲Rなど)内のみを瞳孔あるいは虹彩の中心位置の探索範囲に設定する。
一方、この判定結果が「NO」の場合には、ステップS03に進み、このステップS03においては、顔画像全体を瞳孔あるいは虹彩の中心位置の探索範囲に設定する。
First, for example, in step S01 shown in FIG. 3, the first line-of-sight direction detection unit 27 or the second line-of-sight direction detection unit 28 in the frame (previous frame) immediately before the frame that is the current processing target (current frame). It is determined whether the line-of-sight direction is detected.
If this determination is “YES”, the flow proceeds to step S 02, and this step S 02
In this case, the pupil is limited to a predetermined range near the center position of the pupil or iris detected in the previous frame, that is, a predetermined range smaller than the entire face image (for example, the left range L and the right range R shown in FIG. 2). Or it sets to the search range of the center position of an iris.
On the other hand, if this determination is “NO”, the flow proceeds to step S 03, where the entire face image is set as the search range for the center position of the pupil or iris.

次に、ステップS04においては、瞳孔位置検出部24により検出される眼の瞳孔あるいは虹彩の中心位置と、角膜反射像検出部25により検出される角膜表面の反射点との相対距離に基づき視線方向(第1視線方向)を検知する第1視線検知の処理を実行する。   Next, in step S04, the line-of-sight direction is based on the relative distance between the center position of the eye pupil or iris detected by the pupil position detection unit 24 and the reflection point of the corneal surface detected by the corneal reflection image detection unit 25. A first gaze detection process for detecting (first gaze direction) is executed.

次に、ステップS05においては、瞳孔位置検出部24により検出される眼の瞳孔あるいは虹彩の中心位置と、目頭検出部26により検出される目頭の位置との相対距離に基づき視線方向(第2視線方向)を検知する第2視線検知の処理を実行する。   Next, in step S05, the line-of-sight direction (second line of sight) is based on the relative distance between the center position of the pupil or iris of the eye detected by the pupil position detector 24 and the position of the eye head detected by the eye detector 26. A second line-of-sight detection process for detecting (direction) is executed.

次に、ステップS06においては、第1視線検知および第2視線検知の両方で各視線方向が不検知であったか否かを判定する。
この判定結果が「YES」の場合には、ステップS07に進み、このステップS07においては、車両の乗員の視線方向が不検知であることを示す信号を出力し、リターンに進む。
一方、この判定結果が「NO」の場合には、ステップS08に進む。
Next, in step S06, it is determined whether or not each gaze direction is undetected in both the first gaze detection and the second gaze detection.
If this determination is “YES”, the flow proceeds to step S 07, and in this step S 07, a signal indicating that the sight line direction of the vehicle occupant is not detected is output, and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S08.

そして、ステップS08においては、第1視線検知および第2視線検知の両方で各視線方向を検知したか否かを判定する。
この判定結果が「NO」の場合には、後述するステップS12に進む。
一方、この判定結果が「YES」の場合には、ステップS09に進む。
そして、ステップS09においては、第1視線方向の角度と第2視線方向の角度との差の絶対値が第1所定値未満であるか否かを判定する。
この判定結果が「YES」の場合には、ステップS10に進み、このステップS10においては、車両の乗員の視線方向として第1視線方向を採用して、リターンに進む。
一方、この判定結果が「NO」の場合には、ステップS11に進み、このステップS11においては、車両の乗員の視線方向として第2視線方向を採用して、リターンに進む。
In step S08, it is determined whether or not each gaze direction is detected by both the first gaze detection and the second gaze detection.
If this determination is “NO”, the flow proceeds to step S 12 described later.
On the other hand, if this determination is “YES”, the flow proceeds to step S 09.
In step S09, it is determined whether or not the absolute value of the difference between the angle in the first line-of-sight direction and the angle in the second line-of-sight direction is less than a first predetermined value.
If this determination is “YES”, the flow proceeds to step S 10, in which the first line-of-sight direction is adopted as the line-of-sight direction of the vehicle occupant and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S 11, in which the second line-of-sight direction is adopted as the line-of-sight direction of the vehicle occupant and the flow proceeds to return.

そして、ステップS12においては、第1視線検知で第1視線方向を検知したか否かを判定する。
この判定結果が「NO」の場合には、上述したステップS11に戻る。
一方、この判定結果が「YES」の場合には、ステップS13に進む。
そして、ステップS13においては、現フレームと前フレームとの第1視線方向の角度の差の絶対値が第2所定値未満であるか否かを判定する。
この判定結果が「YES」の場合には、ステップS14に進み、このステップS14においては、車両の乗員の視線方向として現フレームの第1視線方向を採用して、リターンに進む。
一方、この判定結果が「NO」の場合には、ステップS15に進み、このステップS15においては、車両の乗員の視線方向として前フレームの視線方向を採用して、リターンに進む。
In step S12, it is determined whether or not the first gaze direction is detected by the first gaze detection.
If this determination is “NO”, the flow returns to step S 11 described above.
On the other hand, if the determination is “YES”, the flow proceeds to step S13.
In step S13, it is determined whether or not the absolute value of the angle difference between the current frame and the previous frame in the first line-of-sight direction is less than a second predetermined value.
If this determination is “YES”, the flow proceeds to step S 14, and in this step S 14, the first gaze direction of the current frame is adopted as the gaze direction of the vehicle occupant, and the flow proceeds to return.
On the other hand, if this determination is “NO”, the flow proceeds to step S 15, in which the line-of-sight direction of the previous frame is adopted as the line-of-sight direction of the vehicle occupant, and the flow proceeds to return.

上述したように、本実施の形態による視線検出装置10によれば、瞳孔の位置および角膜反射像の位置に基づいて第1視線方向を検出する第1視線方向検知部27では、角膜反射像によらずに第2視線方向を検出する第2視線方向検知部28に比べて、視線方向の検出精度が高くなる可能性は高いが検出方向の信頼性や安定性は低くなる場合がある。
このため、第1視線方向と第2視線方向との差の絶対値が第1所定値未満になることで第2視線方向によって第1視線方向の信頼性および安定性が高いと判断することができる場合には、第1視線方向に基づき乗員の視線方向を検出することによって、視線方向の検出精度を向上させることができる。
また、第1視線方向と第2視線方向との差の絶対値が第1所定値以上になることで第1視線方向の信頼性および安定性が低いと判断することができる場合には、第2視線方向に基づき乗員の視線方向を検出することによって、視線方向の検出結果に対して所望の信頼性および安定性を確保することができる。
As described above, according to the line-of-sight detection device 10 according to the present embodiment, the first line-of-sight direction detection unit 27 that detects the first line-of-sight direction based on the position of the pupil and the position of the cornea reflection image generates the cornea reflection image. Regardless of the second gaze direction detection unit 28 that detects the second gaze direction, there is a high possibility that the gaze direction detection accuracy is high, but the reliability and stability of the detection direction may be low.
For this reason, when the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is less than the first predetermined value, it is determined that the reliability and stability of the first line-of-sight direction are high depending on the second line-of-sight direction. If possible, the detection accuracy of the line-of-sight direction can be improved by detecting the line-of-sight direction of the occupant based on the first line-of-sight direction.
In addition, when the absolute value of the difference between the first line-of-sight direction and the second line-of-sight direction is equal to or greater than the first predetermined value, it can be determined that the reliability and stability of the first line-of-sight direction are low. By detecting the sight line direction of the occupant based on the two sight line directions, it is possible to ensure desired reliability and stability with respect to the detection result of the sight line direction.

さらに、瞳孔位置検出部24は、処理対象となるフレームに対して、1つ前のフレーム(前フレーム)で瞳孔の位置を検出した場合には、この瞳孔の位置の近傍の所定範囲内を探索することから、顔画像の全体を探索する場合に比べて、処理負荷を低減することができる。   Further, when the pupil position detection unit 24 detects the position of the pupil in the previous frame (previous frame) with respect to the frame to be processed, the pupil position detection unit 24 searches within a predetermined range in the vicinity of the pupil position. Therefore, the processing load can be reduced as compared with the case where the entire face image is searched.

さらに、第2視線方向のみが検出された場合には、この第2視線方向を乗員の視線方向として採用することによって、視線方向の検出結果に対して所望の信頼性および安定性を確保することができる。
また、第1視線方向のみが検出された場合であっても、この検出結果(現フレームでの第1視線方向)と1つ前のフレームでの視線方向(前フレームでの視線方向)との差が第2所定値未満であれば、現フレームでの第1視線方向が所望の信頼性および安定性を有すると判断することができ、この現フレームでの第1視線方向を乗員の視線方向として採用することによって所望の信頼性および安定性を確保して視線方向を検出することができる。
一方、この検出結果(現フレームでの第1視線方向)と1つ前のフレームでの視線方向(前フレームでの視線方向)との差が第2所定値以上であれば、前フレームでの視線方向を乗員の視線方向として採用することによって所望の信頼性および安定性を確保して視線方向を検出することができる。
Furthermore, when only the second line-of-sight direction is detected, the second line-of-sight direction is used as the occupant's line-of-sight direction, thereby ensuring desired reliability and stability for the detection result of the line-of-sight direction. Can do.
Even when only the first line-of-sight direction is detected, the detection result (first line-of-sight direction in the current frame) and the line-of-sight direction in the previous frame (line-of-sight direction in the previous frame) If the difference is less than the second predetermined value, it can be determined that the first line-of-sight direction in the current frame has the desired reliability and stability, and the first line-of-sight direction in the current frame is determined as the line-of-sight direction of the occupant. As a result, it is possible to ensure the desired reliability and stability and detect the line-of-sight direction.
On the other hand, if the difference between the detection result (first line-of-sight direction in the current frame) and the line-of-sight direction in the previous frame (line-of-sight direction in the previous frame) is greater than or equal to the second predetermined value, By adopting the line-of-sight direction as the line-of-sight direction of the occupant, the line-of-sight direction can be detected while ensuring the desired reliability and stability.

なお、第2視線方向検知部28は、目頭の位置に限定されず、例えば目尻の位置などの目の他の位置と、瞳孔位置検出部24により検出される眼の瞳孔あるいは虹彩の中心位置との相対距離に基づき視線方向(第2視線方向)を検知してもよい。   Note that the second gaze direction detection unit 28 is not limited to the position of the head of the eye, for example, another position of the eye such as the position of the corner of the eye, and the center position of the pupil or iris of the eye detected by the pupil position detection unit 24. The line-of-sight direction (second line-of-sight direction) may be detected on the basis of the relative distance.

なお、上述したステップS15においては、車両の乗員の視線方向として前フレームの視線方向を採用するとしたが、これに限定されず、車両の乗員の視線方向が不検知であることを示す信号を出力してもよい。   In step S15 described above, the line-of-sight direction of the front frame is adopted as the line-of-sight direction of the vehicle occupant. However, the present invention is not limited to this, and a signal indicating that the line-of-sight direction of the vehicle occupant is not detected is output. May be.

10 視線検出装置
11 光源
12 乗員カメラ(撮像手段)
24 瞳孔位置検出部(位置検出手段)
25 角膜反射像検出部(位置検出手段)
27 第1視線方向検知部(第1検出手段)
28 第2視線方向検知部(第2検出手段)
29 比較部(比較手段)
30 視線方向選択部(視線方向検出手段)
DESCRIPTION OF SYMBOLS 10 Eye-gaze detection apparatus 11 Light source 12 Crew camera (imaging means)
24 Pupil position detection unit (position detection means)
25 Cornea reflection image detection unit (position detection means)
27 1st gaze direction detection part (1st detection means)
28 Second visual line direction detection unit (second detection means)
29 Comparison part (comparison means)
30 Gaze direction selection unit (Gaze direction detection means)

Claims (3)

ひとの顔を撮像して顔画像を出力する撮像手段と、前記ひとが存在する方向に向かって光を照射可能に配置された光源と、前記撮像手段から出力された前記顔画像から瞳孔の位置および前記光源から照射された前記光による角膜反射像の位置を検出して検出結果を出力する位置検出手段と、該位置検出手段から出力された前記検出結果に基づいて前記ひとの視線方向を検出して検出結果を出力する第1検出手段とを備える視線検出装置であって、
前記位置検出手段から出力された前記検出結果の前記瞳孔の位置と、前記位置検出手段から出力された前記検出結果の前記角膜反射像を含まない前記顔画像での所定部位の位置とに基づいて前記ひとの視線方向を検出して検出結果を出力する第2検出手段と、
前記第1検出手段から出力された前記検出結果と前記第2検出手段から出力された前記検出結果とを比較して前記検出結果同士の差を比較結果として出力する比較手段と、
前記比較手段から出力された前記比較結果の前記差が所定値未満である場合には前記第1検出手段から出力された前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出し、前記比較手段から出力された前記比較結果の前記差が所定値以上である場合には前記第2検出手段から出力された前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出する視線方向検出手段とを備えることを特徴とする視線検出装置。
Imaging means for imaging a human face and outputting a face image, a light source arranged so as to be able to irradiate light in a direction in which the person exists, and a position of a pupil from the face image output from the imaging means And a position detection means for detecting a position of a corneal reflection image by the light emitted from the light source and outputting a detection result, and detecting the gaze direction of the person based on the detection result output from the position detection means A line-of-sight detection device comprising first detection means for outputting a detection result.
Based on the position of the pupil of the detection result output from the position detection means and the position of a predetermined part in the face image not including the corneal reflection image of the detection result output from the position detection means. Second detection means for detecting the direction of the line of sight of the person and outputting a detection result;
A comparison unit that compares the detection result output from the first detection unit with the detection result output from the second detection unit and outputs a difference between the detection results as a comparison result;
When the difference between the comparison results output from the comparison unit is less than a predetermined value, the human gaze direction is detected based on the gaze direction of the detection results output from the first detection unit, Gaze direction detection that detects the gaze direction of the person based on the gaze direction of the detection result output from the second detection means when the difference of the comparison results output from the comparison means is greater than or equal to a predetermined value. A line-of-sight detection device.
前記位置検出手段は、前記撮像手段から順次出力された前記顔画像を所定時間毎に区分した各フレーム毎に前記瞳孔の位置を前記顔画像全体から検出すると共に、前記各フレームのうち所定のフレームよりも1つ前のフレームにおいて前記瞳孔の位置を検出した場合には、前記所定のフレームにおいて、前記1つ前のフレームにおいて検出した前記瞳孔の位置の近傍における前記顔画像全体よりも小さい所定範囲内から前記瞳孔の位置を検出することを特徴とする請求項1に記載の視線検出装置。 The position detection means detects the position of the pupil from the entire face image for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals, and a predetermined frame among the frames. When the position of the pupil is detected in the frame immediately before, the predetermined range is smaller than the entire face image in the vicinity of the pupil position detected in the previous frame in the predetermined frame. The line-of-sight detection device according to claim 1, wherein the position of the pupil is detected from within. 前記位置検出手段は、前記撮像手段から順次出力された前記顔画像を所定時間毎に区分した各フレーム毎に前記瞳孔の位置および前記角膜反射像の位置を検出して検出結果を出力しており、
前記視線方向検出手段は、前記第1検出手段および前記第2検出手段の何れか一方のみから前記検出結果が出力されたときに、
前記検出結果が前記第2検出手段から出力された場合、あるいは、前記各フレームのうち適宜のフレームにおいて前記検出結果が前記第1検出手段から出力されたときに当該適宜のフレームにおいて前記第1検出手段から出力された前記検出結果の前記視線方向と前記適宜のフレームの1つ前のフレームにおいて前記視線方向検出手段により検出された前記視線方向との差が所定値未満である場合に、前記検出結果の前記視線方向に基づき前記ひとの視線方向を検出することを特徴とする請求項1または請求項2に記載の視線検出装置。
The position detection means detects the position of the pupil and the position of the cornea reflection image for each frame obtained by dividing the face image sequentially output from the imaging means at predetermined time intervals, and outputs a detection result. ,
When the detection result is output from only one of the first detection unit and the second detection unit,
When the detection result is output from the second detection unit, or when the detection result is output from the first detection unit in an appropriate frame among the frames, the first detection is performed in the appropriate frame. The detection is performed when a difference between the line-of-sight direction of the detection result output from the means and the line-of-sight direction detected by the line-of-sight direction detection means in a frame immediately before the appropriate frame is less than a predetermined value. The gaze detection apparatus according to claim 1, wherein the gaze direction of the person is detected based on the gaze direction of the result.
JP2010200059A 2010-09-07 2010-09-07 Gaze detection device Expired - Fee Related JP5358540B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2010200059A JP5358540B2 (en) 2010-09-07 2010-09-07 Gaze detection device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2010200059A JP5358540B2 (en) 2010-09-07 2010-09-07 Gaze detection device

Publications (2)

Publication Number Publication Date
JP2012055428A true JP2012055428A (en) 2012-03-22
JP5358540B2 JP5358540B2 (en) 2013-12-04

Family

ID=46053259

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2010200059A Expired - Fee Related JP5358540B2 (en) 2010-09-07 2010-09-07 Gaze detection device

Country Status (1)

Country Link
JP (1) JP5358540B2 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150085710A (en) * 2014-01-16 2015-07-24 삼성전자주식회사 Dispaly apparatus and controlling method thereof
CN105607255A (en) * 2014-11-17 2016-05-25 精工爱普生株式会社 Head-mounted display device, method of controlling head-mounted display device, and computer program
US9760774B2 (en) 2014-08-29 2017-09-12 Alps Electric Co., Ltd. Line-of-sight detection apparatus
CN111176434A (en) * 2018-11-13 2020-05-19 本田技研工业株式会社 Gaze detection device, computer-readable storage medium, and gaze detection method
CN111208905A (en) * 2020-01-08 2020-05-29 北京未动科技有限公司 Multi-module sight tracking method and system and sight tracking equipment
JP7542563B2 (en) 2019-07-05 2024-08-30 マジック リープ, インコーポレイテッド Eye tracking latency improvement

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008212359A (en) * 2007-03-02 2008-09-18 Toyota Motor Corp Visual axis detector
JP2009073462A (en) * 2007-08-28 2009-04-09 Toyota Central R&D Labs Inc Operating state determination device and operation support device

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008212359A (en) * 2007-03-02 2008-09-18 Toyota Motor Corp Visual axis detector
JP2009073462A (en) * 2007-08-28 2009-04-09 Toyota Central R&D Labs Inc Operating state determination device and operation support device

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150085710A (en) * 2014-01-16 2015-07-24 삼성전자주식회사 Dispaly apparatus and controlling method thereof
KR102227284B1 (en) * 2014-01-16 2021-03-12 삼성전자주식회사 Dispaly apparatus and controlling method thereof
US9760774B2 (en) 2014-08-29 2017-09-12 Alps Electric Co., Ltd. Line-of-sight detection apparatus
CN105607255A (en) * 2014-11-17 2016-05-25 精工爱普生株式会社 Head-mounted display device, method of controlling head-mounted display device, and computer program
CN105607255B (en) * 2014-11-17 2020-03-10 精工爱普生株式会社 Head-mounted display device, method of controlling the same, and computer-readable storage medium
CN111176434A (en) * 2018-11-13 2020-05-19 本田技研工业株式会社 Gaze detection device, computer-readable storage medium, and gaze detection method
JP7542563B2 (en) 2019-07-05 2024-08-30 マジック リープ, インコーポレイテッド Eye tracking latency improvement
CN111208905A (en) * 2020-01-08 2020-05-29 北京未动科技有限公司 Multi-module sight tracking method and system and sight tracking equipment

Also Published As

Publication number Publication date
JP5358540B2 (en) 2013-12-04

Similar Documents

Publication Publication Date Title
JP5923180B2 (en) Biological information measuring device and input device using the same
JP5230748B2 (en) Gaze direction determination device and gaze direction determination method
JP5406328B2 (en) VEHICLE DISPLAY DEVICE, ITS CONTROL METHOD AND PROGRAM
CN108027977B (en) Sight line detection device and sight line detection method
JP6364627B2 (en) Gaze direction detection device and gaze direction detection method
JP5358540B2 (en) Gaze detection device
US10499808B2 (en) Pupil detection system, gaze detection system, pupil detection method, and pupil detection program
JP6346525B2 (en) Gaze detection device
EP3479293A1 (en) Systems and methods for performing eye gaze tracking
US20130215388A1 (en) Image processing apparatus, diagnostic support system, and image processing method
US9877649B2 (en) Photorefraction method and product
US10504237B2 (en) Method for determining a viewing direction of a person
CN111902070A (en) Reliability of left and right eye gaze tracking data
US11112602B2 (en) Method, apparatus and system for determining line of sight, and wearable eye movement device
JP2012014505A (en) Inattentive-driving determination apparatus
JP6587254B2 (en) Luminance control device, luminance control system, and luminance control method
US11266307B2 (en) Evaluation device, evaluation method, and non-transitory storage medium
JP2017126151A (en) Sight line detection device and sight line detection method
JP2017138645A (en) Sight-line detection device
US11653831B2 (en) Visual performance examination device, visual performance examination method, and computer program
KR102224209B1 (en) Apparatus and method for measuring angle of strabismus
JP2017126245A (en) Face direction detection method
JP5144412B2 (en) Vehicle object determination device
JP6269177B2 (en) Pupil detection device, gaze detection device, and pupil detection method
JP6909391B2 (en) Line-of-sight detection device and line-of-sight detection method

Legal Events

Date Code Title Description
A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20121211

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20121218

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20130208

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20130806

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20130902

R150 Certificate of patent or registration of utility model

Ref document number: 5358540

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

Free format text: JAPANESE INTERMEDIATE CODE: R150

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees