GB2170910A - Eye tracking system - Google Patents

Abstract

For providing a control signal responsive to the 'direction of look' of a person's eyes electrodes 1 and 2 are placed on each side of the eyes and/or above and below the eyes to detect horizontal and/or vertical movement of the eyeballs by means of the unequal charge distribution which occurs naturally on the eyeballs 4 and 5. Such a control signal may be used by a patient incapacitated except for eye movement to effect communication by presenting the patient with a display of various functions which are selected by viewing the selected function. In military applications it could be used for target tracking. <IMAGE>

Description

SPECIFICATION Eye-tracking system This invention relates to a method and apparatus to determine the 'direction of look' of a person's eyes, and provide a signal in response to the direction of look.

According to one aspect of the invention a method of determining the orientation of an eyeball in its socket comprises the steps of mounting two electrodes in fixed positions on or closely adjacent to the head and deriving a potential difference between the electrodes responsive to charge distribution within the eyeball, the charge distribution being movable with the eyeball so that the potential difference is indicative of the orientation of the eyeball in its socket, and providing a signal responsive to the potential difference.

The signal may be a control signal to be used, for example, by a severely disabled person, incapacitated except for eye movement. In such a case, communication or environmental control can be effected by the person viewing a display of various characters or functions which can be selected by moving the eyes to look at the appropriate character. A signal responsive to the direction of look could then be utilised for control or communication.

The invention could also be used in military application for target tracking or other purposes.

A potential difference can be detected around the eyes which changes with eye position. The cause of this potential is believed to be that when light enters the eye via the pupil and is focussed onto the retina, the retina becomes negatively charged and the front of the eye becomes positively charged. Thus a potential difference is generated between the front and rear of the eye.

In the method according to the above invention the rate of change of potential difference may be measured to distiguish deliberate eye movements and first and second values of the potenial difference may be measured when the rate of change of the potential difference respectively exceeds and falls below a threshold value, the difference between the two values representing the extent of a deliberate eye movement.

The method may further comprise the steps of calculating the average change of potential difference for eye movements between a pair of known orientations of the eyeball, recording the average change and thereby calibrating eyeball movement against change in potential difference.

A further potential difference may be derived, indicative of the orientation of the head, and calibrated against head orientation by recording the potential difference generated at two or more known head orientations.

The eyeball orientation may be combined with the head orientation to determine the 'direction of look' of the eyeball, and a control signal provided responsive to the 'direction of look'.

The method may further comprise the steps of displaying symbols at known positions on a display so that the separation between any two symbols is known, comparing the change of 'direction of look' on moving from a first to a second 'direction of look' with the known displacement between a first symbol corresponding to the first 'direction of look' and all other symbols, to determine a second symbol most nearly corresponding to the second 'direction of look'.

According to another aspect of the invention apparatus for determining the orientation of an eyeball in its socket comprises two electrodes in fixed positions on or closely adjacent to the head, means to derive a potential difference between the electrodes responsive to charge distribution within the eyeball, the charge distribution being movable with the eyeball so that the potential difference is indicative of the orientation of the eyeball in its socket, and means to provide a signal responsive to the potential difference. The electrodes may be positioned one on each side of the head in operation, the potentials acquired by the two electrodes on lateral movement of the eyes relative to the head then being sum of the potentials.The apparatus may further comprise means to measure the rate of change of the potential difference, and thresholding means to distinguish deliberate eye movements.

The apparatus may further comprise means to derive a further potential difference indicative of the head orientation, which means may comprise means to detect incidence of light from a scanning narrow light beam- on a photocell or reflector, the photocell or reflector being adapted to be attached to the head.

The apparatus may include comparison means for determining a drift error in an eyeball orientation signal corresponding to the potential difference.

One embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings of which: Figures 1A and 1B are diagrammatic plan views of a pair of eyeballs between electrodes; Figure 2 is a graph of potential difference against time showing the effect of sudden eye movement; Figure 3 is a block diagram of an electrode voltage measurement circuit; Figure 4 is a display of functions; Figure 5 shows the steps required to print the word "How?" using the display of Fig. 4.

and Figure 6 shows a circuit for estimating electrode voltage drift.

Referring to Fig. 1A, two electrodes 1, 2 are positioned one against each side of the head (not shown) with the centre of each electrode aligned approximately with the centres of the eyeballs 4, 5. Conductive electrode jelly is used to lower skin resistance and ensure a good electrical contact. Light entering the eye via the pupil 7 is focussed onto the retina 9 causing the retina to become negatively charged and the front of the eye to become positively charged. With the electrodes positioned as described and the eye looking straight ahead, positive and negative charges are induced in approximately equal quantities on each electrode, and the potential difference between the electrodes would be expected to be close to zero. There is however, a small, unbalanced, d.c. voltage due mainly to interaction between the skin and the electrodes.

As the eyes rotate, as in Fig. 1B, the charge distribution rotates with the eyeballs causing one of the electrodes, 1, to become net positively charged and the other, 2, to become net negatively charged. The potential difference E between the electrodes is believed to be proportional to the light magnitude, K and the angle of rotation from the 'straight-ahead' reference direction.

Thus, E=KO+C where C is the unbalanced d.c. level.

From experiment, K8215 to 30 ,uV The d.c. voltage C depends upon many variable factors including the exact position of the electrodes, the contact pressure, the electrode jelly, the type of electrodes used and possibly changes within the skin. The electrodes are made with a large surface area (27X25 mm) to minimise some of these effects. C is eliminated by using electrode voltage differences. Any remainder, caused by C changing between voltage readings, can be corrected by the circuit to be described below.

So, EA#EB=K(OA#6# where A, B are two different eye orientations. Thus, the change in potential difference caused by eye movement is linearly proportional to the change is eyeball orientation.

This linearity can be seen in Fig. 2 which shows a graph of electrode potential difference E against time. From t=O to t=tA the voltage drifts randomly as C varies. At t=tA a deliberate eye movement begins. As the eyeball rotates from orientation A to orientation B, the electrode voltage changes in linear proportion to the angle of rotation. The difference, E,,#EA can thus be converted directly to an angle.

The two electrodes positioned on the sides of the head detect lateral movements of the eyeball. Two further electrodes could be attached to the head one above and one below one eye to detect elevation movements. This provides a more useful system since the overall eye movement can be determined from the sum of the movements in the two places, and the user is not limited to viewing points on the same horizontal plane.

Eye movement is detected by constantly measuring the rate of change of the potential difference between the electrodes and comparing it against a threshold rate of 29 per second.

When the eyes move deliberately from one position to another, they move at a rate greater than 2"/sex. Thus the start and finish of eye movement is very easily detected from the very slow rate of change due to electrode drift.

The circuit for employing the potential difference and measuring its rate of change is shown in Fig. 3. The output from the electrodes is applied to very low noise differential amplifier 11. Low pass filter 13 gives a bandwidth of d.c. to 2 Hz to allow eye movement to be picked out from other, higher frequency, signals, present from, for example, brain and muscle actions. The filtered output is amplified by amplifier 15 with a gain of about 100, then passed through filter 17, which is a notch filter at 50 Hz to eliminate 'mains' interference. A second low pass filter 19 eliminates any frequencies above 2 Hz introduced since the first one, 13, and further amplification is provided by amplifier 21 having a gain of 10.30.

Output 27 from amplifier 21 is fed directly to a microcomputer 28. It also feeds a differentiator 23 which continuously measures the rate of change of the output signal, and comparator 25 which compares the signal rate of change with the threshold value for eye movement of 20/Sec.

When the rate increases through this threshold a signal is provided on output 26 to enable the microcomputer to sample the signal value on output 27. When the rate falls through 2"/sec the microcomputer again samples the signal value.

The voltage level on output 27 is monitored by the computer for a predetermined time, typically 0.5 to 1 second. If the level remains approximately constant during this time, the computer 28 goes ahead with its calculation of eye movement. This time delay is imposed to distinguish deliberate looks from glances as the eyes move about naturally. It has been found that, unless the eyes are deliberately focussing on a particular point, the time spent at rest at any point is generally much less than 0.5 seconds.

Until deliberate eye movement is detected at comparator 25, and a deliberate stare is registered by the computer 28, the output 27 from amplifier 21 is also applied to a drift correction loop consisting of a comparator 29 and an integrator 31. Comparator 29 compares the signal level with a predetermined optimum level, chosen to suit the amplifier 21 and the microcomputer 28. In this case the reference voltage applied to the comparator is +0.8V which is the midoperating point of the analogue to digital converter in the BBC Micro. Output from the comparator 29 is applied to integrator 31 which acts to bring the comparator output to zero. The integrator output then rises and falls with signal level. If the comparator output is not zero, the integrator output increases or decreases with time, acting to restore the input difference to amplifier 21 to the optimum value.When eye movement is detected, the circuit is switched out at switch 30 controlled by the output of comparator 25. This prevents errors which would occur if the integrator 31 attempted to correct the large and rapid voltage changes due to eye movement.

The difference between the samples output voltages at the start and finish of eye movement is calculated by the microcomputer. This difference represents a change in eyeball orientation.

These measurements suffice to determine the 'direction of look' if it is assumed that the head does not move, but it is in fact impossible to keep the head perfectly still. For high precision, or circumstances in which the head is allowed to move freely it is necessary to measure head orientation in addition to eyeball orientation. One possible method of measuring head orientation is to use a very small photodetector attached to the head. A scanning light beam is directed generally towards the head and at some point during the scan it is detected at the head. The photodetector is made small so that it only detects light within a very narrow angle which enables the point of scan to be accurately determined. The area of scan and detector are arranged so that when the head is facing the scanner directly, the detector picks up the light at the centre of the scan.As the head turns, the point of detection of the beam moves away from the centre, and the precise head orientation can be determined from this.

It is still of course necessary to keep the eyes horizontal, as this will not allow for head rotation about the horizontal axis from the back of the head to the front, although if necessary a second head orientation detector could be added to cover this rocking head movement.

An alternative method of determining head orientation would be to use a gyroscope.

Head orientation information, if it is obtained, is combined in the microcomputer with the eyeball orientation information, to give an overall indication of the 'direction of look' of a person's eyes.

Before eye or head orientation can be determined, the detectors must be calibrated. For the eye calibration, the user is required to look repeatedly between certain pairs of fixed points. The average voltage change is recorded for each pair of points, thus providing a reference 'expected' voltage change for eye movements through any angle, because the voltage changes linearly with the angle. If two pairs of electrodes are used, each pair is calibrated separately. The calibration procedure could be conducted by an assistant, or by software or simply by the user according to a preset routine. The head detector can be calibrated in a similar way.

Fig. 4 shows one possible function display 39 Fig. 4 shows one possible function display 39 for use with the invention. Rows 4 to 6 carry the standard typewriter numeric characters; rows 1 and 3 carry sumbols for various control functions relating to the characters. For example, the PRINT function would activate a printer connected to the system for letter writing, say. PROG would put an associated microcomputer into programming mode, so that computer programs could be developed. Row 2 carries various environmental control symbols, such as lighting at two different points or intercom connection to two different rooms. There are of course very many possible functions that could be provided; Fig. 4 is intended purely as one example of a display. Furthermore, where the user has movement in one finger, for example, even more possibilities are opened up.The display is shown on a VDU, but it could of course be simply a fixed sheet of paper, or points about the room could be used as reference directions. It would also be possible to use the invention without any direction guides, by simply remembering that looking to bottom left activates one function, and to top right another, and so on.

Referring to Fig. 4, each character or function is selected by the user viewing that symbol on the display. Before functions can be reliably selected however, calibration must be made as described above so that the expected voltage change is known for a change in direction of look between any two symbols, the co-ordinates of all symbols being stored in the computer.

Once calibration is complete, the user looks at a first symbol which has been designated as the starting reference point, and from there to the first desired function symbol. Fig. 5, as an example of the operation of the system, shows the steps required to print out the word "Howl". The designated starting reference point for the screen of of Fig. 4 is the centre of the square showing REPEAT. This is chosen simply because it is fairly central to view, but the designated starting point could be at any one of the stored co-ordinates.

The rate of change of electrode voltage is measured continuously by differentiator 23 of Fig 3, and as the eyes move (deliberately) from REPEAT towards SHIFT, the rate rises above the threshold of 2"/sex. This prompts the microcomputer to sample and store the voltage on output 27. This is EREFAT. If the head orientation is being recorded, a reading is now made simuitaneously with the voltage sample. The high rate of change of voltage is maintained until the eye comes to rest on the next symbol, SHIFT, when the rate falls back below 2"/sex.

The user must then stare at SHIFT for at least the predetermined minimum 'stare' time. If the voltage, which is now being measured continuously, remains approximately constant for this minimum time, the microcomputer takes the voltage reading, SHIFT, and a head orientation reading if necessary. This "stare time" condition is imposed to distinguish deliberate function selection from glances. The difference, ESHIFT~EREPEAT, represents the angle through which the eyeballs have rotated. This is combined with the calculated change in head orientation, to give an overall change in 'direction of look'.

The co-ordinates of the centre of each square are stored, so, from the calibration, the expected change in voltage between any two squares is known. The measured change is correlated against this information to determine which symbol is being viewed. The measured change is unlikely to be precisely the same as the expected change because of electrode drift during movement and also because the eyes might not focus exactly in the centre of a square, in which case the microcomputer selects the nearest symbol. The measured change is corrected by the computer to the perfect 'expected' change. The nature of the d.c. voltage, C is such that it tends to drift in the same direction for long periods of time. If the erro were allowed to accumulate, it would eventually lead the computer to a false conclusion in its estimate of the nearest function.

So the measured voltage change is corrected to the value expected for a movement from REPEAT to SHIFT making allowance of course for any head movement. The control signal corresponding to SHIFT is then provided. In this case it is to activate the upper function of each square for the next selected function only (unless LOCK is that next function). The eyes then move to PRINT. The same process takes place-the computer samples the voltage and head orientation as the eyes leave SHIFT, and when they are at rest on PRINT, after the stare delay.

The measured voltage change, E,,,,,-E,,,,,, is correlated against expected changes to find the closest, and this corrected value is used to update 0. The control program continues as shown in Fig. 5. Letters of the alphabet are addressed by a two-digit code.

A further improvement can be made to the system by making some sophisticated allowance for electrode drift. Fig. 6 shows a circuit for estimating electrode voltage drift during non eye movement periods.

The estimator is based on the theory of the steady state Kalman Filter. During non-eye movement periods the system can be adequately modelled by the following differential equations O =d (1) d =n (2) where 8 is the eye angle (voltage) d is the drift rate (voltage) n is the gaussian white noise process of spectral denity q. This noise represents any small changes in the drift rate during the estimation period. The eye angle ft can be measured during the non-eye movement period and the measured angle ftm is given by 0,,=0+y where v is a gaussian white noise process of spectral density and representing the measurement noise.

An estimator can be constructed using equations (1) and (2) by appending corrective terms proportional to the error between the predicted and measured eye angle. The estimator equations are given by U a+ kl ( m~fi) d = k2 (Q-0) where^ denotes estimates.

The steady state Kalman gains are given by k, = [2 (q/r)l ] } k2= (q/.) The gains can be calculated off line and stored in look up tables. The transfer function from input 0,, to output ft is: 0 P k1s+k2 s2+K1s+k2 and from 0,, to a is a k2s 8, s2+k1s+k2 Typical values of q and r are: q (0.001)2 r = (.o5)2 Therefore k1 =0.2 K2 =0.02 The natural frequency of the drift estimator is Vk2=0.141 rad/sec.

When the eye movement detector 43 indicates no eye movement the switch 44 is closed and the circuit acts to estimate a and 0 by integrating fA!=a d=o When an eye movement is detected, the switch 44 is opened and the first estimated angle voltage 0, is read by the computer on output 45.

When the eye movement detector indicates that eye movement has ceased, the switch 44 is closed and the second estimated eye angle voltage 02 is read by the computer. This voltage is monitored by the computer for the predetermined minimum 'stare time' before the voltage change is calculated and responded to.

The inputs a(O) and t)(0) to integrators 46 and 47 are constants of integration, added to the signal after it has been integrated in each case. ~3(0) is reinitialised by 0 after each eye movement, so that the voltage on output 45 is a cumulative total.

Claims (31)

1. A method of determining the orientation of an eyeball in its socket comprising the steps of mounting two electrodes in fixed positions on or closely adjacent to the head and deriving a potential difference between said electrodes responsive to charge distribution within the eyeball, said charge distribution being movable with the eyeball so that said potential difference is indicative of the orientation of the eyeball in its socket, and providing a signal responsive to said potential difference.

2. A method according to Claim 1 wherein the rate of change of said potential difference is measured to distinguish deliberate eye movements.

3. A method according to Claim 2 and further comprising the step of measuring first and second values of said potential difference when the rate of change of said potential difference respectively exceeds and falls below a threshold value, the difference between said values representing the extent of a deliberate eye movement.

4. A method according to any preceding claim and further comprising the steps of calculating the average change of potential difference for eye movements between a pair of known orientations of the eyeball, recording said average change and thereby calibrating eyeball movement against change in potential difference.

5. A method according to any preceding claim and further comprising the step of deriving a further potential difference indicative of the orientation of the head.

6. A method according to Claim 5 and further comprising the step of calibrating said further potential difference against head orientation by recording the potential difference generated at two or more known head orientations.

7. A method according to Claim 6 wherein said eyeball orientation is combined with said head orientation to determine the 'direction of look' of said eyeball.

8. A method according to Claim 7 and further comprising the step of providing a control signal responsive to said 'direction of look'.

9. A method according to Claim 8 wherein said control signal is withheld until said 'direction of look' has been maintained for a predetermined time.

10. A method according to Claim 7 and further comprising the steps of providing an initial reference direction, calculating the changes of orientation of the eyeball and of the head for each successive change in 'direction of look' and calculating said 'direction of look' from said initial reference direction and the sum of all of said changes of orientation.

11. A method according to Claim 10, further comprising the steps of displaying symbols at known positions on a display so that the separation between any two symbols is known, comparing said change of 'direction of look' on moving from a first to a second 'direction of look' with the# known displacement between a first symbol corresponding to said first 'direction of look' and all other symbols, to determine a second symbol, most nearly corresponding to said second 'direction of look'.

12. A method according to Claim 11, further comprising the step of correcting the measured change of potential difference arising from said change of 'direction of look' to the value corresponding to the known displacement between said first and said second symbols.

13. A method according to any of Claims 7 to 12 wherein drift in potential difference between said electrodes is estimated during periods of no deliberate eye or head movement and the signals indicative of orientation corrected accordingly.

14. A method according to any preceding claim wherein two pairs of electrodes are mounted on or closely adjacent to the head, the potential difference between the electrodes of one of said pairs being indicative of movement of the eyeball in its socket in a first plane and the potential difference between the electrodes of the other of said pairs being indicative of movement of the eyeball in its socket in a second, transverse plane.

15. A method according to Claim 14 wherein said first plane is an elevation plane and said second plane is an azimuthal plane.

16. Apparatus for determining the orientation of an eyeball in its socket comprising two electrodes in fixed positions on or closely adjacent to the head, means to derive a potential difference between said electrodes responsive to charge distribution within the eyeball, said charge distribution being movable with the eyeball so that said potential difference is indicative of the orientation of the eyeball in its socket, and means to provide a signal responsive to said potential difference.

17. Apparatus according to Claim 16 wherein said electrodes are positioned one on each side of the head in operation, the potentials acquired by the two electrodes on lateral movement of the eyes relative to the head then being the sum of the potentials.

18. Apparatus as claimed in Claim 17, further comprising means to measure the rate of change of said potential difference, and threshoiding means to distinguish deliberate eye movements.

19. Apparatus as claimed in Claim 18, further comprising means to measure first and second values of said potential difference when the rate of change of said potential difference respectively exceeds and falls below a threshold value, rates of change of said potential difference in excess of said threshold value indicating deliberate eye movement.

20. Apparatus according to any of Claims 16 to 19 further comprising means for calculating the average change in potential difference for eye movements between a pair of known orientations of the eyeball, and means for storing said average change.

21. Apparatus according to any of Claims 16 to 20 further comprising means to derive a further potential difference indicative of the head orientation.

22. Apparatus according to Claim 21 wherein said means to derive a further potential difference comprises means to detect incidence of light from a scanning narrow light beam on a photocell or reflector, said photocell or reflector being adapted to be attached to the head.

23. Apparatus according to Claim 22, further comprising means to combine said eyeball orientation with said head orientation, and thereby to determine the 'direction of look' of the said eyeball.

24. Apparatus according to Claim 23, further comprising means to provide a control signal responsive to said direction of look.

25. Apparatus according to any of Claims 16 to 24 including comparison means for determining a drift error in an eyeball orientation signal corresponding to said potential difference, integrating means having a long time constant compared to the period of deliberate eye movement for integrating said drift error, the integrated error period being subtracted from the eyeball orientation signal to suppress said drift error.

26. Apparatus according to any of Claims 16 to 24, further comprising means for estimating the drift in potential difference between said electrodes during periods of no deliberate eye movements and means to correct accordingly the signals indicative of orientation.

27. Apparatus according to Claim 26 wherein said means for estimating said drift and said means to correct said signals comprise a Kalman Filter circuit.

28. Apparatus according to any of Claims 16-27 including low pass filter means to which said eyeball orientation signal is applied to suppress variations arising from extraneous body sources.

29. Apparatus according to Claim 28, including band stop filter means to which said eyeball orientation signal is applied to suppress variations arising from supply system interference.

30. Apparatus according to any of Claims 16 to 18, further comprising two electrodes positioned one above and one below the eye, the potential difference across said electrodes being indicative of the elevation of said eyeball.

31. Apparatus substantially as hereinbefore described with reference to the accompanying drawings.