GB2236874A - Control system. - Google Patents

Abstract

A control system primarily for patients having limited mobility and limb movement, and comprising a computer controlled display of menu sequences each showing a sequence of utility items and selectable by a cursor. Control of the cursor and activation of the utility item so selected is effected by the patient by means of a pressure sensor (1), either mouth or finger operated or both, in conjunction with an eye direction sensor (11), the two sharing the selection and activation processes at the will of the patient. The patient's sensors (1, 11) are linked to the computer by an I. R. link so that there is no fear of electrocution and also to avoid the inconvenience and danger of trailing cables.

Description

Control System This invention relates to a control system providing an interface between a human being and a range of machine and electronic devices, that is, a form of cybernetic system, and particularly to such a system for use by a human operator whose activities are restricted either by personal incapacity or by the demands of the environment in which he is operating. Typical examples are paraplegic or quadriplegic people and persons having restricted mobility due to limited confinement who require the use of all or most of their limbs (including head-movement) for other purposes, eg operators of bathyspheres, mini-submarines, space capsules, aircraft cockpits and simulators related to these systems.

Previous arrangements for use in circumstances of the above kind have included electrode devices for attachment to the operator's head to sense charge variations on the eyeball. The resulting electrical signals may then be coded, by their duration, rate of change etc, to indicate requirements. While such arrangements may be satisfactory in specific circumstances they are of limited application.

Dependence on the eyes alone for this control function can become irksome, especially when the operator is an invalid. A further disadvantage of known systems of this kind is the need for trailing connections between the operator's 'head set' and the computer.

Such connections can constitute an actual nuisance and, perhaps more important, a psychological stress, in that there is a direct connection between the operator's head and a computer system which is, or appears to be, fed from 'the mains An object of the present invention is therefore to provide a control system of the above type which significantly improves the operator's control capability while still permitting operation according to selected individual modes.

According to the present invention, a control system for use by a human operator comprises computer means adapted to control a range of utility items, display means controlled by the computer means and adapted to display at least one menu of the utility items, pressure sensor means adapted to be operated by the operator, eye-direction sensor means adapted to be fitted to an operator's head the pressure sensor means and the eye-direction sensor means being coupled to the computer means by an electro-magnetic link to control selection and/or activation of the utility items.

The electromagnetic link is preferably an infra-red link.

The display means may be adapted to display two or more stages of menu, a menu of the second or a subsequent stage constituting a the utility item of a menu of the preceding stage.

The pressure sensor may be an air pressure sensor responsive to pressures above and below atmospheric produced by blowing and sucking. A cursor may be movable over displayed menu items by the sucking or the blowing process and activation of a selected menu item effected by the complementary process.

The eye-direction sensor means and the pressure sensor means may control, one, the selection of menu items, and the other, the activation of selected menu items.

The computer means may be adapted to scan menu items automatically until an item is selected by operation of the pressure sensor means.

The control system may be adapted to operate in an eye control mode, the eye-direction sensor means controlling the selection of menu items, and the system including means responsive to the dwell time of the eye direction to activate any menu item subjected to the same eye direction for a predetermined period.

The control system may further include means responsive to the closure of one but not two eyes to switch between operational modes, and particularly from a standby to an active mode.

A control system for use by a human operator whose mobility is restricted by environment or personal incapacity will now be described, by way of example, with reference to the accompanying drawings, of which: Figure 1 is a block diagram of a preferred arrangement providing a choice between, or combination of eye direction sensor and pressure sensor; Figure 2 is a diagram of an abbreviated version of the arrangement of Figure 1 showing a pressure sensor and pressure transducer in some detail; Figure 3 is a control mode menu to be displayed to the operator; Figure 4 is a main menu following automatically from Figure 3; Figure 5(a) is a particular sub-menu of a typewriter mode; Figure 5(b) is a sub-menu expansion of part of the menu of Figure 5(a); Figure 6 is a sub-menu resulting from selection of radio equipment.

and Figures 7 and 8 show, respectively, transmitting and receiving units of an I.R. link.

Referring to Figure 1 the system comprises an air pressure sensor unit 1, shown in detail in Figure 2, coupled to.a computer 3 by way of an infra-red link 5. The computer controls a display 7 and also a number of utility devices 9 which may include radio, TV, recorder, fan, lights, telephone, motor/servo units, robots, wheelchairs and many more.

The computer is programmed to associate the utility devices 9 with items on a menu displayed by the display system 7. A cursor on the display is moved to select a particular item which is associated either with further displays in sub-menus or with a particular utility device 9 directly. An activation signal may be provided by the pressure sensor unit 1 to the computer 3 to activate the selected item, ie the selected utility device.

An eye-direction sensor system 11 is incorporated to give additional control and flexibility to the overall system. This eye-sensor system comprises a pair of electrodes 13 which are fitted to the operator's head in close contact with his temples. The charge distribution around a person's eyes is affected by the 'direction-of-look'. Suitably placed electrodes (13) can therefore be employed to sense this charge disturbance and hence the direction-of-look.

The potential difference arising can be used to control the cursor of the display system 7 by way of the computer 3 to perform the selection and, as will be seen, the activation, of menu items.

In an extreme case where the pressure sensor cannot be used, the activation of a selected menu item can be achieved by responding to the dwell time of the eyes on the selected item. A sufficiently long dwell time, in excess of a predetermined threshold, produces an activation signal.

More often however, the eye sensor would be used for only one of the two functions, selection and activation and most conveniently that one would be the selection function. An advantage of the system is however the flexibility prov ided by the ability to choose either eye sensor or pressure sensor or a combination, for selection and activation.

Referring now to Figure 2, the pressure sensor unit 1 of Figure 1 comprises essentially a suck/blow mouthpiece 15 and a pressure transducer 17 which converts the pressure signals to electrical signals.

A manually operable valve 19 enables the mouthpiece to be isolated from the transducer 17.

A further pressure sensor 21, which may be one of a series connected to a common closed tube 23, permits a pressure signal to be sent to the pressure transducer by means of, say, finger contact on an elastic surface. Operation of such sensors does of course require the suck/blow mouthpiece 15 to be isolated by the valve 19. The suck/blow mouthpiece and the tactile sensor 21 are shown associated with parallel branch tubes. They could, of course, be fitted to the same tube.

The pressure transducer 17 responds to both positive and negative pressure, that is, pressure above and below atmospheric.

Positive pressure is sensed by a normally open contact 25 coupled to a diaphragm, not shown, and negative pressure is sensed by a normally open contact 27 coupled to a diaphragm also not shown.

The two contacts provide three output lines: a common line 29 connected to both 'wiper' contacts; and two lines 31 and 33 connected to the respective other contacts. These three lines are connected to control an infra red transmitter 36 linked tyo a remote infra red receiver 37 as in Figure 1, to control the selection and activation of the displayed menu items by the computer. As shown, selection is achieved by the positive pressure switch 25, and activation by the negative pressure switch 27, ie 'blow' for selection and 'suck' for activation. At the wish of the operator, this arrangement can be reversed by a reversing switch 35 which interchanges the signals on the lines 31 and 33.

The use of an I.R. link has the advantage of making the sensor systems self-contained and local to the operator. There is no wire connection to any equipment which is fed from 'the mains'. The I.R.

receiver 37, the computer 3, display 7 and interface with the utility devices 9 are all relatively remote from the operator and he can feel secure from any high voltages that they might exhibit.

There is also the advantage that no trailing wires, cables etc, extend from the operator to the control equipment to constitute an accident hazard.

As described above, the selection of a menu item was achieved by steering the cursor horizontally either by picking off a voltage from the eye direction sensor or by control via the positive pressure (blow) signal. An alternative method that is available in accordance with the invention is for the computer to control an automatic scan. Since no vertical control is required (manually) the scan can be two dimensional and cover a matrix or array of menu items. The operator then merely has to activate the system when the required item is 'lit upon'. The scan may be of the same form as the conventional TV scan but with only 6 to 10 (say) lines per frame. The scan is then cyclic and may be operable at two speeds.

Figure 3 shows a suitable control mode menu for setting up the mode of operation according to the operator's capabilities or preference. The menu comprises the following control modes: (1) Eyescan.

This requires the eye-direction sensor, which is calibrated to the individual operator, and a mechanism such as the pressure sensor described above for activation of the menu item selected by the eye-sensor. Other mechanisms, such as a sound activation unit or a joystick, may be incorporated. The essential operation is however, selection by eye direction and activation by other mechanism.

(2) Autoscan fast and slow.

This has been described above, the menu items being scanned automatically and selection and activation effected by operation of the pressure mechanism at the appropriate time. There are two scan rates available.

(3) 'Vacuscan'.

Here the selection and activation is performed manually, both by the same mechanism, eg the blow/suck pressure device may be used to select by blowing and to activate by sucking. In the selection process each 'blow' causes the cursor to move along the one-dimensional array of menu items in predetermined steps.

(4) 'Eye pause'.

This mode involves only the eye-direction sensor.

Selection is performed as in (1) above but activation is also achieved by means of the eye-direction sensor. Thus, on selecting a particular menu item by looking directly at the item, the operator then holds the position for a predetermined period, eg three seconds. A timer is then triggered and the selected item activated.

The eye-direction sensor means is sensitive both to the 'direction-of-look' and to the open/closed state of the eye, the detected voltage levels corresponding distinctively to these conditions.A control signal can thus be provided by blinking the eyes. However, to avoid ambiguity arising from involuntary blinking, an exclusive-OR function is included to detect only single-eye blinking, ie a 'wink'. A minimum wink duration is also imposed to ensure that the movement is deliberate.

The control mode may be arranged to default to a particular one of the modes whon in Figure 3 either to suit the particular operator or, in general, the least demanding mode, ie the Eye-pause mode.

The operator, initially presented with the Figure 3 menu, may then keep the eye-pause mode or, using that mode select one of the others.

Each of the control modes when selected and activated is followed by a menu such as that of Figure 4. The three menu items shown are TYP for 'typewriter mode'; BYW for 'standby mode'; and DEV for 'device mode'.

Activation of the typewriter mode causes a menu such as shown in Figurte 5(a) to be displayed, this being a text-writing menu.

The items consist of the alphabet, various editing items, and an outlet to the next menu, 'page 2'. The 30 items are arranged in 6 columns and the cursor 39 can be controlled to move horizontally by blowing on the pressure sensor mouthpiece. A particular column is selected by positioning the cursor and, assuming operation in the eye-scan mode, this selection activated by sucking on the mouthpiece. The selected column is then displayed in a horizontal row in the menu. A further cursor selection and activation is performed so selecting a particular item from the row. A sequence of letters can thus be called up to form a piece of text which occupies the 1text workspaceZ area. This may then be printed out in a hard copy by selection and activation of the "print" item in column 6 of the original menu.

Figure 5(b) shows the sub menu for a selection of the sixth column of Figure 5(a) and shows the cursor positioned under page 2". This selection is activated as before by sucking on the mouthpiece to give the menu denoted PA2 (page 2) which may be related to the text writing function or may be an exit to a previous or other menu.

The operator may require a rest in the middle of some (particularly typewriting) operation. The BYE (standby) mode of Figure 4 provides this function by effectively storing the current state of an operation until such time as the operator wishes to come back to it. When in the standby mode and also in the eye-pause control mode, exit from the standby mode is effected by a wink, which returns the operation to the state immediatgely prior to the rest period.

The third operational mode available from the menu of Figure 4 is the device control mode (DEV). Selection of this mode produces a menu comprising a list of devices that can be remotely controlled by the computer, eg radio, telephone, door locks etc. These items are selected and activated as required. One further menu available as an adjunct to, say, the radio facility, is shown in Figure 6. This comprises a linear scale of frequency which can be selected for tuning a radio in either an analogue or digital manner. The selected frequency is displayed.

Each menu available has an exit, which is either automatic, on a time-out, or is selectable as one menu item to enable the operator to revert to the main menu, to proceed to an associated menu or perhaps to close down.

Figures 7 and 8 show respectively the transmitting and receiving units of the infra red link.

The transmitter, local to the operator, is coupled in this instance to a joystick device 41 which produces left/right and up/down output signals according to the direction of hand pressure applied. The resulting signals are encoded by an integrated circuit unit 43 and arranged to pulse a pair of I.R. light-emitting-diodes 45 accordingly.

Figure 8 shows the corresponding circuitry of the receiver unit local to the computer and fed by a photo-sensitive diode 47.

The use of ajoystick in this embodiment does simplify the selection of (particularly textual) items from a rectangular array, removing one stage of selection. It may well, however, be beyond the capabilities of certain handicapped operators.

With regard to the application of the described system to hospital patients, whatever the injuries, disease, disabilities, confinement conditions or requirements for the use of limbs, in by far the largest number of cases, movement of the direction of look of the eyes is the last ability to be unavailable. Hence the system utilises the measurement of direction of look of the eyes in its main mode. However, it has been found that the effort of concentration needed if this is the only property that is used is too great for patients to use for usefully long and frequent periods and it has been found that this is due to the relatively short period of steady visual concentration that will be tolerated by a patient in the condition postulated.

Hence the system incorporates the 'visual-only' mode as a back-up providing communication and control for the rare cases where the eyes are the only available movement that the patient can control. There are also rare cases, eg eye damaged patients, in which eye movement is not adequately available but some other movement is. A minimum movement, minimum pressure sensor as described above is used to control an auto-scan of the display and select the menu item. In this way, the rare extreme cases are catered for, even if the best in communication and control that can be obtained for a patient in that condition is limited.

The great majority of cases where eye control is in use will utilise the main mode above which enables, without significant patient strain, almost continuous and fully effective operation in both communication and control as often as the patient or medical authority require.

This mode uses both the eye sensor and the pressure sensor in such a way that the scan of the display is carried out by the eye direction-of-look measurement and the activation of selected items from the scan is carried out by the minimum-movement-minimum-pressure sensor. In this way, the eyes are given continual rests from concentration when not scanning the display prior to selection or after selection. It is found a patient's concentration fades if eye control is attempted continuously after too short a period in most cases. Several orders better usage can be obtained if rests and alternative control procedures are provided both to eye control and to control by the force/pressure switch, and by utilising both at once and also providing autoscan. The main mode is based on this mix of techniques.

In all modes of the system, progressive menu driving, involving main menu, sub-menus and sub-sub-menus etc, each with clear 'repeat to previous level' indication and simple clarity throughout is an inherent requirement and provision in the preferred system. If not provided, the complications to both patient and medical authority are found to militate against actual use of the equipment. Wide spacing is used between display items. These combinations allow elimination of the need to monitor head movement in this proposed system and allows the use of smaller, standard sensor designs, eg ECG sensors, which the patient will tolerate much more readily. In large-sensor designs, it has been found the patient tolerates them not at all.

Claims (11)

1. A control system for use by a human operator and comprising computer means adapted to control a range of utility items, display means controlled by said computer means and adapted to display at least one menu of said utility items, pressure sensor means adapted to be operated by the operator, eye-direction sensor means adapted to be fitted to an operator's head said pressure sensor means and said eye-direction sensor means being coupled to said computer means by an electro-magnetic link to control selection and/or activation of the utility items.

2. A control system according to Claim 1, wherein said electromagnetic link is an infra-red link.

3. A control system according to Claim 1 or Claim 2, wherein said display means is adapted to display two or more stages of menu, a menu of the second or a subsequent stage constituting a said utility item of a menu of the preceding stage.

4. A control system according to any preceding claim, wherein said pressure sensor is an air pressure sensor responsive to pressures above and below atmospheric produced by blowing and sucking.

5. A control system according to Claim 4, wherein a cursor is movable over displayed menu items by the sucking or the blowing process and activation of a selected menu item is effected by the complementary process.

6. A control system according to any of Claims 1 to 4, wherein said eye-direction sensor means and said pressure sensor means control, one, the selection of menu items, and the other, the activation of selected menu items.

7. A control system according to any of Claims 1 to 4, wherein said computer means is adapted to scan menu items automatically until an item is selected by operation of said pressure sensor means.

8. A control system according to Claim 1 adapted to operate in an eye control mode, said eye-direction sensor means controlling the selection of menu items, and the system including means responsive to the dwell time of the eye direction to activate any menu item subjected to the same eye direction for a predetermined period.

9. A control system according to Claim 8, further including means responsive to the closure of one but not two eyes to switch between operational modes.

10. A control system according to Claim 9, wherein closure of one but not two eyes effects a switch from a standby to an active mode.

11. A control system substantially as hereinbefore described with reference to any of the embodiments described herein.