WO1986001963A1

WO1986001963A1 - Improvements relating to an apparatus and method related to control through eye gaze direction

Info

Publication number: WO1986001963A1
Application number: PCT/AU1985/000232
Authority: WO
Inventors: Andrew Richard Downing
Original assignee: The University Of Adelaide
Priority date: 1984-09-24
Filing date: 1985-09-24
Publication date: 1986-03-27
Also published as: GB2177276A; GB8612298D0; JPS62500493A

Abstract

An eye gaze direction control arrangement using a light source (4) and a detector (5) giving the direction of eye gaze by detecting the positon on a user's eye (3) of the reflection of the light source and using such information to locate a cursor position on a video screen (2) and providing for a recalibration initiation achievable by a user to ensure consistent alignment of such a cursor with actual viewing direction. The calibration may be initiated by the user (1) providing multiple interruptions of the incident light within a predetermined time period (e.g. eye closures). The use (1) is then required to attempt to direct his gaze at various predetermined locations (7, 8, 9, 10) from which an average provides the calibration.

Description

IMPROVEMENTS RELATING TO AN APPARATUS AND METHOD RELATED TO CONTROL THROUGH EYE GAZE DIRECTION

This invention relates to a method and apparatus associated with control through direction of eye gaze.

The use of the direction in which a person is looking to effect control of some operation while not new of itself has hitherto incurred some difficulties.

In order to ensure that there is an adequate correlation between the direction that a person is looking and a target location, it has been found that an act effecting a calibration provides significant advantage.

As a major application of this method and apparatus will be to assist handicapped persons who may have difficulty keeping a head still, or even activate through limb pressure a controlling switch, it is a problem that this invention addresses to provide reliable means by which reasonable alignment can be maintained.

The problem is to interpret whether a user has a temporary difficulty such as for instance a tremor, or whether a major shift to relative positions has occurred so that original alignment characteristics no longer apply.

An object of this invention accordingly is to provide means which will maintain through even difficult situations, good correspondence between a viewing direction of a user and a screen based cursor. According to a preferred arrangement, there is a source of radiant energy providing a beam of light which when directed into a person ' s eye will result in a reflection and the position of this reflection with respect to the eye is responsive to the direction of gaze of that person.

If a beam of radiant energy is shone into a person's eye, it is possible to identify the reflection of that radiant energy from an appropriate portion of the eye and by knowing the position and direction of the beam of radiant energy, it then becomes possible by determining the location of intersection onto a target which has a known relationship with respect to the source of the beam, some assumptions as to the gaze direction of the eye.

Unfortunately, however, unless a person's head is fully restrained so as to maintain a certain selected eye position and the eye itself with respect to different people has idential characteristics, any assumptions made with respect to the eye direction have to be very approximate indeed.

The purpose of this invention according to its broadest concept is to provide a technique for effecting a calibration procedure so as to provide for any proposed user and a system either a higher accuracy than has hitherto been available or a method to achieve such accuracy which has advantage over any system that has hitherto been used. In the application of the invention, especially in the preferred embodiment, the techniques involve both computers, hardware items of appropriate type, and software control of the hardware items.

In broader references to the invention it is accordingly understood that such techniques can be achieved either by software controlling hardware, or by hardware in which the techniques are integral with respect to the hardware items.

Accordingly the invention can be said to reside in an eye gaze direction control arrangement including ¬

(1) a light source,

(2) a detector adapted to detect the position of the reflection of the light source on the user's eye,

(3) and means responsive to such detected position and adapted to control the position of a cursor on a screen and maintain this in an approximate alignment with a viewing direction of the user, said means being further adapted to effect a recalibration attempt sequence for calibration of the cursor viewing direction alignment upon an activation signal being effected by the user.

It has been found according to a preferred technique that such an activation signal as can be effected by the user can include interrupting the reflected light pathway a selected number of times within a selected time period. This can simply mean the closing of the eyelids twice within a two-second period which would not be a normal closure and opening rate, but is something that a user can easily and quickly achieve even in most extreme circumstances.

A turning away of the eye so that reflection communication is entirely lost, can also be used to achieve this result.

For more able users, any limb movement can be used either to intersect a communicating beam or in another instance such movement can be used to activate a switch which can be interrogated from time to time in any controlling programme.

When such a recalibration attempt sequence has been activated, it is a problem that a user may not momentarily be looking at the first calibration viewing target location.

Assistance can be achieved by using a second calibration viewing target location and then comparing correction factors that need to be applied in the case of one location as compared to the second, and if there is not a sufficiently close correlation, proceed to a further calibration viewing target location and comparing again a correction factor with previous results.

Accordingly it may be .preferred that recalibration is effected by a first calibration viewing target location to be caused to be noticeable by a user, and causing information to be stored identifying the angular direction then detected of a reflection of the radiant beam from the eye when it is to be expected to be viewing the calibration viewing target location then causing a second calibration viewing target location to be noticeable by the user, at a position different from the first said calibration position and such as to cause the eye of a user to assume a different angular relationship with respect to the detection equipment and causing information identifying such second angular position to be stored and then effecting a correction factor to be used for maintaining alignment of the position of the cursor on the screen with a viewing direction of the user.

There is advantage if the calibration viewing targets are located in a non axial alignment such as along a horizontal axis, or in a further case along a vertical axis.

In this way it is easier to compare results in which an x, y co-ordinate location should with a further calibration viewing target location on a common horizontal axis, have a very similar x co-ordinate reading, and likewise with locations on a vertical axis in which the co-ordinates will have a common or very close y co-ordinate reading.

Some people have difficulty maintaining a consistent eye direction. This can be because of an essential instability in the eye or it may be a body tremor or it may be because of difficulties resulting from a disease such as, Parkinsons disease. While an accurate correlation between eye direction and cursor position on a screen is desirable, a greater tolerance of position can be accommodated with still the apparatus being useful, and accordingly for each recalibration process there is proposed that the said means are adapted to compare the correction factor calculated for the first of the target locations with a correction factor calculated for the second target locations and if the difference between the two is larger than a selected allowable difference, provide for a further target location to be offered to the user and continue the calculation of correction factor and comparison until at least two target locations provide a correction factor within the selected range.

However, if the target locations needed to be offered exceeds a selected number, then there will be provided an enlargement of the range of acceptable difference in correction factor and a continuing offering of target locations.

The method of this invention can be said to reside in the method of determining eye gaze direction control which includes the steps of directing a light source through a user's eye and detecting such reflection with means to enable the viewing direction of the eye to then be determinable subject to calibration, and providing a calibration of a cursor viewing direction alignment upon an activating signal being effected by the user. Reference has not been made thus far to the specific type of hardware that is being used as the target for viewing and while a video display unit can be used, at least according to the broader concept of this invention such calibration locations can be located in any environment which is useful for the purpose.

The invention will be better understood when referred to a preferred embodiment which shall be described with the assistance of a drawing in which FIG. 1 is a schematic drawing showing a user arranged with a viewing screen and with a target light being directed through the eye of the user and being detected by a video type camera.

Accordingly, the user 1 is positioned facing a video screen 2 with eyes 3 in approximate horizontal alignment.

A small target light 4 radiates light onto the eyes 3 so that the position of the reflection of this light from the eyes is detectable within the video camera 5.

Such an arrangement is a known arrangement.

However, to provide for consistent alignment of the apparent direction of sight of the eyes 3 of the user 1, there is separately provided software within a computer which acting on previously provided factors such as the relative position and direction of the light source 4 and the camera 5 and the position of the viewing screen 2, a correction factor which will modify each received target light reflection and cause a cursor to be aligned on the video screen 2 in a perceived alignment with the viewing direction of the eye 3.

However, substantial difficulties in the location of the head 6 of the user 1 can alter the relative angular relationships of the light 4, the eye 3 and the screen 2.

Such a substantial realignment is best then followed by altering the correction factor so that viewing direction once again can be brought into alignment with a cursor and therefore the location of interest at any time on the viewing video screen 2.

Accordingly, a calibration technique is provided in which a first calibration viewing target location 7 is displayed on the video screen 2 this comprising a cross which is rotated about its centre fast enough to be noticeable but not too fast as to be indistinguishable. Such a rotation speed can be two rotations each second. The programme provides for the first calibration viewing target location 7 to be maintained for a sufficient period of time for calculation of a new correction factor, and also to ensure consistent viewing direction during a first calibration period of time.

Such a period for the first location 7 can be three seconds after which a second location at 8 is then used, after which a third location 9 is used, after which a fourth location 10 is used.

For each of the further locations, a further correction factor is calculated, such a correction factor being a discernible difference between a first standard x and y co-ordinate position of a cursor and the difference that would appear to be necessary to bring the cursor to a new position being the actual viewing impact position of the eye of the user 3.

Subsequent to a second viewing target location being proposed at 8, and this being at an inclined relative relationship to the position of target 7, there will then be two correction factors available,

The programme is adapted to compare the respective correction factors and if necessary compare any difference between the two correction factors to a preselected maximum tolerance.

By using two targets which have an inclined relationship with respect to both x and y axes, a single correction factor can then be proposed taking into account both x and y deviations and also deviations in inclination.

A further check is then available by performing two further calculations on the new locations

9 and 10 and obtaining a proposed correction factor for once again both x and y axes and orientation, and then comparing such correction factor with that obtained with targets 7 and 8.

If these are within a selected tolerance, preferably the average of the two correction factors can be calculated and placed in memory for then afterwards continuing application to each of the incoming detected reflected eye angular relationships.

As stated previously, however, it is not always clear that a user has been looking precisely at the target location or within an acceptable range for such a target location, and accordingly a further check is made comparing the y axis deviation between perceived eye angles for target locations 7 and 9 which are indeed on a common horizontal axis and 8 and 10 which are also on a common horizontal axis, and likewise the tolerance of x axis alignment for target locations 7 and 1 in one instance and 9 and 8 in the other.

It is not always easy for a user 1 to call up a recalibration attempt sequence, and accordingly there is provided that if within any two seconds of time while the apparatus is in use and the eye of the user is having a target light reflection detected during other than a recalibration attempt sequence, the path of light is intersected twice for a period which lies within the range of 0.1 to 0.5 of a second on each occasion, then this will activate a recalibration sequence in which the apparatus will revert from whatever other task it is operating to commence showing on the screen a recalibration target location and continuing with the sequence as has been previously described.

The cutting off of reflection can be achieved in any of many different ways, but according to this embodiment this is achieved by the eyelids of the user being closed within the time constraint of two times within a two-second period.

The apparatus will not be able to readily interpret whether the shut-off of reflection is by way of an eyelid or by way of a full turning of the head away from the viewing apparatus, but of course the result can be the same provided the same time restraint is achieved. Likewise, a separately activated switch can be operated and can be discernible to call for a recalibration attempt sequence in a further embodiment.

In the case of some users, it will be difficult for them to ensure a sufficiently consistent eye direction for a sufficient period of time to enable adequate recalibration at any instance.

The apparatus is intended to be of value to even the most highly handicapped persons and it is well understood that in some instances accuracy of alignment of eye direction is less important than continuing ability to control the position of a cursor and activities controlled by such position, even though such control within an adequate correction factor might be quite approximate.

In other words, it could be that a person using the apparatus could be placed at risk if at least some control is not available even if generally inaccurate.

Accordingly, there is proposed that before a recalibration attempt, a first comparison of correction factors is made and compared to a first selected tolerance.

If the comparison is not within the selected tolerance, a further two inclined orientated target viewing locations will be provided and once again such correction factors as are calculated compared with an immediately preceding calculated correction factor. Such sequence will be repeated four times after which the tolerance allowable will be enlarged by a preselected factor and a further recalibration attempt sequence will be effected.

Such enlargement upon a selected number of unsuccessful recalibration attempt sequences will be continued until at least the best possible correction factor in the circumstances has been achieved.

This then describes the embodiments presently put into practice.

The appartus and techniques are applied by computer programmes.

The techniques of computer programming are common place and the embodiment can be effected by appropriate programming of any appropriate computer with sufficient memory and with appropriate means to interpret results from a video camera, and control the positioning of a cursor and other illustrations on a video display screen.

While reference has been made to an activation signal for a recalibration attempt sequence being achievable by the user comprising breaking the beam, a further signal can include detection of a selected eye movement of the user, or sequence of movements.

If such a sequence is not that which is likely to occur in normal usage, such activation can then be voluntarily effected. In a preferred instance the movement of the eye to follow from a first target to a second target, which is horizontally coincident with the first target but spaced well apart from this, and then back again and have the detection for recalibration attempt sequence effected upon this movement being repeated at least three times within a three-second period can be appropriate.

Obviously alternate sequences intended not to duplicate a sequence ordinarily occurring can be used.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. An eye gaze direction control arrangement including -

(1) a light source,

(2) a detector adapted to detect the position of light from the light source being reflected from a user's eye,

2. An eye gaze direction control arrangment as in Claim 1 further characterised in that the said means are adapted to effect a recalibration attempt sequence upon a selected number of interruptions of the reflected light within a selected period of time.

3. An eye gaze direction control arrangement as in Claim 1 or 2 further characterised in that recalibration is effected by a first calibration viewing target location to be caused to be noticeable by a user, and causing information to be stored identifying the angular direction then detected of a reflection of the radiant beam from the eye when it is to be expected to be viewing the calibration viewing target location then causing a second calibration viewing target location to be noticeable by the user, at a position different from the first said calibration position and such as to cause the eye of a user to assume a different angular relationship with respect to the detection equipment and causing information identifying such second angular position to be stored and then effecting a correction factor to be used for maintaining alignment of the position of the cursor on the screen with a viewing direction of the user.

4. An eye gaze direction control arrangement as in Claim 2 or 3 further characterised in that the said means are adapted to enter a recalibration attempt sequence upon a selected eye of the user being closed and opened a selected number of times within a selected period of time.

5. An eye gaze direction control arrangement as in either of Claims 3 or 4 further characterised in that the said means are adapted to effect a recalibration attempt sequence upon an initial triggering by the user in which the sequence includes at least three calibration viewing targets, two of which are aligned in a vertical direction and two of which are aligned in a horizontal direction.

6. An eye gaze direction control arrangement as in Claim 5 further characterised in that the said means are adapted to compare horizontal and vertical alignment data from the calibration viewing targets and if the difference is larger than a selected allowable difference, provide for a further target location to be offered to the user and continue the calculation of correction factor and comparison until at least three target locations provide a comparison within the selected range.

7. An eye gaze direction control arrangement as in Claim 6 further characterised in that the said means are adapted to enlarge the range of acceptable difference in horizontal and vertical alignment of the detected eye gaze direction when viewing the calibration target locations after more than a selected number of comparisons have not been within the selected range.

8. A method of determining eye gaze direction control which includes the steps of directing a light source through a user's eye and detecting such reflection with means to enable the viewing direction of the eye to then be determinable subject to calibration, and providing a calibration of a cursor viewing direction alignment upon an activating signal being effected by the user.

9. An eye gaze direction control arrangement substantially described in the specification with reference to and as illustrated by the accompanying drawings .