WO2013138647A1 - Using convergence angle to select among different ui elements - Google Patents
Using convergence angle to select among different ui elements Download PDFInfo
- Publication number
- WO2013138647A1 WO2013138647A1 PCT/US2013/031632 US2013031632W WO2013138647A1 WO 2013138647 A1 WO2013138647 A1 WO 2013138647A1 US 2013031632 W US2013031632 W US 2013031632W WO 2013138647 A1 WO2013138647 A1 WO 2013138647A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- hmd
- eye
- gaze
- wearer
- images
- Prior art date
Links
- 238000000034 method Methods 0.000 claims description 46
- 230000006870 function Effects 0.000 claims description 10
- 238000003384 imaging method Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 abstract description 3
- 210000004556 brain Anatomy 0.000 abstract description 2
- 210000001747 pupil Anatomy 0.000 description 23
- 230000003287 optical effect Effects 0.000 description 16
- 238000004891 communication Methods 0.000 description 9
- 210000003128 head Anatomy 0.000 description 8
- 238000005192 partition Methods 0.000 description 8
- 239000011521 glass Substances 0.000 description 7
- 230000003190 augmentative effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000004424 eye movement Effects 0.000 description 3
- 230000004886 head movement Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000010267 cellular communication Effects 0.000 description 2
- 239000011248 coating agent Substances 0.000 description 2
- 238000000576 coating method Methods 0.000 description 2
- 210000004087 cornea Anatomy 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 210000005069 ears Anatomy 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- HBBGRARXTFLTSG-UHFFFAOYSA-N Lithium ion Chemical compound [Li+] HBBGRARXTFLTSG-UHFFFAOYSA-N 0.000 description 1
- 241000287181 Sturnus vulgaris Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000000193 eyeblink Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005057 finger movement Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 229910001416 lithium ion Inorganic materials 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
- G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
- G09G3/003—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background to produce spatial visual effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2354/00—Aspects of interface with display user
Definitions
- Wearable systems can integrate various elements, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, into a device thai can be worn by a user.
- Such devices provide a mobile and lightweight solution to communicating, computing and interacting with one's environment.
- an artificial image By placing an image display element close to the wearer's eye(s), an artificial image can be made to overlay the wearer's view of the real world.
- image display elements are incorporated into systems also referred to as “near-eye displays”, “head- mounted displays” (HMDs) or “heads-up displays” (HUDs).
- HMDs head- mounted displays
- HUDs heads-up displays
- the artificial image may fill or nearly fill the wearer's field of view.
- a wearable computing device includes a head-mounted display (HMD), The HMD is configured to display images. The images are viewable from at least one of a first viewing location or a second viewing location.
- the wearable computing device further includes at least one infrared light source.
- the infrared light source is configured to illuminate at least one of the first viewing location or the second viewing location with infrared light such that the infrared light is reflected from the at least one illuminated viewing location as reflected infrared light.
- the wearable computing device further includes at least one camera. The at least one camera is configured to acquire at least one image of the at least one illuminated viewing location by collecting the reflected infrared light.
- the wearable computing device further includes a computer.
- the computer is configured to determine a vergence angle based on the at least one image of the at least one illuminated vie wing location, determine a gaze point based on the vergence angle, select an image based on the gaze point, and control the HMD to display the selected image.
- a method in a second aspect, includes optically determining a first gaze direction and a second gaze direction within a field of view provided by a head-mounted display (HMD).
- the HMD is configured to display images within the field of view.
- the method further includes determining a gaze point based on a vergence angle between the first and second gaze directions.
- the method further includes selecting a target object from the images based on the gaze point and a depth of the target object.
- a method in a third aspect, includes optically determining a first gaze direction and a second gaze direction within a field of vie w provided by a head-mounted display (HMD).
- the HMD is configured to display images within the field of view.
- the method further includes determining a gaze point based on a vergence angle between the first and second gaze directions.
- the method further includes adjusting the images based on the gaze point,
- a non-transitory computer readable medium has stored therein instructions executable by a computing device that cause the computing device to perform functions, including: (1) causing a head-mounted display (HMD) to acquire images of first and second viewing locations, wherein ihe HMD is configured to display images; (2) determining a first gaze direction and a second gaze direction based on the images of the first and second viewing locations; (3) determining a gaze point based on a vergenee angle between the first and second gaze directions; and (4) selecting a target object from the images based on the gaze point and a depth of the target object.
- HMD head-mounted display
- Figure 1 is a schematic diagram of a wearable computing device, in accordance with an example embodiment
- Figure 2A is a perspective view of a head-mounted display, in accordance with an example embodiment.
- Figure 2.B is a perspective view of a head-mounted display, in accordance with an example embodiment.
- Figure 2C is a perspective view of a head-mounted display, in accordance with an example embodiment.
- Figure 3 A is a side view of an eye-tracking system with a forward gaze direction, in accordance with an example embodiment.
- Figure 3B is a side view of the eye-tracking system of Figure 3A with an upward gaze direction, in accordance with an example embodiment.
- Figure 4A is a real-world scene, in accordance with an example embodiment.
- Figure 4B is a real-world scene of Figure 4A, in accordance with an example embodiment.
- Figure 4C is a real-world scene of Figure 4A and Figure 4B, in accordance with an example embodiment.
- Figure 5 is a flowchart of a method, in accordance with an example embodiment.
- Figure 6 is a flowchart of a method, in accordance with an example embodiment. DETAILED DESCRIPTION
- a head-mounted display may enable its wearer to observe the wearer's real-world surroundings and also view a displayed image, such as a computer- generated image.
- the displayed image may overlay a portion of the wearer's field of view of the real world.
- the wearer of the HMD is going about his or her daily activities, such as walking, driving, exercising, etc., the wearer may be able to see a displayed image generated by the HMD at the same time that the wearer is looking out at his or her real-world surroundings.
- the displayed image which could be a virtual image, might include, for example, graphics, text, and/or video.
- the content of the displayed image could relate to any number of contexts, including but not limited to the wearer's current environment, an activity in which the wearer is currently engaged, the biometric status of the wearer, and any audio, video, or textual communications that have been directed to the wearer.
- the images displayed by the HMD may also be part of an interactive user interface.
- the HMD could be part of a wearable computing device.
- the images displayed by ihe HMD could include menus, selection boxes, navigation icons, or other user interface features that enable the wearer to invoke functions of the wearable computing device or otherwise interact with the wearable computing device,
- the images displayed by the HMD could appear anywhere in the wearer's field of view.
- the displayed image might occur at or near the center of the wearer's field of view, or the displayed image might be confined to the top, bottom, or a corner of the wearer's field of view.
- the displayed image might be at ihe periphery of or entirely outside of the wearer's normal field of view.
- the displayed image might be positioned such that it is not visible when the wearer looks straight ahead but is visible when the wearer looks in a specific direction, such as up, down, or to one side.
- the displayed image might overlay only a small portion of the wearer's field of view, or the displayed image might fill most or all of the wearer's field of view.
- the displayed image could be displayed continuously or only at certain times (e.g., only when the wearer is engaged in certain activities).
- the displayed images may appear fixed relative to the wearer's environment.
- the images may appear anchored to a particular object or location within the wearer's environment.
- displayed images may appear fixed relative to the wearer's field of view.
- the HMD may include a graphical user interface (GUI) that may stay substantially anchored to the wearer's field of view regardless of the HMD orientation. Both types of imagery may be implemenied together within the context of the current disclosure.
- GUI graphical user interface
- an optical system in the HMD may include a light source, such as a light-emitting diode (LED), that is configured to illuminate a display panel, such as a liquid crysial-on-silicon (LCOS) display.
- a light source such as a light-emitting diode (LED)
- LCOS liquid crysial-on-silicon
- the display panel generates light patterns by spatially modulating the light from the light source, and the light patterns may be viewable as images at a viewing location.
- the HMD may obtain data from the wearer in order to perform certain functions, for instance to provide context-sensitive images to the wearer.
- the HMD may obtain information regarding the wearer and the wearer's environment and respond accordingly.
- the HMD may use a pupil position recognition technique, wherein if the HMD recognizes that the wearer's pupil location, and thus a corresponding gaze axis, is inclined with respect to a reference axis, the HMD may display images related to objects located above the wearer.
- the HMD may recognize, by a similar pupil position recognition technique, that the wearer is looking downward. Accordingly, the HMD may display images related to objects located below a reference axis of the wearer.
- the wearer's pupil may be illuminated by an infrared light source or multiple infrared light sources.
- An infrared camera may image the pupil and other parts of the HMD wearer's eye.
- the infrared light source(s) could be located in the HMD optical path, or could alternatively be located off- axis.
- the infrared camera could aiso be located in the HMD optical path or off-axis.
- Possible eye tracking modalities that could be used include dark pupil imaging and dual-glint Purkinje image tracking, among other techniques known in the art.
- a processor may implement an image processing algorithm to find the edges or extents of the imaged pupil.
- the image processing algorithms may include pattern recognition, Canny edge detection, thresholding, contrast detection, or differential edge detection, to name a few. Those skilled in the art will understand that a variety of different image processing techniques could be used individually or in combination with other methods in order to obtain pupil location.
- the processor may determine a gaze axis, which may be defined as an axis extending from a viewing location and through a gaze point located within the wearer's field of view.
- a HMD can present a field of view to one eye or to both eyes of a HMD wearer.
- the field of view could include views of the real world environment as well as displayed images that could be presented to one or both eyes.
- the HMD may display the images at various apparent distances relative to each eye of ihe a wearer in order, for instance, to give the illusion that objects are in different distance planes relative to the wearer.
- the brain generally coordinates the eyes to jointly change a vergence angle, which can be defined as the angle made by two intersecting gaze axes.
- the vergence angle could be determined when the HMD wearer focuses upon an object in the real -world environment or when the HMD wearer attempts to vie images displayed by the HMD. In this way, a distance plane at which the HMD wearer is gazing could be determined.
- a depth of the displayed images is known, for instance because the display of images may be controlled by a user interface (UI), and the HMD wearer is using an eye- tracking sy stem, it may be possible to identify at which of the objects the user is gazing. This may allow the placement of UI elements in display locations that are perceived to be very close, or even overlapping, while ihe wearer may be able to discriminate an object of interest in the set of displayed images.
- UI user interface
- images may be adjusted to correspond to the determined distance plane, for instance to appear as in-focus text information while viewing a target object.
- the images may also be displayed at other distance planes to give the effect of an apparent 'background' or 'foreground'.
- Such images could be displayed, for instance, to present a three-dimensional augmented reality to an HMD wearer,
- Vergence angle could also be determined in order to select a target object within a field of view of a HMD wearer.
- an HMD wearer may be looking around a real-world scene and may fixate upon an object.
- the HMD wearer's eyes may individually align with the object and have respective gaze axes.
- the eye -tracking system may determine the wearer's gaze axes and a combined vergence angle.
- the vergence angle could be defined as the (generally smaller) angle defined between the two gaze axes of the HMD user's eyes. From this information, a computer may determine a wearer's gaze point, or the place in three-dimensional space at which the HMD wearer is gazing. In such a manner, a target object (in the form of an image or real-world object) could be selected.
- determining a gaze axis for both eyes can be used to disambiguate potential target objects. For instance, in an office environment, it may be difficult to determine whether a HMD wearer is looking at a pane of glass or a computer monitor beyond it. By determining a gaze depth and/ or gaze point based on the vergence angle, the two situations can be disambiguated. Thus, image adjustment and/or the selection of real-world target objects could be more reliably performed.
- vergence measurements may be useful when gazing at objects or displayed images within a range of about 3 meters. Outside of that range, vergence measurements may be less accurate at determining gaze depth and gaze point. Accordingly, the HMD may use other means to estimate the gaze depth and gaze point if the HMD determines that the target object/gaze depth may lie outside approximately 3 meters.
- HMD Head-mounted display
- eye-tracking system for vergence angle determination
- FIG. 1 is a schematic diagram of a wearable computing device or a head- mounted display (HMD) 100 that may include several different components and subsystems.
- the HMD 100 includes an eye- tracking system 102, a HMD-tracking system 104, an optical system 106, peripherals 108, a power supply 110, a processor 1 12, a memory 1 14, and a user interface 1 15.
- the eye-tracking system 102 may include hardware such as at least one infrared camera 1 16 and at least one infrared light source 1 18.
- the HMD-tracking system 104 may include a gyroscope 120, a global positioning system (GPS) 122, and an accelerometer 124.
- GPS global positioning system
- the optical system 106 may include, in one embodiment, a display panel 12.6, a display light source 128, and optics 130.
- the peripherals 108 may include a wireless communication interface 134, a touchpad 136, a microphone 138, a camera 140, and a speaker 142.
- HMD 100 includes a see-through display.
- the wearer of HMD 100 may observe a portion of the real-world environment, i.e., in a particular field of view provided by the optical system 106.
- the optical system 106 may observe a portion of the real-world environment, i.e., in a particular field of view provided by the optical system 106.
- HMD 100 is operable to display images that are superimposed on the field of view, for example, to provide an "augmented reality" experience. Some of the images displayed by
- HMD 100 may be superimposed over particular objects in the field of view. HMD 100 may also display images that appear to hover within the field of view instead of being associated with particular objects in the field of v ew.
- Components of the HMD 100 may be configured to work in an interconnected fashion with other components within or outside their respective systems.
- at least one mfrared camera 1 1 6 may image one or both of the HMD wearer's eyes.
- the infrared camera 1 16 may deliver image information to the processor 1 12, which may access the memory 1 14 and make a determination regarding the gaze axis (or axes) of the HMD wearer's eye(s).
- the processor 1 12 may subsequently determine a vergence angle that could establish, for instance, the gaze depth of the HMD wearer.
- the processor 1 12 may further accept input from the GPS unit 122, the gyroscope 120, and/or the accelerometer 124 to determine the location and orientation of the HMD 100. Subsequently, the processor 1 12 may control the user interface 1 15 and the display panel 126 to display- images to the HMD wearer that may include context-specific information based on the HMD location and orientation as well as the HMD wearer's vergence angle.
- HMD 100 could be configured as, for example, eyeglasses, goggles, a helmet, a hat, a visor, a headband, or in some other form that can be supported on or from the wearer's head. Further, HMD 100 may be configured to display images to both of the wearer's eyes, for example, using two see-through displays. Alternatively, HMD 100 may- include only a single see-through display and may display images to only one of the wearer's eyes, either the left eye or the right eye.
- the HMD 100 may also represent an opaque display configured to display images to one or both of the wearer's eyes without a view of the real-world environment.
- an opaque display or displays could provide images to both of the wearer's eyes such that the wearer could experience a virtual reality version of the real world.
- the HMD wearer may experience an abstract virtual reality environment that could be substantially or completely detached from the real world.
- the HMD 100 could provide an opaque display for a first eye of the wearer as well as provide a view of the real- world env ironment for a second eye of the wearer.
- a power supply 1 10 may provide power to various HMD components and could represent, for example, a rechargeable lithium-ion battery. Various other power supply materials and types known in the art are possible.
- the functioning of the HMD 100 may be controlled by a processor 1 12 that executes instructions stored in a non-transitory computer readable medium, such as the memory 1 14.
- the processor 1 12 in combination with instructions stored in the memory 1 14 ma function as a controller of HMD 100, As such, the processor 1 12. ma control the user interface 1 15 to adjust the images displayed by HMD 100.
- the processor 1 12 may also control the wireless communication interface 134 and various other components of the HMD 100.
- the processor 1 12 may additionally represent a plurality of computing devices that may serve to control individual components or subsystems of the HMD 100 in a distributed fashion.
- the memory 1 14 may siore data that may include a set of calibrated wearer eye pupil positions and a collection of past eye pupil positions.
- the memory 1 14 may function as a database of information related to gaze direction. Such information may be used by HMD 100 to anticipate where the user will look and determine what images are to be displayed to the wearer.
- Calibrated wearer eye pupil positions may include, for instance, information regarding the extents or range of the wearer's eye pupil movement (right/left and upwards/downwards) as well as wearer eye pupil positions that may relate to various reference axes.
- Reference axes could represent, for example, an axis extending from a viewing location and through a target object or the apparent center of a field of view (i.e. a central axis that may project through a center point of the apparent display panel of the HMD). Other possibilities for reference axes exist. Thus, a reference axis may further represent a basis for determining dynamic gaze direction.
- information may be stored in the memory 1 14 regarding possible control instructions that may be enacted using eye movements.
- possible control instructions may be enacted using eye movements.
- two consecutive wearer eye blinks may represent a control instruction directing the HMD 100 to capture an image using camera 140.
- Another possible embodiment may include a configuration such that specific eye movements may represent a control instruction.
- a HMD wearer may lock or unlock the user interface 1 15 with a series of predetermined eye movements.
- Control instructions could be based on dwell-based selection of a target object. For instance, if a wearer fixates visually upon a particular displayed image or real-world object for longer than a predetermined time period, a control instruction may be generated to select the displayed image or real-world object as a target object. Many other control instructions are possible.
- the HMD 100 may include a user interface 1 15 for providing information to the wearer or receiving input from the wearer.
- the user interface 115 could be associated with, for example, the displayed images and/or one or more input devices in peripherals 108, such as touchpad 136 or microphone 138.
- the processor 1 12 may control the functioning of the HMD 100 based on inputs received through the user interface 1 15. For example, the processor 1 12 may utilize user input from the user interface 115 to control how the HMD 00 displays images within a field of view or to determine what images the HMD 100 displays.
- An eye-tracking system 102 may be included in the HMD 100.
- an eye-tracking system 102 may deliver information to the processor 1 12 regarding the ey e position of a wearer of the HMD 100,
- the eye-tracking data could be used, for instance, to determine a direction in which the HMD wearer may be gazing.
- the processor 1 12 could determine target objects among the displayed images based on information from the eye-tracking system 102.
- the processor 1 12. may control the user interface 1 15 and the display panel 126 to adjust the target object and/or other displayed images in various ways. For instance, a HMD wearer could interact with a mobile-type menu-driven user interface using eye gaze movements.
- the infrared camera 1 16 may be utilized by the eye-tracking system 102 to capture images of a viewing location associated with the HMD 100.
- the infrared camera 1 16 may image the eye of a HMD wearer that may be located at the viewing location.
- the images could be either video images or still images.
- the images obtained by the infrared camera 1 16 regarding the HMD wearer's eye may help determine where the wearer is looking within the HMD field of view, for instance by allowing the processor 1 12 to ascertain the location of the HMD wearer's eye pupil. Analysis of the images obtained by the infrared camera 1 16 could be performed by the processor 1 12 in conjunction with the memory 1 14 to determine, for example, a gaze direction.
- the imaging of the viewing location could occur continuously or at discrete times depending upon, for instance, user interactions with the user interface 1 15 and/or the state of the infrared light source 1 18 which may serve to illuminate the viewing location.
- the infrared camera 1 16 could be integrated into the optical system 106 or mounted on the HMD 100. Alternatively, the infrared camera could be positioned apart from the HMD 100 altogether. Furthermore, the infrared camera 116 could additionally represent a conventional visible light camera with sensing capabilities in the infrared wavelengths.
- the infrared camera 1 16 could be operated at video rate frequency (e.g. 60 Hz) or a multiple of video rates (e.g. 240 Hz), which may be more amenable to combining multiple frames while determining a gaze direction.
- the infrared light source 118 could represent one or more infrared light- emitting diodes (LEDs) or infrared laser diodes that may illuminate a viewing location.
- LEDs infrared light- emitting diodes
- One or both eyes of a wearer of the HMD 100 may be illuminated by the infrared light source 18.
- the infrared light source 1 18 may be positioned along an optical axis common to the infrared camera, and/or the infrared light source 1 18 may be positioned elsewhere.
- the infrared light source 1 18 may illuminate the viewing location continuously or may be turned on at discrete times. Additionally, when illuminated, the infrared light source 1 18 may be modulated at a particular frequency. Other types of modulation of the infrared light source 1 18, such as adjusting the intensity level of the infrared light source 1 18, are possible.
- the eye-tracking system 102 could be configured to acquire images of glint reflections from the outer surface of the cornea, which are also called first Purkinje images.
- the eye-tracking system 102 could be configured to acquire images of reflections from the inner, posterior surface of the lens, which are termed fourth Purkinje images.
- the eye -tracking system 102 could be configured to acquire images of the eye pupil with so-called bright and/or dark pupil images.
- a combination of these glint and pupil imaging techniques may be used for rotational eye tracking, accuracy, and redundancy. Other imaging and tracking methods are possible.
- Those knowledgeable in the art will understand that there are several alternative ways to achieve eye tracking with a combination of infrared illuminator and camera hardware,
- At least one eye- tracking system 102 may be utilized with one or more infrared cameras 1 16 and one or more infrared light sources 1 18 in order to track the position of one eye or both eyes of the HMD wearer.
- the HMD-tracking system 104 could be configured to provide a HMD position and a HMD orientation to the processor 1 12. This position and orientation data may help determine a central axis to which a gaze direction is compared. For instance, the central axis may correspond to the orientation of the HMD.
- the gyroscope 120 could be a niicroelecrromechanical system (MEMS) gyroscope, a fiber optic gyroscope, or another type of gyroscope known in the art.
- the gyroscope 120 may be configured to provide orientation information to the processor 1 12.
- the GPS unit 12.2 could be a receiver that obtains clock and other signals from GPS satellites and may be configured to provide real-time location information to the processor 1 12.
- the HMD-tracking system 104 could further include an accelerometer 124 configured to provide motion input data to the processor 1 12.
- the optical system 106 could include components configured to provide images at a viewing location.
- the viewing location may correspond to the location of one or both eyes of a wearer of a HMD 100.
- the components could include a display panel 126, a display light source 128, and optics 130. These components may be optically and/or electrically-coupled to one another and may be configured to provide viewable images at a viewing location.
- one or two optical systems 106 could be provided in a HMD apparatus.
- the HMD wearer could view images in one or both eyes, as provided by one or more optical systems 106.
- the optical system(s) 106 could include an opaque display and/or a see-through display, which may- allow a view of the real-world environment while providing superimposed images.
- the HMD 100 may include a wireless communication interface 134 for wirelessly communicating with one or more devices directly or via a communication network.
- the wireless communication interface 134 could use 3G cellular communication, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communication, such as WiMAX or LTE.
- the wireless commumcation interface 134 could communicate with a wireless local area network (WLA.N), for example, using WiFi.
- WLA.N wireless local area network
- the wireless communication interface 134 could communicate directly with a device, for example, using an infrared link, Bluetooth, or ZigBee.
- the wireless communication interface 134 could interact with devices that may include, for example, components of the HMD 100 and/or externally-located devices.
- Figure 1 shows various components of the HMD 100 (i.e., wireless commumcation interface 134, processor 1 12, memory 1 14, infrared camera 1 1 6, display panel 126, GPS 122, and user interface 1 15) as being integrated into HMD 100
- these components could be physically separate from HMD 100
- the infrared camera 1 1 6 could be mounted on the wearer separate from HMD 100.
- the HMD 100 could be part of a wearable computing device in the form of separate devices that can be worn on or carried by the wearer.
- the separate components that make up the wearable computing device could be communicatively coupled together in either a wired or wireless fashion,
- Figures 2A and 2B illustrate two of many possible embodiments involving head-mounted displays with gaze axis vergence determination.
- the example systems could be used to receive, transmit, and display data
- the HMD 200 may have a glasses format.
- the HMD 200 has a frame 202 that could include nosepiece 224 and earpieces 218 and 220.
- the frame 202, nosepieee 224, and earpieces 21 8 and 220 could be configured to secure the HMD 200 to a user's face via a user's nose and ears.
- Each of the frame elements, 202 , 224, and 218 may be formed of a solid structure of plastic and/or metal, or may be formed of a hollow structure of similar material so as io allow wiring and component interconnects to be internally routed through the HMD 200. Other materials may be possible as well.
- the earpieces 218 and 220 could be attached to projections that extend away from the lens frame 202 and could be positioned behind a user's ears to secure the HMD 2.00 to the user.
- the projections could further secure the HMD 200 to the user by extending around a rear portion of the user's head.
- the HMD 200 could connect to or be affixed within a head-mounted helmet structure. Other possibilities exist as well.
- Lens elements 210 and 212 could be mounted in frame 202.
- the lens elements 210 and 212 could be formed of any material that can suitably display a projected image or graphic.
- Each of the lens elements 2.10 and 2.12 could be sufficiently transparent to allow a user to see through the lens element. Combining these two features of the lens elements may facilitate an augmented reality or a heads-up display where the projected image or graphic is superimposed over a real-world view as perceived by the user through Jens elements 2.10 and 212.
- the HMD 200 may include a computer 214, a touch pad 216, a camera 222, and a display 204.
- the computer 214 is shown to be positioned on the extending side arm of the HMD 200; however, the computer 214 may be provided on other parts of the HMD 200 or may be positioned remote from the HMD 200 (e.g. the computer 214 could be wire- or wirelessly-connected to the HMD 200).
- the computer 214 could include a processor and memory, for example.
- the computer 2.14 may be configured to receive and analyze data from the camera 222. and ihe touch pad 216 (and possibly from other sensory devices, user- interfaces, or both) and generate images for output by the lens elements 210 and 212.
- a camera 222 could be positioned on an extending side arm of the HMD 200, however, the camera 222 may be provided on other parts of the HMD 200.
- the camera 222 may be configured to capture images at various resolutions or at different frame rates.
- the camera 222 could be configured as a video camera and/or as a still camera.
- a camera with small form factor, such as those used in cell phones or webcams, for example, may be incorporated into an example embodiment of HMD 200.
- Figure 2A illustrates one camera 222
- more cameras could be used, and each may be configured to capture the same view, or to capture different views.
- camera 222 may be forward facing to capture at least a portion of the real-world view perceived by the user. This forward facing image captured by the camera 2.22 may then be used to generate an augmented reality where computer generated images appear to interact with the real world view perceived by the user.
- HMD 200 Other sensors could be incorporated into HMD 200.
- Other sensors may include one or more of a gyroscope or an accelerometer, for example.
- Other sensing devices may be included in HMD 200.
- the touch pad 216 is shown on an extending side arm of the HMD 200. However, the touch pad 216 may be positioned on other parts of the HMD 200. Also, more than one touch pad may be present on the HMD 200.
- the touch pad 216 may be used by a user to input commands.
- the touch pad 216 may sense at least one of a position and a movement of a finger via capacitive sensing, resistance sensing, or a surface acoustic wave process, among other possibilities.
- the touch pad 216 may be capable of sensing finger movement in a direction parallel or planar to the pad surface, in a direction normal to the pad surface, or both, and may also be capable of sensing a level of pressure applied to the pad surface.
- the touch pad 216 may be formed of one or more translucent or transparent insulating layers and one or more translucent or transparent conducting layers. Edges of the touch pad 216 may be formed to have a raised, indented, or roughened surface, so as to provide tactile feedback to a user when the user's finger reaches the edge, or other area, of the touch pad 2.16. If more than one touch pad is present, each touch pad may be operated independently, and may provide a different function.
- the HMD 200 may include eye-tracking systems 206 and 208, which may be configured to track the eye position of each eye of the HMD wearer.
- the eye- tracking systems 206 and 208 may each mclude one or more infrared light sources and one or more infrared cameras.
- Each of the eye-tracking systems 206 and 208 could be configured to image one or both of the HMD wearer's eyes.
- two eye-tracking systems are depicted in Figure 2A, other embodiments are possible. For instance, one eye-tracking system could be used to track both eyes of a user.
- Display 204 could represent, for instance, an at least partially reflective surface upon which images could be projected using a projector.
- the lens elements 210 and 212 could act as a combiner in a light projection system and may include a coating that reflects the light projected onto them from projectors. In some embodiments, a reflective coating may not be used (e.g. when the projectors are scanning laser devices). The images could be thus viewable to a HMD user.
- the display 204 is depicted as presented to the right eye of the HMD wearer, other example embodiments could include a display for both eyes or a single display viewable by both eyes.
- the lens elements 210 and 212 could themselves mclude: a transparent or semi- transparent matrix display, such as an electroluminescent display or a liquid crystal display, one or more waveguides for delivering an image to the user's eyes, or other optical elements capable of delivering an in focus near-to-eye image to the user.
- a corresponding display driver may be disposed within the frame 202 for driving such a matrix display.
- a laser or fight-emitting diode (LED) source and scanning system could be used to draw a raster display directly onto the retina of one or more of the user's eyes.
- LED fight-emitting diode
- FIG. 2B a HMD 226 with monocle design is illustrated.
- the HMD 226 may include a single display 204 that may be coupled to one of the side arms or the nose piece 2.24, In one example, the single display 204 could be coupled to the inner side (i.e. the side exposed to a portion of a user's head when worn by the user) of the extending side ar of frame 2.02.
- the display 204 could be posiiioned in front of or proximate to a user's eye when the HMD 200 is worn by a user.
- the display 204 could be configured to overlay computer-generated graphics upon the user's view of the physical world.
- eye-tracking systems 2.06 and 208 could be mounted on nosepiece 224.
- the eye -tracking systems 206 and 208 could be configured to track the eye position of both eyes of a HMD wearer.
- the HMD 226 could include a computer 214 and a display 2.04 for one eye of the HMD wearer.
- Figure 2C illustrates a HMD 228 with a binocular design.
- separate displays could be provided for each eye of a HMD user.
- displays 204 and 230 could be provided to the right and left eye of the HMD user, respectively.
- a single display could provide images to both eyes of the HMD user.
- the images provided to each eye may be different or identical to one another. Further, the images could be provided to each eye in an effort to create a stereoscopic illusion of depth.
- Figures 3A and 3B are side and front views of an eye of a HMD user gazing forward and gazing upward, respectively.
- light sources 308 and 310 could be configured to illuminate the HMD user's eye 302.
- Glint reflections 314 and 316 from the HMD user's eye 302 could be generated based on the illumination from the light sources 308 and 310,
- These glint reflections 314 and 316 could be first Purkinje images from reflections from the outer surface of the HMD user's cornea.
- the glint reflections 314 and 316 as well as the eye pupil 304 could be imaged by a camera 318.
- Images could be sent to a processor that may, in turn, analyze the glint locations 32.4 and 326 with respect to a coordinate system 320 in order to determine and/or confirm a pupil location 322,
- the pupil location may be determined to be near the center of the reference coordinate system 320.
- a gaze direction 312 may be determined to be straight ahead.
- a gaze point may be determined to be at a point along the gaze direction 312.
- FIG. 3B depicts a scenario 328 where a HMD user is gazing upward. Similar to the aforementioned example, light sources 308 and 310 could induce respective glint reflections 330 and 332 from the HMD user's eye 302. In this scenario, however, the glint reflections 330 and 332 may appear in different locations due to the change in the eye gaze direction of the HMD wearer and asymmetry of the shape of the eye 302. Thus glint reflections 338 and 340 may move with respect to reference coordinate system 320. Image analysis could be used to determine the pupil location 336 within the reference coordinate system 320. From the pupil location 336, a gaze direction 342 may be determined. A gaze point could be determined as a point along the gaze direction 342.
- gaze directions could be optically determined for both eyes, for example, as described above for Figures 3A and 3B.
- the gaze directions from both eyes could be used to find a vergence angle in a determination of where the user may be gazing.
- a gaze direction could be optically determined for only one eye, and the gaze direction for the other eye could he inferred.
- the verge ce angle could be determined based on the optically-determined gaze direction and the inferred gaze direction.
- a gaze direction could be inferred from head movements. For example, because of the head's natural tendency to keep the eyes centered (i.e., the head lags behind the eyes, but tends to "frame" the subject), it is possible to look for eye fixations that cluster around a certain point (converting fixations into gaze points), and then use other sensors (e.g., one or more of the sensors in HMD-tracking system 104) to detect movement of the head. When that head movement ceases, but the optically-tracked eye remains off-center in one direction, it is possible to infer that the other eye is similarly off-center in the other direction.
- this pattern of head movements can be indicative of the person's eyes converging on a nearby object, with the person's head ''framing" the nearby object.
- the gaze directions of both eyes would be at the same angle from the forward direction (defined by the position of the person's head) but from opposite sides.
- Figures 4A, 4B, and 4C illustrate scenarios in which the aforementioned system could be applied
- ⁇ scenario 400 a HMD wearer 402 with first and second eyes (404 and 406) could be in a real-world environment with a partition that may include a wall portion 414 and a glass portion 410.
- a computer monitor 416 may be viewable through the glass portion 410.
- the computer monitor 416 could be located on a desk 418.
- the partition and the computer monitor 416 could be located at a first depth plane 412 and a second depth plane 420, with respect to a HMD wearer plane 408.
- a HMD wearer may be looking at the computer monitor 416 (scenario 428).
- the eye -tracking data from both eyes of the HMD wearer may allow the processor 1 12. to determine gaze axes 422 and 424.
- a corresponding vergence angle 426 could be determined.
- processor 1 12 may determine that the HMD wearer is gazing at the computer monitor 416.
- the determination of a vergence angle may help to disambiguate an actual target object from a set of candidate target objects.
- the actual target object may be ambiguous if eye-tracking data from only one eye is used or if only HMD-tracking data is used.
- the vergence angle 426 of the gaze axes 422. and 424 may reduce the uncertainty of objeci selection.
- Figure 4C illustrates how various notifications may be generated upon vergence angle determination.
- Figure 4C depicts a HMD wearer 402 with first and second eyes (404 and 406).
- the HMD wearer could be in a real- world environment that includes a partition that may include a wail portion 414 and a glass portion 410. Beyond the partition, a computer monitor 416 may be viewable through the glass portion 410.
- the computer monitor 416 could be located on a desk 418.
- the partition and the computer monitor 416 could be located at a first depth plane 412 and a second depth plane 420, respectively.
- the system may determine thai the HMD wearer is gazing at the computer monitor 416. Accordingly, notifications in the form of images could be generated.
- An image message 434 that states, "Partition” could help alert the HMD wearer not to run into it when walking, for instance.
- a notification 432. that states, "Computer” could help further identify the objeci at which the HMD wearer is gazing.
- Other target object-dependent notification messages are possible.
- the effective distances for determining gaze point using vergence angle may be up to around 3 meters. Therefore, the following methods may be useful in close- to mid-range interactions such as the aforementioned office example. In long range situations (greater than 3 meters), vergence may he a less useful way to determine gaze depth and/or to select target objects.
- a method 500 is provided for selecting target objects by determining the vergence angle between the gaze axes of the eyes of a HMD wearer.
- the method could be performed using an apparatus shown in Figures 1-4C and as described above, however, other configurations could be used.
- Figure 5 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.
- Method step 502 includes optically determining a first gaze direction and a second gaze direction within a field of view provided by a head-mounted display (FIMD).
- the HMD is configured to display images within the field of view.
- the first and second gaze directions could represent the gaze axis of each eye of a HMD wearer.
- the first and second gaze directions could be optically determined using various apparatuses known in the art including the eye -tracking system described above.
- the HMD may include at least one display configured to generate images viewable to one or both eyes of the HMD wearer.
- Method step 504 includes determining a gaze point based on the vergence angle between the first and second gaze directions.
- the vergence angle is the angle created when the first and second gaze directions intersect, for instance when the HMD wearer is looking at a nearby object.
- the vergence angle may strongly indicate the point at which the FIMD wearer is gazing.
- a gaze point may be determined from the vergence angle using basic geometric methods.
- Method step 506 includes selecting a target object from the images based on the gaze point and the depth of the target object. The selected target object could have a similar or identical depth as the gaze point.
- the selected target object could be any member of the set of images displayed by the HMD.
- the target object selection could be performed immediately upon determination of a gaze point / target object location match, or could take place after a predetermined period of time. For instance, the target object selection could happen once a HMD wearer stares at an image for 500 milliseconds.
- a method 600 is provided for adjusting images based on a gaze point, which can be determined from a vergence angle between the gaze axes of the eyes of a head- mounted display (HMD) wearer.
- the method could be performed using an apparatus shown in Figures 1-4C and as described above, however, other configurations could be used.
- Figure 6 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.
- the first two steps of method 600 could be similar or identical to the corresponding steps of method 500.
- an eye-tracking system or other optical means could be utilized to determine a first gaze direction and a second gaze direction within a field of view of the HMD (step 602).
- a gaze point ma then be determined based on the vergence angle between the first and second gaze directions (step 604).
- images displayed in the field of view for the HMD could be adjusted based on the determined gaze point.
- the determined gaze point could relate to a target object that could include real-world objects or displayed images.
- the adjusted images could include any graphical or text element displayed by the HMD.
- the eye-tracking system could determine that a HMD wearer is gazing at a computer screen based on the vergence angle of his or her eyes.
- images (such as icons or other notifications) could be adjusted away from the gaze location so as to allow an unobstructed view of the real-world object.
- the images could be adjusted dynamically, or, for instance, only when a new, contextually-important gaze point is determined.
- an image upon recognition that a HMD wearer is gazing at a target object, an image could be displayed that provides information about the target object.
- a notification may be generated.
- the notification could take the form of an image viewable to the HMD wearer as apparently adjacent to the computer screen.
- the notification could include specific information about the computer such as machine owner, model number, operating state, etc. Other notification types and content are possible.
- Non-transitory computer readable medium could be, for example, a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache memory, one or more magnetically encoded discs, one or more optically encoded discs, or any other form of non-transitory data storage.
- the non-transitory computer readable medium could also be distributed among multiple data storage elements, which could be remotely located from each other.
- the computing device that executes the stored instructions could be a wearable computing device, such as a wearable computing device 100 illustrated in Figure I .
- the computing device that executes the stored instructions could be another computing device, such as a server in a server network.
- a non-transitory computer readable medium may store instxuciions executable by the processor 1 12 to perform various functions.
- instructions that could be used to carry out method 500 may be stored in memory 1 14 and could be executed by processor 1 12. in such an embodiment, upon receiving gaze information from the eye-tracking system 102, the processor 1 12. may carry out instructions to determine a gaze axis for both eyes of a user. Accordingly, a vergence angle may be calculated. Based on at least the determined vergence angle, a target object may be selected fro the set of displayed images.
Abstract
A wearable computing system may include a head-mounted display (HMD). The HMD could be configured to present a field of view that could include views of the real world environment as well as displayed images. As the viewer attempts to see objects at different real or apparent depths within the field of view, the brain may generally coordinate the eyes to jointly change a vergence angle. If the depth is known (because it may be generated by a user interface (UI)) and the user is wearing an eye-tracking system, it is possible to determine at which of the objects the user intends to look. This may allow the interface to place UI elements in locations that are perceived to be very close, or even overlapping, while the wearer may able to discriminate the object of interest, which is generally not possible with non-stereoscopic displays.
Description
Using Convergence Angle to Select Among Different IJi Elements
CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application claims priority to United States Provisional Patent
Application Serial No. 61 /611, 188 filed March 15, 2012, the contents of which are hereby incorporated by reference.
BACKGROUND [0Θ02] Wearable systems can integrate various elements, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, into a device thai can be worn by a user. Such devices provide a mobile and lightweight solution to communicating, computing and interacting with one's environment. With the advance of technologies associated with wearable systems and miniaturized optical elements, it has become possible to consider wearable compact optical displays that augment the wearer's experience of the real world,
[0003] By placing an image display element close to the wearer's eye(s), an artificial image can be made to overlay the wearer's view of the real world. Such image display elements are incorporated into systems also referred to as "near-eye displays", "head- mounted displays" (HMDs) or "heads-up displays" (HUDs). Depending upon the size of the display element and the distance to the wearer's eye, the artificial image may fill or nearly fill the wearer's field of view.
SUMMARY
[0004J In a first aspect, a wearable computing device is provided. The wearable computing device includes a head-mounted display (HMD), The HMD is configured to display images. The images are viewable from at least one of a first viewing location or a second viewing location. The wearable computing device further includes at least one infrared light source. The infrared light source is configured to illuminate at least one of the first viewing location or the second viewing location with infrared light such that the infrared light is reflected from the at least one illuminated viewing location as reflected infrared light. The wearable computing device further includes at least one camera. The at least one camera is configured to acquire at least one image of the at least one illuminated viewing location by collecting the reflected infrared light. The wearable computing device further includes a computer. The computer is configured to determine a vergence angle based on the at least one image of the at least one illuminated vie wing location, determine a gaze point based on the vergence angle, select an image based on the gaze point, and control the HMD to display the selected image.
I.0005J In a second aspect, a method is provided. The method includes optically determining a first gaze direction and a second gaze direction within a field of view provided by a head-mounted display (HMD). The HMD is configured to display images within the field of view. The method further includes determining a gaze point based on a vergence angle between the first and second gaze directions. The method further includes selecting a target object from the images based on the gaze point and a depth of the target object.
[0006J In a third aspect, a method is provided. The method includes optically determining a first gaze direction and a second gaze direction within a field of vie w provided by a head-mounted display (HMD). The HMD is configured to display images within the field of view. The method further includes determining a gaze point based on a vergence
angle between the first and second gaze directions. The method further includes adjusting the images based on the gaze point,
[3)807] In a fourth aspect, a non-transitory computer readable medium is provided. The non-transitory computer readable medium has stored therein instructions executable by a computing device that cause the computing device to perform functions, including: (1) causing a head-mounted display (HMD) to acquire images of first and second viewing locations, wherein ihe HMD is configured to display images; (2) determining a first gaze direction and a second gaze direction based on the images of the first and second viewing locations; (3) determining a gaze point based on a vergenee angle between the first and second gaze directions; and (4) selecting a target object from the images based on the gaze point and a depth of the target object.
BRIEF DESCRIPTION OF THE DRAWINGS
[8808] Figure 1 is a schematic diagram of a wearable computing device, in accordance with an example embodiment,
[0009] Figure 2A is a perspective view of a head-mounted display, in accordance with an example embodiment.
[0810] Figure 2.B is a perspective view of a head-mounted display, in accordance with an example embodiment.
[0811] Figure 2C is a perspective view of a head-mounted display, in accordance with an example embodiment.
[0012] Figure 3 A is a side view of an eye-tracking system with a forward gaze direction, in accordance with an example embodiment.
[0013] Figure 3B is a side view of the eye-tracking system of Figure 3A with an upward gaze direction, in accordance with an example embodiment.
[0814] Figure 4A is a real-world scene, in accordance with an example embodiment.
[8815] Figure 4B is a real-world scene of Figure 4A, in accordance with an example embodiment.
[0016] Figure 4C is a real-world scene of Figure 4A and Figure 4B, in accordance with an example embodiment.
[0017] Figure 5 is a flowchart of a method, in accordance with an example embodiment.
[8818] Figure 6 is a flowchart of a method, in accordance with an example embodiment.
DETAILED DESCRIPTION
[Θ819] In the following detailed description, reference is made to the accompanying figures, which form a part thereof. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description and figures are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. Tt will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.
1, Overview
[Θ82Θ] A head-mounted display (HMD) may enable its wearer to observe the wearer's real-world surroundings and also view a displayed image, such as a computer- generated image. In some cases, the displayed image may overlay a portion of the wearer's field of view of the real world. Thus, while the wearer of the HMD is going about his or her daily activities, such as walking, driving, exercising, etc., the wearer may be able to see a displayed image generated by the HMD at the same time that the wearer is looking out at his or her real-world surroundings.
[Θ821] The displayed image, which could be a virtual image, might include, for example, graphics, text, and/or video. The content of the displayed image could relate to any number of contexts, including but not limited to the wearer's current environment, an activity in which the wearer is currently engaged, the biometric status of the wearer, and any audio, video, or textual communications that have been directed to the wearer. The images displayed by the HMD may also be part of an interactive user interface. For example, the HMD could be part of a wearable computing device. Thus, the images displayed by ihe
HMD could include menus, selection boxes, navigation icons, or other user interface features that enable the wearer to invoke functions of the wearable computing device or otherwise interact with the wearable computing device,
[Θ822] The images displayed by the HMD could appear anywhere in the wearer's field of view. For example, the displayed image might occur at or near the center of the wearer's field of view, or the displayed image might be confined to the top, bottom, or a corner of the wearer's field of view. Alternatively, the displayed image might be at ihe periphery of or entirely outside of the wearer's normal field of view. For example, the displayed image might be positioned such that it is not visible when the wearer looks straight ahead but is visible when the wearer looks in a specific direction, such as up, down, or to one side. In addition, the displayed image might overlay only a small portion of the wearer's field of view, or the displayed image might fill most or all of the wearer's field of view. The displayed image could be displayed continuously or only at certain times (e.g., only when the wearer is engaged in certain activities).
[0823] The displayed images may appear fixed relative to the wearer's environment. For instance, the images may appear anchored to a particular object or location within the wearer's environment. Alternatively, displayed images may appear fixed relative to the wearer's field of view. For example, the HMD may include a graphical user interface (GUI) that may stay substantially anchored to the wearer's field of view regardless of the HMD orientation. Both types of imagery may be implemenied together within the context of the current disclosure.
[8824] To display an image to the wearer, an optical system in the HMD may include a light source, such as a light-emitting diode (LED), that is configured to illuminate a display panel, such as a liquid crysial-on-silicon (LCOS) display. The display panel generates light
patterns by spatially modulating the light from the light source, and the light patterns may be viewable as images at a viewing location.
[3)825] The HMD may obtain data from the wearer in order to perform certain functions, for instance to provide context-sensitive images to the wearer. In an example embodiment, the HMD may obtain information regarding the wearer and the wearer's environment and respond accordingly. For instance, the HMD may use a pupil position recognition technique, wherein if the HMD recognizes that the wearer's pupil location, and thus a corresponding gaze axis, is inclined with respect to a reference axis, the HMD may display images related to objects located above the wearer. Alternatively, the HMD may recognize, by a similar pupil position recognition technique, that the wearer is looking downward. Accordingly, the HMD may display images related to objects located below a reference axis of the wearer.
[0026J In order to determine the actual position of a HMD wearer's pupil and to determine a corresponding gaze axis, the wearer's pupil may be illuminated by an infrared light source or multiple infrared light sources. An infrared camera may image the pupil and other parts of the HMD wearer's eye. The infrared light source(s) could be located in the HMD optical path, or could alternatively be located off- axis. The infrared camera could aiso be located in the HMD optical path or off-axis. Possible eye tracking modalities that could be used include dark pupil imaging and dual-glint Purkinje image tracking, among other techniques known in the art.
[0027] A processor may implement an image processing algorithm to find the edges or extents of the imaged pupil. The image processing algorithms may include pattern recognition, Canny edge detection, thresholding, contrast detection, or differential edge detection, to name a few. Those skilled in the art will understand that a variety of different image processing techniques could be used individually or in combination with other
methods in order to obtain pupil location. After image processing, the processor may determine a gaze axis, which may be defined as an axis extending from a viewing location and through a gaze point located within the wearer's field of view.
[0828] A HMD can present a field of view to one eye or to both eyes of a HMD wearer. The field of view could include views of the real world environment as well as displayed images that could be presented to one or both eyes. The HMD may display the images at various apparent distances relative to each eye of ihe a wearer in order, for instance, to give the illusion that objects are in different distance planes relative to the wearer. As the HMD wearer attempts to see each of these objects, the brain generally coordinates the eyes to jointly change a vergence angle, which can be defined as the angle made by two intersecting gaze axes.
[0829] By tracking the gaze axis of both eyes of an HMD wearer, the vergence angle could be determined when the HMD wearer focuses upon an object in the real -world environment or when the HMD wearer attempts to vie images displayed by the HMD. In this way, a distance plane at which the HMD wearer is gazing could be determined.
[0030] If a depth of the displayed images is known, for instance because the display of images may be controlled by a user interface (UI), and the HMD wearer is using an eye- tracking sy stem, it may be possible to identify at which of the objects the user is gazing. This may allow the placement of UI elements in display locations that are perceived to be very close, or even overlapping, while ihe wearer may be able to discriminate an object of interest in the set of displayed images.
[0031] Further, images may be adjusted to correspond to the determined distance plane, for instance to appear as in-focus text information while viewing a target object. The images may also be displayed at other distance planes to give the effect of an apparent
'background' or 'foreground'. Such images could be displayed, for instance, to present a three-dimensional augmented reality to an HMD wearer,
[0832] Vergence angle could also be determined in order to select a target object within a field of view of a HMD wearer. For instance, an HMD wearer may be looking around a real-world scene and may fixate upon an object. The HMD wearer's eyes may individually align with the object and have respective gaze axes. The eye -tracking system may determine the wearer's gaze axes and a combined vergence angle. The vergence angle could be defined as the (generally smaller) angle defined between the two gaze axes of the HMD user's eyes. From this information, a computer may determine a wearer's gaze point, or the place in three-dimensional space at which the HMD wearer is gazing. In such a manner, a target object (in the form of an image or real-world object) could be selected.
[0033] In addition, determining a gaze axis for both eyes (and thus determining a vergence angle) can be used to disambiguate potential target objects. For instance, in an office environment, it may be difficult to determine whether a HMD wearer is looking at a pane of glass or a computer monitor beyond it. By determining a gaze depth and/ or gaze point based on the vergence angle, the two situations can be disambiguated. Thus, image adjustment and/or the selection of real-world target objects could be more reliably performed.
[0034] In practice, vergence measurements may be useful when gazing at objects or displayed images within a range of about 3 meters. Outside of that range, vergence measurements may be less accurate at determining gaze depth and gaze point. Accordingly, the HMD may use other means to estimate the gaze depth and gaze point if the HMD determines that the target object/gaze depth may lie outside approximately 3 meters.
[0035] It will be evident to those skilled in the art that there are a variety of ways to implement such a method for vergence determination and subsequent target object selection or image adjustment/selection in a HMD system. The details of such implementations may
depend on, for example, the type of data provided to the HMD, the local environmental conditions, the location of the user, and the task to be performed.
[8836] Certain illustrative examples of using eye-iracking data to determine eye gaze vergence so as to select target objects and to adjust images displayed by a HMD are described below. It is to be understood, however, that other embodiments are possible and are implicitly considered within the context of the following example embodiments.
2, Head-mounted display (HMD) with eye-tracking system for vergence angle determination
[0837] Figure I is a schematic diagram of a wearable computing device or a head- mounted display (HMD) 100 that may include several different components and subsystems. As shown, the HMD 100 includes an eye- tracking system 102, a HMD-tracking system 104, an optical system 106, peripherals 108, a power supply 110, a processor 1 12, a memory 1 14, and a user interface 1 15. The eye-tracking system 102 may include hardware such as at least one infrared camera 1 16 and at least one infrared light source 1 18. The HMD-tracking system 104 may include a gyroscope 120, a global positioning system (GPS) 122, and an accelerometer 124. The optical system 106 may include, in one embodiment, a display panel 12.6, a display light source 128, and optics 130. The peripherals 108 may include a wireless communication interface 134, a touchpad 136, a microphone 138, a camera 140, and a speaker 142.
[003SJ In an example embodiment, HMD 100 includes a see-through display. Thus, the wearer of HMD 100 may observe a portion of the real-world environment, i.e., in a particular field of view provided by the optical system 106. In the example embodiment,
HMD 100 is operable to display images that are superimposed on the field of view, for example, to provide an "augmented reality" experience. Some of the images displayed by
HMD 100 may be superimposed over particular objects in the field of view. HMD 100 may
also display images that appear to hover within the field of view instead of being associated with particular objects in the field of v ew.
[Θ839] Components of the HMD 100 may be configured to work in an interconnected fashion with other components within or outside their respective systems. For instance, in an example embodiment, at least one mfrared camera 1 1 6 may image one or both of the HMD wearer's eyes. The infrared camera 1 16 may deliver image information to the processor 1 12, which may access the memory 1 14 and make a determination regarding the gaze axis (or axes) of the HMD wearer's eye(s). The processor 1 12 may subsequently determine a vergence angle that could establish, for instance, the gaze depth of the HMD wearer. The processor 1 12 may further accept input from the GPS unit 122, the gyroscope 120, and/or the accelerometer 124 to determine the location and orientation of the HMD 100. Subsequently, the processor 1 12 may control the user interface 1 15 and the display panel 126 to display- images to the HMD wearer that may include context-specific information based on the HMD location and orientation as well as the HMD wearer's vergence angle.
[ΘΘ4©] HMD 100 could be configured as, for example, eyeglasses, goggles, a helmet, a hat, a visor, a headband, or in some other form that can be supported on or from the wearer's head. Further, HMD 100 may be configured to display images to both of the wearer's eyes, for example, using two see-through displays. Alternatively, HMD 100 may- include only a single see-through display and may display images to only one of the wearer's eyes, either the left eye or the right eye.
[Θ841] The HMD 100 may also represent an opaque display configured to display images to one or both of the wearer's eyes without a view of the real-world environment. For instance, an opaque display or displays could provide images to both of the wearer's eyes such that the wearer could experience a virtual reality version of the real world. Alternatively, the HMD wearer may experience an abstract virtual reality environment that
could be substantially or completely detached from the real world. Further, the HMD 100 could provide an opaque display for a first eye of the wearer as well as provide a view of the real- world env ironment for a second eye of the wearer.
[Θ842] A power supply 1 10 may provide power to various HMD components and could represent, for example, a rechargeable lithium-ion battery. Various other power supply materials and types known in the art are possible.
[Θ843] The functioning of the HMD 100 may be controlled by a processor 1 12 that executes instructions stored in a non-transitory computer readable medium, such as the memory 1 14. Thus, the processor 1 12 in combination with instructions stored in the memory 1 14 ma function as a controller of HMD 100, As such, the processor 1 12. ma control the user interface 1 15 to adjust the images displayed by HMD 100. The processor 1 12 may also control the wireless communication interface 134 and various other components of the HMD 100. The processor 1 12 may additionally represent a plurality of computing devices that may serve to control individual components or subsystems of the HMD 100 in a distributed fashion.
[1)844] in addition to instructions that may be executed by the processor 1 12, the memory 1 14 may siore data that may include a set of calibrated wearer eye pupil positions and a collection of past eye pupil positions. Thus, the memory 1 14 may function as a database of information related to gaze direction. Such information may be used by HMD 100 to anticipate where the user will look and determine what images are to be displayed to the wearer. Calibrated wearer eye pupil positions may include, for instance, information regarding the extents or range of the wearer's eye pupil movement (right/left and upwards/downwards) as well as wearer eye pupil positions that may relate to various reference axes.
[0045] Reference axes could represent, for example, an axis extending from a viewing location and through a target object or the apparent center of a field of view (i.e. a central axis that may project through a center point of the apparent display panel of the HMD). Other possibilities for reference axes exist. Thus, a reference axis may further represent a basis for determining dynamic gaze direction.
[0046J In addition, information may be stored in the memory 1 14 regarding possible control instructions that may be enacted using eye movements. For instance, two consecutive wearer eye blinks may represent a control instruction directing the HMD 100 to capture an image using camera 140. Another possible embodiment may include a configuration such that specific eye movements may represent a control instruction. For example, a HMD wearer may lock or unlock the user interface 1 15 with a series of predetermined eye movements.
Θ047] Control instructions could be based on dwell-based selection of a target object. For instance, if a wearer fixates visually upon a particular displayed image or real-world object for longer than a predetermined time period, a control instruction may be generated to select the displayed image or real-world object as a target object. Many other control instructions are possible.
[Θ848] The HMD 100 may include a user interface 1 15 for providing information to the wearer or receiving input from the wearer. The user interface 115 could be associated with, for example, the displayed images and/or one or more input devices in peripherals 108, such as touchpad 136 or microphone 138. The processor 1 12 may control the functioning of the HMD 100 based on inputs received through the user interface 1 15. For example, the processor 1 12 may utilize user input from the user interface 115 to control how the HMD 00 displays images within a field of view or to determine what images the HMD 100 displays.
[Θ849] An eye-tracking system 102 may be included in the HMD 100. In an example embodiment, an eye-tracking system 102 may deliver information to the processor 1 12 regarding the ey e position of a wearer of the HMD 100, The eye-tracking data could be used, for instance, to determine a direction in which the HMD wearer may be gazing. The processor 1 12 could determine target objects among the displayed images based on information from the eye-tracking system 102. The processor 1 12. may control the user interface 1 15 and the display panel 126 to adjust the target object and/or other displayed images in various ways. For instance, a HMD wearer could interact with a mobile-type menu-driven user interface using eye gaze movements.
[0050] The infrared camera 1 16 may be utilized by the eye-tracking system 102 to capture images of a viewing location associated with the HMD 100. Thus, the infrared camera 1 16 may image the eye of a HMD wearer that may be located at the viewing location. The images could be either video images or still images. The images obtained by the infrared camera 1 16 regarding the HMD wearer's eye may help determine where the wearer is looking within the HMD field of view, for instance by allowing the processor 1 12 to ascertain the location of the HMD wearer's eye pupil. Analysis of the images obtained by the infrared camera 1 16 could be performed by the processor 1 12 in conjunction with the memory 1 14 to determine, for example, a gaze direction.
0851] The imaging of the viewing location could occur continuously or at discrete times depending upon, for instance, user interactions with the user interface 1 15 and/or the state of the infrared light source 1 18 which may serve to illuminate the viewing location. The infrared camera 1 16 could be integrated into the optical system 106 or mounted on the HMD 100. Alternatively, the infrared camera could be positioned apart from the HMD 100 altogether. Furthermore, the infrared camera 116 could additionally represent a conventional visible light camera with sensing capabilities in the infrared wavelengths. The infrared
camera 1 16 could be operated at video rate frequency (e.g. 60 Hz) or a multiple of video rates (e.g. 240 Hz), which may be more amenable to combining multiple frames while determining a gaze direction.
[Θ852] The infrared light source 118 could represent one or more infrared light- emitting diodes (LEDs) or infrared laser diodes that may illuminate a viewing location. One or both eyes of a wearer of the HMD 100 may be illuminated by the infrared light source 18. The infrared light source 1 18 may be positioned along an optical axis common to the infrared camera, and/or the infrared light source 1 18 may be positioned elsewhere. The infrared light source 1 18 may illuminate the viewing location continuously or may be turned on at discrete times. Additionally, when illuminated, the infrared light source 1 18 may be modulated at a particular frequency. Other types of modulation of the infrared light source 1 18, such as adjusting the intensity level of the infrared light source 1 18, are possible.
[0053J The eye-tracking system 102 could be configured to acquire images of glint reflections from the outer surface of the cornea, which are also called first Purkinje images. Alternatively, the eye-tracking system 102 could be configured to acquire images of reflections from the inner, posterior surface of the lens, which are termed fourth Purkinje images. In yet another embodiment, the eye -tracking system 102 could be configured to acquire images of the eye pupil with so-called bright and/or dark pupil images. In practice, a combination of these glint and pupil imaging techniques may be used for rotational eye tracking, accuracy, and redundancy. Other imaging and tracking methods are possible. Those knowledgeable in the art will understand that there are several alternative ways to achieve eye tracking with a combination of infrared illuminator and camera hardware,
f>854] The locations of both eyes could be determined optically and/or inferred based on other information in order to determine respective gaze axes and the corresponding vergence angle between the axes. Accordingly, at least one eye- tracking system 102 may be
utilized with one or more infrared cameras 1 16 and one or more infrared light sources 1 18 in order to track the position of one eye or both eyes of the HMD wearer.
[0055] The HMD-tracking system 104 could be configured to provide a HMD position and a HMD orientation to the processor 1 12. This position and orientation data may help determine a central axis to which a gaze direction is compared. For instance, the central axis may correspond to the orientation of the HMD.
[Θ856] The gyroscope 120 could be a niicroelecrromechanical system (MEMS) gyroscope, a fiber optic gyroscope, or another type of gyroscope known in the art. The gyroscope 120 may be configured to provide orientation information to the processor 1 12. The GPS unit 12.2 could be a receiver that obtains clock and other signals from GPS satellites and may be configured to provide real-time location information to the processor 1 12. The HMD-tracking system 104 could further include an accelerometer 124 configured to provide motion input data to the processor 1 12.
[Θ857] The optical system 106 could include components configured to provide images at a viewing location. The viewing location may correspond to the location of one or both eyes of a wearer of a HMD 100. The components could include a display panel 126, a display light source 128, and optics 130. These components may be optically and/or electrically-coupled to one another and may be configured to provide viewable images at a viewing location. As mentioned above, one or two optical systems 106 could be provided in a HMD apparatus. In other words, the HMD wearer could view images in one or both eyes, as provided by one or more optical systems 106. Also, as described above, the optical system(s) 106 could include an opaque display and/or a see-through display, which may- allow a view of the real-world environment while providing superimposed images.
[0058] Various peripheral devices 108 may be included in the HMD 100 and may serve to provide information to and from a wearer of the HMD 100. Tn one example, the
HMD 100 may include a wireless communication interface 134 for wirelessly communicating with one or more devices directly or via a communication network. For example, the wireless communication interface 134 could use 3G cellular communication, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communication, such as WiMAX or LTE. Alternatively, the wireless commumcation interface 134 could communicate with a wireless local area network (WLA.N), for example, using WiFi. In some embodiments, the wireless communication interface 134 could communicate directly with a device, for example, using an infrared link, Bluetooth, or ZigBee. The wireless communication interface 134 could interact with devices that may include, for example, components of the HMD 100 and/or externally-located devices.
[0059] Although Figure 1 shows various components of the HMD 100 (i.e., wireless commumcation interface 134, processor 1 12, memory 1 14, infrared camera 1 1 6, display panel 126, GPS 122, and user interface 1 15) as being integrated into HMD 100, one or more of these components could be physically separate from HMD 100, For example, the infrared camera 1 1 6 could be mounted on the wearer separate from HMD 100. Thus, the HMD 100 could be part of a wearable computing device in the form of separate devices that can be worn on or carried by the wearer. The separate components that make up the wearable computing device could be communicatively coupled together in either a wired or wireless fashion,
[Θ86 ] Figures 2A and 2B illustrate two of many possible embodiments involving head-mounted displays with gaze axis vergence determination. In general, the example systems could be used to receive, transmit, and display data, in one embodiment, the HMD 200 may have a glasses format. As illustrated in Figure 2.A, the HMD 200 has a frame 202 that could include nosepiece 224 and earpieces 218 and 220. The frame 202, nosepieee 224, and earpieces 21 8 and 220 could be configured to secure the HMD 200 to a user's face via a
user's nose and ears. Each of the frame elements, 202 , 224, and 218 may be formed of a solid structure of plastic and/or metal, or may be formed of a hollow structure of similar material so as io allow wiring and component interconnects to be internally routed through the HMD 200. Other materials may be possible as well.
[8861] The earpieces 218 and 220 could be attached to projections that extend away from the lens frame 202 and could be positioned behind a user's ears to secure the HMD 2.00 to the user. The projections could further secure the HMD 200 to the user by extending around a rear portion of the user's head. Additionally or alternatively, for example, the HMD 200 could connect to or be affixed within a head-mounted helmet structure. Other possibilities exist as well.
[8862] Lens elements 210 and 212 could be mounted in frame 202. The lens elements 210 and 212 could be formed of any material that can suitably display a projected image or graphic. Each of the lens elements 2.10 and 2.12 could be sufficiently transparent to allow a user to see through the lens element. Combining these two features of the lens elements may facilitate an augmented reality or a heads-up display where the projected image or graphic is superimposed over a real-world view as perceived by the user through Jens elements 2.10 and 212.
[8863] The HMD 200 may include a computer 214, a touch pad 216, a camera 222, and a display 204. The computer 214 is shown to be positioned on the extending side arm of the HMD 200; however, the computer 214 may be provided on other parts of the HMD 200 or may be positioned remote from the HMD 200 (e.g. the computer 214 could be wire- or wirelessly-connected to the HMD 200). The computer 214 could include a processor and memory, for example. The computer 2.14 may be configured to receive and analyze data from the camera 222. and ihe touch pad 216 (and possibly from other sensory devices, user- interfaces, or both) and generate images for output by the lens elements 210 and 212.
[0064] A camera 222 could be positioned on an extending side arm of the HMD 200, however, the camera 222 may be provided on other parts of the HMD 200. The camera 222 may be configured to capture images at various resolutions or at different frame rates. The camera 222 could be configured as a video camera and/or as a still camera. A camera with small form factor, such as those used in cell phones or webcams, for example, may be incorporated into an example embodiment of HMD 200.
[0065] Further, although Figure 2A illustrates one camera 222, more cameras could be used, and each may be configured to capture the same view, or to capture different views. For example camera 222 may be forward facing to capture at least a portion of the real-world view perceived by the user. This forward facing image captured by the camera 2.22 may then be used to generate an augmented reality where computer generated images appear to interact with the real world view perceived by the user.
[0066] Other sensors could be incorporated into HMD 200. Other sensors may include one or more of a gyroscope or an accelerometer, for example. Other sensing devices may be included in HMD 200.
[0067] The touch pad 216 is shown on an extending side arm of the HMD 200. However, the touch pad 216 may be positioned on other parts of the HMD 200. Also, more than one touch pad may be present on the HMD 200. The touch pad 216 may be used by a user to input commands. The touch pad 216 may sense at feast one of a position and a movement of a finger via capacitive sensing, resistance sensing, or a surface acoustic wave process, among other possibilities. The touch pad 216 may be capable of sensing finger movement in a direction parallel or planar to the pad surface, in a direction normal to the pad surface, or both, and may also be capable of sensing a level of pressure applied to the pad surface. The touch pad 216 may be formed of one or more translucent or transparent insulating layers and one or more translucent or transparent conducting layers. Edges of the
touch pad 216 may be formed to have a raised, indented, or roughened surface, so as to provide tactile feedback to a user when the user's finger reaches the edge, or other area, of the touch pad 2.16. If more than one touch pad is present, each touch pad may be operated independently, and may provide a different function.
[88 8] Additionally, the HMD 200 may include eye-tracking systems 206 and 208, which may be configured to track the eye position of each eye of the HMD wearer. The eye- tracking systems 206 and 208 may each mclude one or more infrared light sources and one or more infrared cameras. Each of the eye-tracking systems 206 and 208 could be configured to image one or both of the HMD wearer's eyes. Although two eye-tracking systems are depicted in Figure 2A, other embodiments are possible. For instance, one eye-tracking system could be used to track both eyes of a user.
[0869] Display 204 could represent, for instance, an at least partially reflective surface upon which images could be projected using a projector. The lens elements 210 and 212 could act as a combiner in a light projection system and may include a coating that reflects the light projected onto them from projectors. In some embodiments, a reflective coating may not be used (e.g. when the projectors are scanning laser devices). The images could be thus viewable to a HMD user.
[Θ87Θ] Although the display 204 is depicted as presented to the right eye of the HMD wearer, other example embodiments could include a display for both eyes or a single display viewable by both eyes.
[0071] In alternative embodiments, other types of display elements may be used. For example, the lens elements 210 and 212 could themselves mclude: a transparent or semi- transparent matrix display, such as an electroluminescent display or a liquid crystal display, one or more waveguides for delivering an image to the user's eyes, or other optical elements capable of delivering an in focus near-to-eye image to the user. A corresponding display
driver may be disposed within the frame 202 for driving such a matrix display. Alternatively or additionally, a laser or fight-emitting diode (LED) source and scanning system could be used to draw a raster display directly onto the retina of one or more of the user's eyes. Other possibilities exist as well.
[Θ872] In Figure 2B, a HMD 226 with monocle design is illustrated. The HMD frame
202 could include nosepiece 224 and earpieces 218 and 220. The HMD 226 may include a single display 204 that may be coupled to one of the side arms or the nose piece 2.24, In one example, the single display 204 could be coupled to the inner side (i.e. the side exposed to a portion of a user's head when worn by the user) of the extending side ar of frame 2.02. The display 204 could be posiiioned in front of or proximate to a user's eye when the HMD 200 is worn by a user. The display 204 could be configured to overlay computer-generated graphics upon the user's view of the physical world.
[0073J As in the aforementioned embodiments, eye-tracking systems 2.06 and 208 could be mounted on nosepiece 224. The eye -tracking systems 206 and 208 could be configured to track the eye position of both eyes of a HMD wearer. The HMD 226 could include a computer 214 and a display 2.04 for one eye of the HMD wearer.
[Θ874] Figure 2C illustrates a HMD 228 with a binocular design. In such an embodiment, separate displays could be provided for each eye of a HMD user. For example, displays 204 and 230 could be provided to the right and left eye of the HMD user, respectively. Alternatively, a single display could provide images to both eyes of the HMD user. The images provided to each eye may be different or identical to one another. Further, the images could be provided to each eye in an effort to create a stereoscopic illusion of depth.
[Θ875] Figures 3A and 3B are side and front views of an eye of a HMD user gazing forward and gazing upward, respectively. In the former scenario, when a HMD user may be
gazing forward 300, light sources 308 and 310 could be configured to illuminate the HMD user's eye 302. Glint reflections 314 and 316 from the HMD user's eye 302 could be generated based on the illumination from the light sources 308 and 310, These glint reflections 314 and 316 could be first Purkinje images from reflections from the outer surface of the HMD user's cornea. The glint reflections 314 and 316 as well as the eye pupil 304 could be imaged by a camera 318. Images could be sent to a processor that may, in turn, analyze the glint locations 32.4 and 326 with respect to a coordinate system 320 in order to determine and/or confirm a pupil location 322, In the case where the HMD user may be gazing forward, the pupil location may be determined to be near the center of the reference coordinate system 320. Accordingly, a gaze direction 312 may be determined to be straight ahead. A gaze point may be determined to be at a point along the gaze direction 312.
[Θ876] Figure 3B depicts a scenario 328 where a HMD user is gazing upward. Similar to the aforementioned example, light sources 308 and 310 could induce respective glint reflections 330 and 332 from the HMD user's eye 302. In this scenario, however, the glint reflections 330 and 332 may appear in different locations due to the change in the eye gaze direction of the HMD wearer and asymmetry of the shape of the eye 302. Thus glint reflections 338 and 340 may move with respect to reference coordinate system 320. Image analysis could be used to determine the pupil location 336 within the reference coordinate system 320. From the pupil location 336, a gaze direction 342 may be determined. A gaze point could be determined as a point along the gaze direction 342.
[Θ877] In some embodiments, gaze directions could be optically determined for both eyes, for example, as described above for Figures 3A and 3B. The gaze directions from both eyes could be used to find a vergence angle in a determination of where the user may be gazing. In other embodiments, a gaze direction could be optically determined for only one
eye, and the gaze direction for the other eye could he inferred. The verge ce angle could be determined based on the optically-determined gaze direction and the inferred gaze direction.
[Θ878] A gaze direction could be inferred from head movements. For example, because of the head's natural tendency to keep the eyes centered (i.e., the head lags behind the eyes, but tends to "frame" the subject), it is possible to look for eye fixations that cluster around a certain point (converting fixations into gaze points), and then use other sensors (e.g., one or more of the sensors in HMD-tracking system 104) to detect movement of the head. When that head movement ceases, but the optically-tracked eye remains off-center in one direction, it is possible to infer that the other eye is similarly off-center in the other direction. This is because this pattern of head movements can be indicative of the person's eyes converging on a nearby object, with the person's head ''framing" the nearby object. In that configuration, the gaze directions of both eyes would be at the same angle from the forward direction (defined by the position of the person's head) but from opposite sides.
[Θ879] Figures 4A, 4B, and 4C illustrate scenarios in which the aforementioned system could be applied, ΐη scenario 400, a HMD wearer 402 with first and second eyes (404 and 406) could be in a real-world environment with a partition that may include a wall portion 414 and a glass portion 410. Beyond the partition, a computer monitor 416 may be viewable through the glass portion 410. The computer monitor 416 could be located on a desk 418. The partition and the computer monitor 416 could be located at a first depth plane 412 and a second depth plane 420, with respect to a HMD wearer plane 408.
[ΘΘ8Θ] In Figure 4B, a HMD wearer may be looking at the computer monitor 416 (scenario 428). The eye -tracking data from both eyes of the HMD wearer may allow the processor 1 12. to determine gaze axes 422 and 424. A corresponding vergence angle 426 could be determined. Based on vergence angle 426, processor 1 12 may determine that the HMD wearer is gazing at the computer monitor 416.
[Θ881] As mentioned above, the determination of a vergence angle may help to disambiguate an actual target object from a set of candidate target objects. In scenario 428, the actual target object may be ambiguous if eye-tracking data from only one eye is used or if only HMD-tracking data is used. In either case, it may be unclear whether the HMD wearer is gazing at the glass portion 410 of the partition, the computer monitor 416, or any other object along the single gaze axis. Thus, the vergence angle 426 of the gaze axes 422. and 424 may reduce the uncertainty of objeci selection.
[0882] Figure 4C illustrates how various notifications may be generated upon vergence angle determination. As described above, Figure 4C depicts a HMD wearer 402 with first and second eyes (404 and 406). The HMD wearer could be in a real- world environment that includes a partition that may include a wail portion 414 and a glass portion 410. Beyond the partition, a computer monitor 416 may be viewable through the glass portion 410. The computer monitor 416 could be located on a desk 418. The partition and the computer monitor 416 could be located at a first depth plane 412 and a second depth plane 420, respectively.
[8883] When the vergence angle of the scenario 430 is determined, the system may determine thai the HMD wearer is gazing at the computer monitor 416. Accordingly, notifications in the form of images could be generated. An image message 434 that states, "Partition" could help alert the HMD wearer not to run into it when walking, for instance. A notification 432. that states, "Computer" could help further identify the objeci at which the HMD wearer is gazing. Other target object-dependent notification messages are possible.
[0884] As stated above, the effective distances for determining gaze point using vergence angle may be up to around 3 meters. Therefore, the following methods may be useful in close- to mid-range interactions such as the aforementioned office example. In long
range situations (greater than 3 meters), vergence may he a less useful way to determine gaze depth and/or to select target objects.
3, Method for target object selection using eye tracking and vergence angle determination
[0085] A method 500 is provided for selecting target objects by determining the vergence angle between the gaze axes of the eyes of a HMD wearer. The method could be performed using an apparatus shown in Figures 1-4C and as described above, however, other configurations could be used. Figure 5 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.
[0886] Method step 502 includes optically determining a first gaze direction and a second gaze direction within a field of view provided by a head-mounted display (FIMD). The HMD is configured to display images within the field of view. The first and second gaze directions could represent the gaze axis of each eye of a HMD wearer. The first and second gaze directions could be optically determined using various apparatuses known in the art including the eye -tracking system described above. The HMD may include at least one display configured to generate images viewable to one or both eyes of the HMD wearer.
[0887] Method step 504 includes determining a gaze point based on the vergence angle between the first and second gaze directions. The vergence angle is the angle created when the first and second gaze directions intersect, for instance when the HMD wearer is looking at a nearby object. In general, the vergence angle may strongly indicate the point at which the FIMD wearer is gazing. Thus, by tracking the eye position of both ey es of a HMD wearer, a vergence angle can be determined. Accordingly, a gaze point may be determined from the vergence angle using basic geometric methods.
[θθδδ] Method step 506 includes selecting a target object from the images based on the gaze point and the depth of the target object. The selected target object could have a similar or identical depth as the gaze point. Further, the selected target object could be any member of the set of images displayed by the HMD. The target object selection could be performed immediately upon determination of a gaze point / target object location match, or could take place after a predetermined period of time. For instance, the target object selection could happen once a HMD wearer stares at an image for 500 milliseconds.
4, Method for image adjustment using eye tracking and vergence angle determination
[0889] A method 600 is provided for adjusting images based on a gaze point, which can be determined from a vergence angle between the gaze axes of the eyes of a head- mounted display (HMD) wearer. The method could be performed using an apparatus shown in Figures 1-4C and as described above, however, other configurations could be used. Figure 6 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.
[0090] The first two steps of method 600 (steps 602. and 604) could be similar or identical to the corresponding steps of method 500. In other words, an eye-tracking system or other optical means could be utilized to determine a first gaze direction and a second gaze direction within a field of view of the HMD (step 602). A gaze point ma then be determined based on the vergence angle between the first and second gaze directions (step 604).
[0091] In a third method step 606, images displayed in the field of view for the HMD could be adjusted based on the determined gaze point. The determined gaze point could relate to a target object that could include real-world objects or displayed images. The adjusted images could include any graphical or text element displayed by the HMD. For
instance, the eye-tracking system could determine that a HMD wearer is gazing at a computer screen based on the vergence angle of his or her eyes. Correspondingly, images (such as icons or other notifications) could be adjusted away from the gaze location so as to allow an unobstructed view of the real-world object. The images could be adjusted dynamically, or, for instance, only when a new, contextually-important gaze point is determined.
[0092] In another embodiment, upon recognition that a HMD wearer is gazing at a target object, an image could be displayed that provides information about the target object. In the case that a HMD wearer is gazing at a computer screen in the real-world environment, a notification may be generated. The notification could take the form of an image viewable to the HMD wearer as apparently adjacent to the computer screen. The notification could include specific information about the computer such as machine owner, model number, operating state, etc. Other notification types and content are possible.
S, A non-transitory computer readable medium for target objec selection using eye tracking ami vergesice angle determination
[0093] Some or all of the functions described above in method 500, method 600 and illustrated in Figures 3A, 3B, 4A, 4B, and 4C, may be performed by a computing device in response to the execution of instructions stored in a non-transitory computer readable medium. The non-transitory computer readable medium could be, for example, a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache memory, one or more magnetically encoded discs, one or more optically encoded discs, or any other form of non-transitory data storage. The non-transitory computer readable medium could also be distributed among multiple data storage elements, which could be remotely located from each other. The computing device that executes the stored instructions could be a wearable computing device, such as a wearable computing device 100 illustrated in Figure I .
Alternatively, the computing device that executes the stored instructions could be another computing device, such as a server in a server network. A non-transitory computer readable medium may store instxuciions executable by the processor 1 12 to perform various functions.
[0894] For instance, instructions that could be used to carry out method 500 may be stored in memory 1 14 and could be executed by processor 1 12. in such an embodiment, upon receiving gaze information from the eye-tracking system 102, the processor 1 12. may carry out instructions to determine a gaze axis for both eyes of a user. Accordingly, a vergence angle may be calculated. Based on at least the determined vergence angle, a target object may be selected fro the set of displayed images.
[0895] Those wiih skill in the art will understand that many oiher instxuciions may be stored by a non-transitory computer readable medium that may relate to the determination of a vergence angle to enhance and/or modify interactions with real world objects and/or displayed images. These other examples are implicitly considered herein.
CONCLUSION
[Θ896] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.
Claims
1. A wearable computing device, comprising:
a head-mounted display (HMD), wherein the HMD is configured to display images, wherein the images are viewable from at least one of a first viewing location or a second viewing location;
at least one infrared light source, wherein the at least one infrared light source is configured to illuminate at least one of the fsrst viewing location or the second viewing location with infrared light such that the infrared light is reflected from the at least one illuminated viewing location as reflected infrared light;
at least one camera, wherein the at least one camera is configured to acquire at least one image of the at least one illuminated viewing location by collecting the reflected infrared light; and
a computer, wherein the computer is configured to determine a vergence angle based on the at least one image of the at least one illuminated viewing location, determine a gaze point based on the vergence angle, select an image based on the gaze point, and control the HMD to display the selected image. 2. The wearable computing device of Claim 1, wherein the HMD comprises a see-through display.
3. The wearable computing device of Claim 1 , wherein the HMD comprises a binocular display.
4. The wearable computing device of Claim 1, wherem the HMD comprises a monocular display.
5. The wearable computing device of Claim 1, wherein the at least one camera is mounted on the HMD,
6. The wearable computing device of Claim 1, wherein the at least one infrared light source is an infrared light-emitting diode (LED),
7. The wearable computing device of Claim 1, wherein ihe at least one infrared light source is mounted on the HMD.
8. The wearable computing device of Claim 1 , wherein the at least one camera is an infrared camera.
9. A method, comprising:
optically determining a first gaze direction and a second gaze direction within a field of view provided by a head-mounted display (HMD), wherein the HMD is configured to display images within the field of view;
determining a gaze point based on a vergence angie between the first and second gaze directions; and
selecting a target object from the images based on the gaze point and a depth of the target object.
10. The method of Claim 9, wherein optically determining a first and second gaze direction comprises:
obtaining at least one image of each eye of a wearer of the HMD: and
determining the first and second gaze direction from the at least one image of each eye.
11. The method of Claim 10, wherein obtaining at least one image of each eye of a wearer of the HMD comprises illuminating each eye with an infrared fight source and imaging each eye with a camera.
12. The method of Claim 9, wherein determining a gaze point comprises determining an intersection of the first and second gaze directions and determining a gaze point based on the intersection and an HMD position.
13. A method, comprising:
optically determining a first gaze direction and a second gaze direction within a field of view pro vided by a head-mounted display (HMD), wherein the HMD is configured to display images within the field of view;
determining a gaze point based on a vergence angle between the first and second gaze directions; and
adj sting the images based on the gaze point.
14. The method of Claim 13, wherein optically determining a first and second gaze direction comprises:
obtaining at least one image of each eye of a wearer of the HMD; and determining the first and second gaze direction from the at least one image of each eye.
15. The method of Claim 14, wherein obtaining at least one image of each eye of a wearer of the HMD comprises illuminating each eye with an infrared light source and imaging each eye with a camera.
16. The method of Claim 13, wherein determining a gaze point comprises determining an intersection of the fsrst and second gaze directions and determining a gaze point based on the intersection and an HMD position.
17. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions comprising:
causing a head-mounted display (HMD) to acquire images of first and second viewing locations, wherein the HMD is configured to display images;
determining a first gaze direction and a second gaze direction based on the images of the first and second viewing locations;
determining a gaze point based on a vergence angle between the first and second gaze directions; and
selecting a target object from the images based on the gaze point and a depth of the target object.
18. The non-transitory computer readable medium of Claim 17, wherein causing the HMD to acquire images of first and second viewing locations comprises acquiring at least one image of each eye of a wearer of the HMD
19, The non-transitory computer readable medium of Claim 18, wherein acquiring at least one image of each eye of a wearer of the HMD comprises illuminating each eye with an infrared source and imaging each eye with a camera,
2.0. The non-transitory computer readable medium of Claim 17, wherein determining a gaze point further comprises determining an intersection of the first and second gaze directions and determining a gaze point based on the intersection and an HMD position.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261611188P | 2012-03-15 | 2012-03-15 | |
US61/611,188 | 2012-03-15 | ||
US13/566,494 US20130241805A1 (en) | 2012-03-15 | 2012-08-03 | Using Convergence Angle to Select Among Different UI Elements |
US13/566,494 | 2012-08-03 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013138647A1 true WO2013138647A1 (en) | 2013-09-19 |
Family
ID=49157130
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2013/031632 WO2013138647A1 (en) | 2012-03-15 | 2013-03-14 | Using convergence angle to select among different ui elements |
Country Status (2)
Country | Link |
---|---|
US (1) | US20130241805A1 (en) |
WO (1) | WO2013138647A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10067561B2 (en) | 2014-09-22 | 2018-09-04 | Facebook, Inc. | Display visibility based on eye convergence |
WO2018237172A1 (en) * | 2017-06-21 | 2018-12-27 | Quantum Interface, Llc | Systems, apparatuses, interfaces, and methods for virtual control constructs, eye movement object controllers, and virtual training |
CN109890266A (en) * | 2016-11-04 | 2019-06-14 | 三星电子株式会社 | Method and apparatus for obtaining information by capturing eyes |
EP3521978A1 (en) * | 2018-02-05 | 2019-08-07 | Samsung Electronics Co., Ltd. | Apparatus and method for tracking a focal point in a head mounted display system |
Families Citing this family (173)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10258259B1 (en) | 2008-08-29 | 2019-04-16 | Gary Zets | Multimodal sensory feedback system and method for treatment and assessment of disequilibrium, balance and motion disorders |
US11273344B2 (en) | 2007-09-01 | 2022-03-15 | Engineering Acoustics Incorporated | Multimodal sensory feedback system and method for treatment and assessment of disequilibrium, balance and motion disorders |
US9400390B2 (en) | 2014-01-24 | 2016-07-26 | Osterhout Group, Inc. | Peripheral lighting for head worn computing |
US9229233B2 (en) | 2014-02-11 | 2016-01-05 | Osterhout Group, Inc. | Micro Doppler presentations in head worn computing |
US20150205111A1 (en) | 2014-01-21 | 2015-07-23 | Osterhout Group, Inc. | Optical configurations for head worn computing |
US9298007B2 (en) | 2014-01-21 | 2016-03-29 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US9952664B2 (en) * | 2014-01-21 | 2018-04-24 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US9715112B2 (en) | 2014-01-21 | 2017-07-25 | Osterhout Group, Inc. | Suppression of stray light in head worn computing |
US9965681B2 (en) | 2008-12-16 | 2018-05-08 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US20130339859A1 (en) | 2012-06-15 | 2013-12-19 | Muzik LLC | Interactive networked headphones |
US9619911B2 (en) | 2012-11-13 | 2017-04-11 | Qualcomm Incorporated | Modifying virtual object display properties |
US10359841B2 (en) * | 2013-01-13 | 2019-07-23 | Qualcomm Incorporated | Apparatus and method for controlling an augmented reality device |
WO2014115387A1 (en) * | 2013-01-28 | 2014-07-31 | ソニー株式会社 | Information processor, information processing method and program |
US20140280503A1 (en) | 2013-03-15 | 2014-09-18 | John Cronin | System and methods for effective virtual reality visitor interface |
US20140280506A1 (en) | 2013-03-15 | 2014-09-18 | John Cronin | Virtual reality enhanced through browser connections |
US20140280502A1 (en) | 2013-03-15 | 2014-09-18 | John Cronin | Crowd and cloud enabled virtual reality distributed location network |
US20140280644A1 (en) | 2013-03-15 | 2014-09-18 | John Cronin | Real time unified communications interaction of a predefined location in a virtual reality location |
US9838506B1 (en) | 2013-03-15 | 2017-12-05 | Sony Interactive Entertainment America Llc | Virtual reality universe representation changes viewing based upon client side parameters |
US20140267581A1 (en) | 2013-03-15 | 2014-09-18 | John Cronin | Real time virtual reality leveraging web cams and ip cams and web cam and ip cam networks |
US9383819B2 (en) * | 2013-06-03 | 2016-07-05 | Daqri, Llc | Manipulation of virtual object in augmented reality via intent |
CN103353677B (en) | 2013-06-28 | 2015-03-11 | 北京智谷睿拓技术服务有限公司 | Imaging device and method thereof |
CN103353663B (en) | 2013-06-28 | 2016-08-10 | 北京智谷睿拓技术服务有限公司 | Imaging adjusting apparatus and method |
CN103353667B (en) | 2013-06-28 | 2015-10-21 | 北京智谷睿拓技术服务有限公司 | Imaging adjustment Apparatus and method for |
US20150003819A1 (en) * | 2013-06-28 | 2015-01-01 | Nathan Ackerman | Camera auto-focus based on eye gaze |
CN103431840B (en) | 2013-07-31 | 2016-01-20 | 北京智谷睿拓技术服务有限公司 | Eye optical parameter detecting system and method |
CN103424891B (en) | 2013-07-31 | 2014-12-17 | 北京智谷睿拓技术服务有限公司 | Imaging device and method |
KR20150020918A (en) * | 2013-08-19 | 2015-02-27 | 엘지전자 주식회사 | Display device and control method thereof |
CN103431980A (en) | 2013-08-22 | 2013-12-11 | 北京智谷睿拓技术服务有限公司 | Eyesight protection imaging system and method |
CN103439801B (en) | 2013-08-22 | 2016-10-26 | 北京智谷睿拓技术服务有限公司 | Sight protectio imaging device and method |
CN103605208B (en) | 2013-08-30 | 2016-09-28 | 北京智谷睿拓技术服务有限公司 | content projection system and method |
CN103500331B (en) * | 2013-08-30 | 2017-11-10 | 北京智谷睿拓技术服务有限公司 | Based reminding method and device |
JP5420793B1 (en) * | 2013-09-10 | 2014-02-19 | テレパシー インク | Head-mounted display with adjustable image viewing distance |
CN103558909B (en) | 2013-10-10 | 2017-03-29 | 北京智谷睿拓技术服务有限公司 | Interaction projection display packing and interaction projection display system |
CN105814516B (en) * | 2013-10-11 | 2019-07-30 | 交互数字专利控股公司 | Watch the augmented reality of driving attentively |
CN103698884A (en) | 2013-12-12 | 2014-04-02 | 京东方科技集团股份有限公司 | Opening type head-mounted display device and display method thereof |
US9761051B2 (en) | 2013-12-26 | 2017-09-12 | Empire Technology Development Llc | Out-of focus micromirror to display augmented reality images |
US9696552B1 (en) | 2014-01-10 | 2017-07-04 | Lockheed Martin Corporation | System and method for providing an augmented reality lightweight clip-on wearable device |
US9810906B2 (en) | 2014-06-17 | 2017-11-07 | Osterhout Group, Inc. | External user interface for head worn computing |
US10254856B2 (en) | 2014-01-17 | 2019-04-09 | Osterhout Group, Inc. | External user interface for head worn computing |
US11227294B2 (en) | 2014-04-03 | 2022-01-18 | Mentor Acquisition One, Llc | Sight information collection in head worn computing |
US9841599B2 (en) | 2014-06-05 | 2017-12-12 | Osterhout Group, Inc. | Optical configurations for head-worn see-through displays |
US9829707B2 (en) | 2014-08-12 | 2017-11-28 | Osterhout Group, Inc. | Measuring content brightness in head worn computing |
US11103122B2 (en) | 2014-07-15 | 2021-08-31 | Mentor Acquisition One, Llc | Content presentation in head worn computing |
US20150277118A1 (en) | 2014-03-28 | 2015-10-01 | Osterhout Group, Inc. | Sensor dependent content position in head worn computing |
US9575321B2 (en) | 2014-06-09 | 2017-02-21 | Osterhout Group, Inc. | Content presentation in head worn computing |
US9299194B2 (en) | 2014-02-14 | 2016-03-29 | Osterhout Group, Inc. | Secure sharing in head worn computing |
US20160019715A1 (en) | 2014-07-15 | 2016-01-21 | Osterhout Group, Inc. | Content presentation in head worn computing |
US9746686B2 (en) | 2014-05-19 | 2017-08-29 | Osterhout Group, Inc. | Content position calibration in head worn computing |
US9529195B2 (en) | 2014-01-21 | 2016-12-27 | Osterhout Group, Inc. | See-through computer display systems |
US10191279B2 (en) | 2014-03-17 | 2019-01-29 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US10649220B2 (en) | 2014-06-09 | 2020-05-12 | Mentor Acquisition One, Llc | Content presentation in head worn computing |
US9448409B2 (en) | 2014-11-26 | 2016-09-20 | Osterhout Group, Inc. | See-through computer display systems |
US9939934B2 (en) | 2014-01-17 | 2018-04-10 | Osterhout Group, Inc. | External user interface for head worn computing |
US10684687B2 (en) | 2014-12-03 | 2020-06-16 | Mentor Acquisition One, Llc | See-through computer display systems |
US9671613B2 (en) | 2014-09-26 | 2017-06-06 | Osterhout Group, Inc. | See-through computer display systems |
US9594246B2 (en) | 2014-01-21 | 2017-03-14 | Osterhout Group, Inc. | See-through computer display systems |
US9532715B2 (en) | 2014-01-21 | 2017-01-03 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US9811159B2 (en) | 2014-01-21 | 2017-11-07 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US20150205135A1 (en) | 2014-01-21 | 2015-07-23 | Osterhout Group, Inc. | See-through computer display systems |
US9753288B2 (en) | 2014-01-21 | 2017-09-05 | Osterhout Group, Inc. | See-through computer display systems |
US9766463B2 (en) | 2014-01-21 | 2017-09-19 | Osterhout Group, Inc. | See-through computer display systems |
US9529199B2 (en) | 2014-01-21 | 2016-12-27 | Osterhout Group, Inc. | See-through computer display systems |
US9836122B2 (en) * | 2014-01-21 | 2017-12-05 | Osterhout Group, Inc. | Eye glint imaging in see-through computer display systems |
US11487110B2 (en) | 2014-01-21 | 2022-11-01 | Mentor Acquisition One, Llc | Eye imaging in head worn computing |
US11669163B2 (en) | 2014-01-21 | 2023-06-06 | Mentor Acquisition One, Llc | Eye glint imaging in see-through computer display systems |
US11737666B2 (en) | 2014-01-21 | 2023-08-29 | Mentor Acquisition One, Llc | Eye imaging in head worn computing |
US9494800B2 (en) | 2014-01-21 | 2016-11-15 | Osterhout Group, Inc. | See-through computer display systems |
US11892644B2 (en) | 2014-01-21 | 2024-02-06 | Mentor Acquisition One, Llc | See-through computer display systems |
US9651784B2 (en) | 2014-01-21 | 2017-05-16 | Osterhout Group, Inc. | See-through computer display systems |
US20150215612A1 (en) * | 2014-01-24 | 2015-07-30 | Ganesh Gopal Masti Jayaram | Global Virtual Reality Experience System |
US9846308B2 (en) | 2014-01-24 | 2017-12-19 | Osterhout Group, Inc. | Haptic systems for head-worn computers |
US9588343B2 (en) | 2014-01-25 | 2017-03-07 | Sony Interactive Entertainment America Llc | Menu navigation in a head-mounted display |
US9401540B2 (en) | 2014-02-11 | 2016-07-26 | Osterhout Group, Inc. | Spatial location presentation in head worn computing |
US20150241964A1 (en) | 2014-02-11 | 2015-08-27 | Osterhout Group, Inc. | Eye imaging in head worn computing |
US9773349B2 (en) * | 2014-02-19 | 2017-09-26 | Daqri, Llc | Active parallax correction |
EP3108801A4 (en) * | 2014-02-21 | 2017-10-25 | Sony Corporation | Head-mounted display, control device, and control method |
US20160187651A1 (en) | 2014-03-28 | 2016-06-30 | Osterhout Group, Inc. | Safety for a vehicle operator with an hmd |
US10853589B2 (en) | 2014-04-25 | 2020-12-01 | Mentor Acquisition One, Llc | Language translation with head-worn computing |
US9672210B2 (en) | 2014-04-25 | 2017-06-06 | Osterhout Group, Inc. | Language translation with head-worn computing |
US9423842B2 (en) | 2014-09-18 | 2016-08-23 | Osterhout Group, Inc. | Thermal management for head-worn computer |
US9651787B2 (en) | 2014-04-25 | 2017-05-16 | Osterhout Group, Inc. | Speaker assembly for headworn computer |
US20150339855A1 (en) * | 2014-05-20 | 2015-11-26 | International Business Machines Corporation | Laser pointer selection for augmented reality devices |
US10663740B2 (en) | 2014-06-09 | 2020-05-26 | Mentor Acquisition One, Llc | Content presentation in head worn computing |
US10198865B2 (en) | 2014-07-10 | 2019-02-05 | Seiko Epson Corporation | HMD calibration with direct geometric modeling |
WO2016017945A1 (en) | 2014-07-29 | 2016-02-04 | Samsung Electronics Co., Ltd. | Mobile device and method of pairing the same with electronic device |
KR102437104B1 (en) * | 2014-07-29 | 2022-08-29 | 삼성전자주식회사 | Mobile device and method for pairing with electric device |
WO2016017997A1 (en) | 2014-07-31 | 2016-02-04 | Samsung Electronics Co., Ltd. | Wearable glasses and method of providing content using the same |
US9829708B1 (en) * | 2014-08-19 | 2017-11-28 | Boston Incubator Center, LLC | Method and apparatus of wearable eye pointing system |
US9699436B2 (en) * | 2014-09-16 | 2017-07-04 | Microsoft Technology Licensing, Llc | Display with eye-discomfort reduction |
KR102194787B1 (en) | 2014-09-24 | 2020-12-24 | 삼성전자주식회사 | Apparatus and method for user based sensor data acquiring |
US9984505B2 (en) * | 2014-09-30 | 2018-05-29 | Sony Interactive Entertainment Inc. | Display of text information on a head-mounted display |
WO2016055317A1 (en) * | 2014-10-06 | 2016-04-14 | Koninklijke Philips N.V. | Docking system |
FR3028326B1 (en) | 2014-11-07 | 2018-08-17 | Thales | HEAD VISUALIZATION SYSTEM COMPRISING AN EYE-CATCH SYSTEM AND MEANS FOR ADAPTING THE IMAGES EMITTED |
US9684172B2 (en) | 2014-12-03 | 2017-06-20 | Osterhout Group, Inc. | Head worn computer display systems |
US10248192B2 (en) * | 2014-12-03 | 2019-04-02 | Microsoft Technology Licensing, Llc | Gaze target application launcher |
USD751552S1 (en) | 2014-12-31 | 2016-03-15 | Osterhout Group, Inc. | Computer glasses |
USD753114S1 (en) | 2015-01-05 | 2016-04-05 | Osterhout Group, Inc. | Air mouse |
KR102325883B1 (en) | 2015-01-13 | 2021-11-11 | 매직 립, 인코포레이티드 | Improved sequential color display |
US20160239985A1 (en) | 2015-02-17 | 2016-08-18 | Osterhout Group, Inc. | See-through computer display systems |
WO2016168788A2 (en) | 2015-04-17 | 2016-10-20 | Tulip Interfaces, Inc. | Containerized communications gateway |
CA2986144A1 (en) * | 2015-05-20 | 2016-11-24 | Magic Leap, Inc. | Tilt shift iris imaging |
US10159411B2 (en) | 2015-06-14 | 2018-12-25 | Facense Ltd. | Detecting irregular physiological responses during exposure to sensitive data |
US10076270B2 (en) | 2015-06-14 | 2018-09-18 | Facense Ltd. | Detecting physiological responses while accounting for touching the face |
US10045699B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Determining a state of a user based on thermal measurements of the forehead |
US10130261B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Detecting physiological responses while taking into account consumption of confounding substances |
US10151636B2 (en) | 2015-06-14 | 2018-12-11 | Facense Ltd. | Eyeglasses having inward-facing and outward-facing thermal cameras |
US10076250B2 (en) | 2015-06-14 | 2018-09-18 | Facense Ltd. | Detecting physiological responses based on multispectral data from head-mounted cameras |
US10092232B2 (en) | 2015-06-14 | 2018-10-09 | Facense Ltd. | User state selection based on the shape of the exhale stream |
US10130308B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Calculating respiratory parameters from thermal measurements |
US10085685B2 (en) | 2015-06-14 | 2018-10-02 | Facense Ltd. | Selecting triggers of an allergic reaction based on nasal temperatures |
US10064559B2 (en) | 2015-06-14 | 2018-09-04 | Facense Ltd. | Identification of the dominant nostril using thermal measurements |
US10136852B2 (en) | 2015-06-14 | 2018-11-27 | Facense Ltd. | Detecting an allergic reaction from nasal temperatures |
US10299717B2 (en) | 2015-06-14 | 2019-05-28 | Facense Ltd. | Detecting stress based on thermal measurements of the face |
US10523852B2 (en) | 2015-06-14 | 2019-12-31 | Facense Ltd. | Wearable inward-facing camera utilizing the Scheimpflug principle |
US10165949B2 (en) | 2015-06-14 | 2019-01-01 | Facense Ltd. | Estimating posture using head-mounted cameras |
US10130299B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Neurofeedback eyeglasses |
US10154810B2 (en) | 2015-06-14 | 2018-12-18 | Facense Ltd. | Security system that detects atypical behavior |
US9968264B2 (en) | 2015-06-14 | 2018-05-15 | Facense Ltd. | Detecting physiological responses based on thermal asymmetry of the face |
US10045726B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Selecting a stressor based on thermal measurements of the face |
US10216981B2 (en) | 2015-06-14 | 2019-02-26 | Facense Ltd. | Eyeglasses that measure facial skin color changes |
US10113913B2 (en) | 2015-10-03 | 2018-10-30 | Facense Ltd. | Systems for collecting thermal measurements of the face |
US10045737B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Clip-on device with inward-facing cameras |
US10080861B2 (en) | 2015-06-14 | 2018-09-25 | Facense Ltd. | Breathing biofeedback eyeglasses |
US10136856B2 (en) | 2016-06-27 | 2018-11-27 | Facense Ltd. | Wearable respiration measurements system |
US10192133B2 (en) | 2015-06-22 | 2019-01-29 | Seiko Epson Corporation | Marker, method of detecting position and pose of marker, and computer program |
US10192361B2 (en) | 2015-07-06 | 2019-01-29 | Seiko Epson Corporation | Head-mounted display device and computer program |
US10757399B2 (en) | 2015-09-10 | 2020-08-25 | Google Llc | Stereo rendering system |
WO2017059522A1 (en) * | 2015-10-05 | 2017-04-13 | Esight Corp. | Methods for near-to-eye displays exploiting optical focus and depth information extraction |
US10466780B1 (en) * | 2015-10-26 | 2019-11-05 | Pillantas | Systems and methods for eye tracking calibration, eye vergence gestures for interface control, and visual aids therefor |
WO2017075100A1 (en) | 2015-10-26 | 2017-05-04 | Pillantas Inc. | Systems and methods for eye vergence control |
US9983709B2 (en) | 2015-11-02 | 2018-05-29 | Oculus Vr, Llc | Eye tracking using structured light |
WO2017079342A1 (en) * | 2015-11-02 | 2017-05-11 | Focure, Inc. | Continuous autofocusing eyewear |
US9709807B2 (en) | 2015-11-03 | 2017-07-18 | Motorola Solutions, Inc. | Out of focus notifications |
US10424117B2 (en) * | 2015-12-02 | 2019-09-24 | Seiko Epson Corporation | Controlling a display of a head-mounted display device |
JP6518582B2 (en) * | 2015-12-21 | 2019-05-22 | 株式会社ソニー・インタラクティブエンタテインメント | Information processing apparatus and operation reception method |
JP6547900B2 (en) * | 2016-03-23 | 2019-07-24 | 日本電気株式会社 | Glasses-type wearable terminal, control method thereof and control program |
US10643390B2 (en) * | 2016-03-30 | 2020-05-05 | Seiko Epson Corporation | Head mounted display, method for controlling head mounted display, and computer program |
US10089000B2 (en) | 2016-06-03 | 2018-10-02 | Microsoft Technology Licensing, Llc | Auto targeting assistance for input devices |
US10223067B2 (en) | 2016-07-15 | 2019-03-05 | Microsoft Technology Licensing, Llc | Leveraging environmental context for enhanced communication throughput |
US10869026B2 (en) * | 2016-11-18 | 2020-12-15 | Amitabha Gupta | Apparatus for augmenting vision |
US10393312B2 (en) | 2016-12-23 | 2019-08-27 | Realwear, Inc. | Articulating components for a head-mounted display |
US10620910B2 (en) * | 2016-12-23 | 2020-04-14 | Realwear, Inc. | Hands-free navigation of touch-based operating systems |
US11507216B2 (en) | 2016-12-23 | 2022-11-22 | Realwear, Inc. | Customizing user interfaces of binary applications |
US11099716B2 (en) | 2016-12-23 | 2021-08-24 | Realwear, Inc. | Context based content navigation for wearable display |
US10437070B2 (en) | 2016-12-23 | 2019-10-08 | Realwear, Inc. | Interchangeable optics for a head-mounted display |
US10936872B2 (en) | 2016-12-23 | 2021-03-02 | Realwear, Inc. | Hands-free contextually aware object interaction for wearable display |
US10679366B1 (en) | 2017-01-30 | 2020-06-09 | Facebook Technologies, Llc | High speed computational tracking sensor |
JP2018137505A (en) * | 2017-02-20 | 2018-08-30 | セイコーエプソン株式会社 | Display device and control method thereof |
WO2018156523A1 (en) * | 2017-02-21 | 2018-08-30 | Oculus Vr, Llc | Focus adjusting multiplanar head mounted display |
US20180373348A1 (en) * | 2017-06-22 | 2018-12-27 | Microsoft Technology Licensing, Llc | Systems and methods of active brightness depth calculation for object tracking |
EP3336848B1 (en) * | 2017-08-15 | 2023-09-27 | Siemens Healthcare GmbH | Method for operating a medical imaging device and medical imaging device |
US11112606B2 (en) * | 2017-09-20 | 2021-09-07 | Facebook Technologies, Llc | Multiple layer projector for a head-mounted display |
WO2019067902A1 (en) | 2017-09-29 | 2019-04-04 | Apple Inc. | Gaze-based user interactions |
CN109725418B (en) * | 2017-10-30 | 2020-10-16 | 华为技术有限公司 | Display device, method and device for adjusting image presentation of display device |
US11181977B2 (en) * | 2017-11-17 | 2021-11-23 | Dolby Laboratories Licensing Corporation | Slippage compensation in eye tracking |
US10732826B2 (en) * | 2017-11-22 | 2020-08-04 | Microsoft Technology Licensing, Llc | Dynamic device interaction adaptation based on user engagement |
CN108592865A (en) * | 2018-04-28 | 2018-09-28 | 京东方科技集团股份有限公司 | Geometric measurement method and its device, AR equipment based on AR equipment |
WO2019217081A1 (en) * | 2018-05-09 | 2019-11-14 | Apple Inc. | Selecting a text input field using eye gaze |
US11170521B1 (en) * | 2018-09-27 | 2021-11-09 | Apple Inc. | Position estimation based on eye gaze |
US20200125169A1 (en) * | 2018-10-18 | 2020-04-23 | Eyetech Digital Systems, Inc. | Systems and Methods for Correcting Lens Distortion in Head Mounted Displays |
US10855979B2 (en) | 2018-10-23 | 2020-12-01 | Microsoft Technology Licensing, Llc | Interpreting eye gaze direction as user input to near-eye-display (NED) devices for enabling hands free positioning of virtual items |
US10718942B2 (en) | 2018-10-23 | 2020-07-21 | Microsoft Technology Licensing, Llc | Eye tracking systems and methods for near-eye-display (NED) devices |
US10838490B2 (en) | 2018-10-23 | 2020-11-17 | Microsoft Technology Licensing, Llc | Translating combinations of user gaze direction and predetermined facial gestures into user input instructions for near-eye-display (NED) devices |
US10996746B2 (en) | 2018-10-23 | 2021-05-04 | Microsoft Technology Licensing, Llc | Real-time computational solutions to a three-dimensional eye tracking framework |
US10852823B2 (en) | 2018-10-23 | 2020-12-01 | Microsoft Technology Licensing, Llc | User-specific eye tracking calibration for near-eye-display (NED) devices |
US11022794B2 (en) * | 2018-12-27 | 2021-06-01 | Facebook Technologies, Llc | Visual indicators of user attention in AR/VR environment |
US10990171B2 (en) | 2018-12-27 | 2021-04-27 | Facebook Technologies, Llc | Audio indicators of user attention in AR/VR environment |
WO2020240864A1 (en) * | 2019-05-31 | 2020-12-03 | 日本電信電話株式会社 | Distance estimation device, distance estimation method, and distance estimation program |
EP3851939A1 (en) * | 2020-01-14 | 2021-07-21 | Apple Inc. | Positioning a user-controlled spatial selector based on extremity tracking information and eye tracking information |
CN116113912A (en) * | 2020-06-22 | 2023-05-12 | 苹果公司 | Event routing in a 3D graphics environment |
US11662574B2 (en) * | 2020-11-10 | 2023-05-30 | Zinn Labs, Inc. | Determining gaze depth using eye tracking functions |
US11269411B1 (en) * | 2020-12-10 | 2022-03-08 | A9.Com, Inc. | Gaze dependent ocular mode controller for mixed reality |
CN114675417A (en) * | 2020-12-24 | 2022-06-28 | 华为技术有限公司 | Display module and virtual image position adjusting method and device |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050206583A1 (en) * | 1996-10-02 | 2005-09-22 | Lemelson Jerome H | Selectively controllable heads-up display system |
US20060250322A1 (en) * | 2005-05-09 | 2006-11-09 | Optics 1, Inc. | Dynamic vergence and focus control for head-mounted displays |
KR20090052169A (en) * | 2007-11-20 | 2009-05-25 | 삼성전자주식회사 | Head-mounted display |
JP2009157634A (en) * | 2007-12-26 | 2009-07-16 | Fuji Xerox Co Ltd | Irradiation control device, irradiation control program, and visual line analysis system |
US20110043644A1 (en) * | 2008-04-02 | 2011-02-24 | Esight Corp. | Apparatus and Method for a Dynamic "Region of Interest" in a Display System |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6658568B1 (en) * | 1995-02-13 | 2003-12-02 | Intertrust Technologies Corporation | Trusted infrastructure support system, methods and techniques for secure electronic commerce transaction and rights management |
US6578962B1 (en) * | 2001-04-27 | 2003-06-17 | International Business Machines Corporation | Calibration-free eye gaze tracking |
WO2003079272A1 (en) * | 2002-03-15 | 2003-09-25 | University Of Washington | Materials and methods for simulating focal shifts in viewers using large depth of focus displays |
US20050047629A1 (en) * | 2003-08-25 | 2005-03-03 | International Business Machines Corporation | System and method for selectively expanding or contracting a portion of a display using eye-gaze tracking |
US7561143B1 (en) * | 2004-03-19 | 2009-07-14 | The University of the Arts | Using gaze actions to interact with a display |
JP4707368B2 (en) * | 2004-06-25 | 2011-06-22 | 雅貴 ▲吉▼良 | Stereoscopic image creation method and apparatus |
US7591558B2 (en) * | 2006-05-31 | 2009-09-22 | Sony Ericsson Mobile Communications Ab | Display based on eye information |
US8494229B2 (en) * | 2008-02-14 | 2013-07-23 | Nokia Corporation | Device and method for determining gaze direction |
US20100149073A1 (en) * | 2008-11-02 | 2010-06-17 | David Chaum | Near to Eye Display System and Appliance |
US7736000B2 (en) * | 2008-08-27 | 2010-06-15 | Locarna Systems, Inc. | Method and apparatus for tracking eye movement |
US20100240988A1 (en) * | 2009-03-19 | 2010-09-23 | Kenneth Varga | Computer-aided system for 360 degree heads up display of safety/mission critical data |
US8977489B2 (en) * | 2009-05-18 | 2015-03-10 | GM Global Technology Operations LLC | Turn by turn graphical navigation on full windshield head-up display |
US8482562B2 (en) * | 2009-12-03 | 2013-07-09 | International Business Machines Corporation | Vision-based computer control |
US20110213664A1 (en) * | 2010-02-28 | 2011-09-01 | Osterhout Group, Inc. | Local advertising content on an interactive head-mounted eyepiece |
US8275375B2 (en) * | 2010-03-25 | 2012-09-25 | Jong Hyup Lee | Data integration for wireless network systems |
US8487787B2 (en) * | 2010-09-30 | 2013-07-16 | Honeywell International Inc. | Near-to-eye head tracking ground obstruction system and method |
US9292973B2 (en) * | 2010-11-08 | 2016-03-22 | Microsoft Technology Licensing, Llc | Automatic variable virtual focus for augmented reality displays |
WO2013028813A1 (en) * | 2011-08-23 | 2013-02-28 | Microsoft Corporation | Implicit sharing and privacy control through physical behaviors using sensor-rich devices |
US9183807B2 (en) * | 2011-12-07 | 2015-11-10 | Microsoft Technology Licensing, Llc | Displaying virtual data as printed content |
US8963805B2 (en) * | 2012-01-27 | 2015-02-24 | Microsoft Corporation | Executable virtual objects associated with real objects |
-
2012
- 2012-08-03 US US13/566,494 patent/US20130241805A1/en not_active Abandoned
-
2013
- 2013-03-14 WO PCT/US2013/031632 patent/WO2013138647A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050206583A1 (en) * | 1996-10-02 | 2005-09-22 | Lemelson Jerome H | Selectively controllable heads-up display system |
US20060250322A1 (en) * | 2005-05-09 | 2006-11-09 | Optics 1, Inc. | Dynamic vergence and focus control for head-mounted displays |
KR20090052169A (en) * | 2007-11-20 | 2009-05-25 | 삼성전자주식회사 | Head-mounted display |
JP2009157634A (en) * | 2007-12-26 | 2009-07-16 | Fuji Xerox Co Ltd | Irradiation control device, irradiation control program, and visual line analysis system |
US20110043644A1 (en) * | 2008-04-02 | 2011-02-24 | Esight Corp. | Apparatus and Method for a Dynamic "Region of Interest" in a Display System |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10067561B2 (en) | 2014-09-22 | 2018-09-04 | Facebook, Inc. | Display visibility based on eye convergence |
CN109890266A (en) * | 2016-11-04 | 2019-06-14 | 三星电子株式会社 | Method and apparatus for obtaining information by capturing eyes |
US11307646B2 (en) | 2016-11-04 | 2022-04-19 | Samsung Electronics Co., Ltd. | Method and apparatus for acquiring information by capturing eye |
CN109890266B (en) * | 2016-11-04 | 2022-07-12 | 三星电子株式会社 | Method and apparatus for obtaining information by capturing eye |
WO2018237172A1 (en) * | 2017-06-21 | 2018-12-27 | Quantum Interface, Llc | Systems, apparatuses, interfaces, and methods for virtual control constructs, eye movement object controllers, and virtual training |
EP3521978A1 (en) * | 2018-02-05 | 2019-08-07 | Samsung Electronics Co., Ltd. | Apparatus and method for tracking a focal point in a head mounted display system |
US10867174B2 (en) | 2018-02-05 | 2020-12-15 | Samsung Electronics Co., Ltd. | System and method for tracking a focal point for a head mounted device |
Also Published As
Publication number | Publication date |
---|---|
US20130241805A1 (en) | 2013-09-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20130241805A1 (en) | Using Convergence Angle to Select Among Different UI Elements | |
US9213185B1 (en) | Display scaling based on movement of a head-mounted display | |
US8955973B2 (en) | Method and system for input detection using structured light projection | |
US10055642B2 (en) | Staredown to produce changes in information density and type | |
EP3531274B1 (en) | Management of content in a 3d holographic environment | |
EP3014338B1 (en) | Tracking head movement when wearing mobile device | |
US8971570B1 (en) | Dual LED usage for glint detection | |
US10037076B2 (en) | Gesture-driven modifications of digital content shown by head-mounted displays | |
CN107111370B (en) | Virtual representation of real world objects | |
EP3097461B1 (en) | Automated content scrolling | |
EP3097460B1 (en) | Gaze swipe selection | |
US9261959B1 (en) | Input detection | |
AU2013351980A1 (en) | Direct hologram manipulation using IMU | |
WO2015116475A1 (en) | Radial selection by vestibulo-ocular reflex fixation | |
KR20140059213A (en) | Head mounted display with iris scan profiling | |
EP2926188A1 (en) | Peripheral display for a near-eye display device | |
JP2022540675A (en) | Determination of Eye Rotation Center Using One or More Eye Tracking Cameras | |
WO2018120554A1 (en) | Image display method and head-mounted display device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13761183 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 13761183 Country of ref document: EP Kind code of ref document: A1 |