US20120287031A1 - Presence sensing - Google Patents

Presence sensing Download PDF

Info

Publication number
US20120287031A1
US20120287031A1 US13/219,573 US201113219573A US2012287031A1 US 20120287031 A1 US20120287031 A1 US 20120287031A1 US 201113219573 A US201113219573 A US 201113219573A US 2012287031 A1 US2012287031 A1 US 2012287031A1
Authority
US
United States
Prior art keywords
user
sensor
image
computing device
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/219,573
Inventor
Edward Allen Valko
Matthew C. Waldon
Rudolph van der Merwe
William Matthew VIETA
Myra M. Haggerty
Alex T. Nelson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Apple Inc
Original Assignee
Apple Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Apple Inc filed Critical Apple Inc
Priority to US13/219,573 priority Critical patent/US20120287031A1/en
Assigned to APPLE INC. reassignment APPLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HAGGERTY, MYRA M., NELSON, ALEX T., VALKO, Edward Allen, VIETA, WILLIAM MATTHEW, WALDON, Matthew C., MERWE, RUDOLPH VAN DER
Priority to US13/469,996 priority patent/US10372191B2/en
Priority to PCT/US2012/037644 priority patent/WO2012155105A1/en
Priority to AU2012253292A priority patent/AU2012253292B2/en
Priority to CN201280023139.7A priority patent/CN103534664B/en
Priority to EP12722259.4A priority patent/EP2712431A1/en
Priority to KR1020137029536A priority patent/KR20140002034A/en
Priority to JP2014510519A priority patent/JP5876142B2/en
Priority to KR1020167003580A priority patent/KR101660215B1/en
Priority to TW101117124A priority patent/TWI506478B/en
Publication of US20120287031A1 publication Critical patent/US20120287031A1/en
Priority to US15/445,623 priority patent/US10402624B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation
    • G06V40/166Detection; Localisation; Normalisation using acquisition arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/20Analysis of motion
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20076Probabilistic image processing
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2330/00Aspects of power supply; Aspects of display protection and defect management
    • G09G2330/02Details of power systems and of start or stop of display operation
    • G09G2330/021Power management, e.g. power saving
    • G09G2330/022Power management, e.g. power saving in absence of operation, e.g. no data being entered during a predetermined time
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00Aspects of interface with display user

Definitions

  • the present disclosure is generally related to devices having computing capabilities and, more particularly, to sensing the presence of a user in local proximity to the device.
  • Many computing devices are equipped with power saving features/modes intended to reduce power consumption when a user is not using the devices. Often, these power saving features are implemented though timers that count down a set amount of time from when the user last provides an input to the device. For example, a particular device may be configured to enter a sleep mode, or other mode that consumes less power than a fully operational mode, when a user has not provided input for five minutes.
  • a device may enter the power saving features/modes while a user is still using the device.
  • the power saving features may be entered because the user failed to provide input within the time period set for the timer while reading content on the device, viewing a movie, or listening to music.
  • recovery from the power saving feature/mode may take time, may even require the user to enter credentials, and generally may be a nuisance to the user.
  • One embodiment may take the form of a method of operating a computing device to provide presence based functionality.
  • the method may include operating the computing device in a reduced power state and collecting a first set of data from a first sensor. Based on the first set of data, the computing device determines if an object is within a threshold distance of the computing device and, if the object is within the threshold distance, the device activates a secondary sensor to collect a second set of data. Based on the second set of data, the device determines if the object is a person. If the object is a person, the device determines a position of the person relative to the computing device and executes a change of state in the computing device based on the position of the person relative to the computing device. If the object is not a person, the computing device remains in a reduced power state.
  • Another embodiment may take the form of a method for determining if a user is in proximity of a computing device.
  • the method includes capturing an image using an image sensor and computing at least one of the following from the captured image: a skin tone detection parameter, a face detection parameter and a movement detection parameter.
  • the method also includes utilizing at least one of the skin tone detection parameter, face detection parameter and the movement detection parameter to make a determination as to whether a user is present and, if it is determined that a user is present, changing a state of the computing device.
  • a computing system having a main processor and an image based presence sensor coupled to the main processor.
  • the image based presence sensor includes an image sensor, and a processor coupled to-the image sensor and processor configured to process the image to determine if a user is present in the image. If the processor determines that a user is present in the image, an indication that a user has been determined to be present is sent from the processor to the main processor and the main processor changes a state of the computing system based on the indication.
  • FIG. 1 illustrates an example computing device having user presence sensing capabilities.
  • FIG. 2 is a block diagram of the computing device of FIG. 1 .
  • FIG. 3 is a plot showing presence sensing results when an object of interest is located at different distances from presence sensor.
  • FIG. 4 is another plot showing presence sensing results when the object of interest is offset an angle from the sensor.
  • FIG. 5 is a flowchart illustrating an example method for operating a tiered presence sensor system.
  • FIG. 6 is a flowchart illustrating a method for determining presence of a user.
  • FIG. 7 is a flowchart illustrating skin tone detection routine for use in presence sensing.
  • FIG. 8 is a flowchart illustrating a face recognition routine for use in presence sensing.
  • FIG. 9 is a flowchart illustrating a motion detection routine for use in presence sensing.
  • FIG. 10 illustrates frames being divided into window for single frame motion detection.
  • FIG. 11 is a flowchart illustrating an example method for single frame motion detection.
  • the embodiments discussed herein are directed to user presence determination and computing device functionality related thereto. It should be appreciated that a user's experience interacting with computing devices equipped with such functionality may be improved. Further, in some embodiments, power saving and/or power efficiency may be realized through implementation of the embodiments discussed herein.
  • One embodiment may take the form of a computing device that is configured to sense the presence and/or absence of a user and provide an operating state based on the presence and/or absence of the user.
  • the computing device may calculate and provide a likelihood or probability score of the user being present or not present.
  • multiple parameters may be determined, weighted, and used in conjunction in making a presence determination. This weighted detection can be used for more informed higher level decision making algorithms, or when fusing data from different sensors.
  • the computing device may be configured to determine when a user arrives or enters into proximity with the computing device and/or a probability that the user is present based on sensor input. In response to a positive determination that the user is present or upon achieving a threshold probability that the user is present, the device may power up, exit a sleep mode, and/or provide some feedback to the user.
  • a system awake may be initiated when it is determined that a user is approaching.
  • the system awake may include a reduced set of routines so that the system is in an operational mode faster than with a conventional power up sequence. For example, the system may power up within a half second rather than six to eight seconds due to the reduced set of routines.
  • the computing device may be configured to determine when a user moves away from the device or leaves the proximity of the device. In response, the device may enter a power saving mode, such as a display sleep mode, a system sleep mode, activation of a screen saver, and so forth. Further, the system may exit the sleep mode partially in order to speed up the computer wake up time based on sensing the presence of a user.
  • the device may also be configured to track the user movements (e.g., vector and speed) and, in response to certain movements, provide feedback and/or enter or change a state of operation. For example, movement toward the device may activate more features, such as providing more options/menus in a user interface, whereas movement away from the device may reduce the number of features available to a user, such as reducing the number of menus/options and/or reducing or increasing the size of the options displayed. Additionally or alternatively, the display may zoom in or zoom out based on movement towards or away from the device. In some embodiments, a lateral movement of by the user (e.g., from left to right) may cause a change in a background and/or a screen saver image displayed on the device. Still further, the changing of the image may correspond generally with the sensed motion. For example, the movement from left to right may cause the image to be replaced in a left to right motion with another image.
  • the user movements e.g., vector and speed
  • movement toward the device may activate more features
  • the presence of the user may be used together with the position of the user relative to the device to provide certain functionality.
  • input and/or output may be based on the position.
  • the device may be configured to provide audio stereo panning (e.g., audio steering) directed to the user's position.
  • microphone steering may be implemented based on the user's position.
  • a plurality of sensors and/or operational states may be implemented in a tiered manner. That is, in a first operational mode a first sensor may be operational. Detection of movement or user presence may result in the activation of a second sensor, and so forth. In some embodiments, the activation of the second sensor may be concurrent with the device entering a second operational mode, while in other embodiments, a second operation mode may not be entered into until a determination is made based upon data retrieved from the second sensor alone or in combination with the data from the first sensor.
  • the presence determination may be made by data collected by one or more sensors.
  • data from one or more sensor is used to determine if a user is present.
  • a neural net, support vector machine or other suitable classifier or probabilistic determiner may be implemented.
  • a large set of data points may be collected, classified and stored for using in the presence determination.
  • subsequently acquired data may be added and used for future determinations.
  • the computing device 100 may generally include one or more sensors 102 that may be utilized for presence sensing.
  • sensors 102 may be utilized for presence sensing.
  • one or more cameras and/or light sensors may be used in the presence sensing.
  • cameras and light sensors will be generally discussed herein with respect to the presence sensing, it should be appreciated that other sensor types may be implemented as well, such as ultrasonic sensors, microwave RADAR, and so forth.
  • various techniques and wavelengths of light may be implemented.
  • proximity may be determined by focusing and defocusing, using active IR reflected power, active IR structured light; active IR time of flight (2D+depth), active IR time of flight (single pixel sensor), Passive IR (Motion detector), passive IR thermal imaging (2D), and so forth.
  • active IR reflected power active IR structured light
  • active IR time of flight (2D+depth)
  • active IR time of flight single pixel sensor
  • passive IR thermal imaging 2D
  • FIG. 2 is a block diagram of the computing device 100 of FIG. 1 .
  • the computing device includes a microprocessor/microcontroller 104 to which other components (e.g., sensors).
  • the microprocessor/microcontroller 104 may be implemented as one or more low power microcontrollers and as a point of data fusion for the data coming from sensors (e.g., camera, proximity, and so forth) as well as for the high-level user present or not-present decision making.
  • the user presence determination and data related thereto may be externalized and isolated from the main operation of the device. That is, the user presence system provides security and privacy by isolating the presence sensing data from the main computer processing unit (CPU) 105 , the operating system, and so forth.
  • CPU computer processing unit
  • a variety of suitable sensors may provide input/data to the microprocessor/microcontroller 104 .
  • a camera based sensor 106 may be communicatively coupled with the microprocessor 104 .
  • Any suitable camera based sensor may be implemented and a variety of different techniques may be utilized.
  • camera sensors available from ST Microelectronics may be used.
  • the camera based sensor may include a full image camera 108 that provides face detection capabilities with an integrated processor 110 . That is, the sensor may have an embedded microprocessor 110 and may be capable of estimating face position and distance. Additionally, the sensor may be used for determining distances of objects.
  • the camera 108 also provides a windowed histogram information from the AGC system, which may be useful for motion detection.
  • the camera 108 may have a horizontal field of view up to or greater than 120 degrees and a vertical field of view up to or greater than 120 degrees.
  • lenses such as fish eye lenses may be used to achieve field of views having angle greater than 120 degrees.
  • the horizontal field of view may be between 75-95 degrees (e.g., approximately 85 degrees) and the vertical field of view may be between 40-80 degrees (e.g., approximately 60 degrees).
  • Faces may be detected at distances up to 20 feet or more. In one embodiment, faces may be detected at approximately 6-14 feet.
  • Face position data may be available at between approximately 0.6-1 Hz and AGC data may be available at the full frame rate, approximately 10-30 Hz.
  • the images captured by the camera based sensor 106 and related raw information may not be available outside of the camera based sensor. Rather, information as to whether a face is detected within the functional range of the sensor, the position of the face and/or movement of the face within that range may be provided.
  • the camera sensor may provide a binary output indicating that a user is or is not present. Additionally, if a user is present, the position of the user relative to the device may be output by the camera based sensor, for example in x-y coordinates.
  • the sensor may be configured to indicate the number of faces that are present (e.g., indicate the number of people present), among other things.
  • the camera based sensor 106 may be implemented independent of other sensors to achieve desired operational characteristics for a device.
  • the camera based sensor may be configured to operate and provide output in a tiered manner. For example, in a first state the camera based sensor may sense for user presence. If a user is present, then it may enter a second state and determine how many people are present. Subsequently, or concurrently, it may determine the location of the people who are present. As the camera moves from a state of operation, it provides an output which may be used by the device to change the state of the device, as will be discussed in greater detail below.
  • the main camera 103 may be a system camera used for video and still image capture by the user of the device and, in some embodiments it may be a separate camera from the camera of the camera based sensor (e.g., there are multiple cameras in the system), while in other embodiments the main camera output may be used by the camera based sensor in lieu of the camera based sensor 106 having a dedicated camera.
  • the main camera output may be provided to an image processor 107 for use by a user as well as to a micro-controller of the camera based sensor 106 for user presence detection. There may be different options on how the image processor and user-detect co-processor communicate and make the data from the main camera available to a user.
  • the output from the main camera may primarily be processed by the micro-controller for the presence sensing determination.
  • the data from the camera may generally not be made available to other components of the system.
  • the output from the main camera may be provided to the image processor 107 .
  • the user may be required to access a camera based application (e.g., video chat application, image capture program, or the like). Otherwise, the image data from the camera may not generally be accessible.
  • a camera based application e.g., video chat application, image capture program, or the like.
  • the main camera output may be routed to a single chip that combines the normal image processing functions and user presence detection functions.
  • the video output from the camera may be streamed to a host for processing by a central processing unit.
  • a second sensor such as a proximity sensor 112
  • a controller 114 may be connected to the microprocessor 104 .
  • a controller 114 may be operated in conjunction with the proximity sensor 112 .
  • the controller 114 may be configured to control the operation of the multiplexer 116 and the LEDs 118 in a time division multiplexed (TDM) manner.
  • TDM time division multiplexed
  • a suitable filter may be implemented to obtain a desirable response with the TDM alternating of the LEDs.
  • a mechanical device e.g., micro electrical-mechanical device
  • the LEDs 118 may operate in any suitable range of wavelengths and, in one example, may operate in the near infrared region of the electromagnetic spectrum.
  • Each of the LEDs (LED 1 -LEDN) may be directed to a particular field of view.
  • each LED 118 may be directed to a discrete field of view, while in other embodiments the field of view of adjacent LEDs may overlap.
  • the array of LEDs 118 may distribute the LEDs about a bezel of the computing device.
  • the LED array 118 may be configured in a row (e.g., across a curved portion of a display screen bezel) with the LEDs directionally positioned to cover different field of view.
  • an average value of the field of views may be obtained and used to determine whether an object is in proximity with the device 100 . If the average value exceeds a threshold value, it may indicate that an object is within proximity of the device 100 .
  • the proximity sensor may be able to more accurately detect proximity over a broad field of view. As each LED is directed to discrete field of view, the position of an object may also be determined using the proximity sensor 112 . As such, in other embodiments, a change in the proximity value from a presumed empty scene may be determined. A largest change (or some rank) may be looked at across the various sensors and this value may be compared with a threshold to determine proximity and/or location.
  • the camera based sensor 106 and the proximity sensor 112 may be utilized in conjunction with the microprocessor 104 to make a determination as to whether a user is in proximity of the computing device 100 .
  • a tiered sensing system may be implemented to provide power savings, to improve a user experience or provide a particular desired user experience, among other purposes and/or functions.
  • the tiered sensing system may include operating a first sensor to initially determine the presence of a user within a threshold distance of the computing device to provide power savings.
  • the threshold distance may be within 2-10 feet (e.g., five feet) of the device 100 .
  • data collected from the first sensor may be used to determine a relative position of the user.
  • a second sensor may be activated.
  • Data from the second sensor alone or in combination with data from the first sensor, may be used to further identify the user/person and/or the position of the user.
  • the data from both the first and Second sensors may be used together to make determinations as to what functions to perform and or what the user is doing. For example, it may be determined how close the user is to the device; if the user is facing the device, if the user is moving away/toward the device; and so forth. Further the data may be used to identify the user (e.g., as a credentialed user).
  • a state of the computing device 100 may change based on the determination that a user is present. For example, if the user is approaching the device, the display may awake, the system may awake, and so forth. If the user is moving left to right, a displayed image may change and may generally move corresponding to the movement of the user. Further, if multiple users are present (as determined based on discerning the presence of multiple faces), the device 100 may be powered to a secure state and may require entry of user credentials to fully access the device.
  • the presence determination may be based on multiple factors that are utilized in a neural network, support vector machine (SVM) or other machine learning based classifier or probabilistic decision system. For example, skin tone/color, presence and movement can be utilized in a weighted manner with a neural net to make a presence determination. As discussed above, based on the presence determination, the device 100 may enter/change operational states.
  • SVM support vector machine
  • skin tone/color, presence and movement can be utilized in a weighted manner with a neural net to make a presence determination.
  • the device 100 may enter/change operational states.
  • the camera based sensor 106 may be implemented as a first tier sensor, while in other embodiments; a proximity sensor 112 may be used as a first tier sensor.
  • a proximity sensor 112 may be used as a first tier sensor.
  • the sensor 112 may chop light at some suitable frequency and measure the phase shift of the returned reflected light signal.
  • the LED 118 outputs may be square waves or other waveforms, and the sensor 112 uses and I/Q demodulation scheme.
  • the light arriving from the sensor 112 is mixed with a sine wave and a cosine wave, giving and I (in-phase) component and a Q (quadrature) component.
  • the sine/cosine waves are synchronized with the LED modulation.
  • These are the ‘raw’ outputs from the sensors, if there is a different internal method for measurement, it may be converted to this scheme. Without loss of generality it may be assumed a period of 2 ⁇ , and that the integration takes place over that period. In practice, a fixed period may be used and will be integrating over some large multiple of the period. These differences result in a fixed scale factor, which may be ignored.
  • the basic measured components are:
  • phase shifts associated with these distances may be denoted as ⁇ and ⁇ .
  • the magnitude of the reflected signals may be defined to be A and B.
  • the incoming light signals are additive in this case and so is integration, so I is:
  • I 2 ⁇ ( A ⁇ cos( ⁇ )+ B ⁇ cos( ⁇ )).
  • the LED/Sensor field of view may be partitioned into N sections indexed from 1 to N. Each of these sections has a solid angle of ⁇ i . Further, each of these solid angles has a fixed reflectance ⁇ i , and is at a fixed radial distance r i . Also, the output from the LED is constant across a given solid angle with emitted intensity per steradian I i . The phase shift for a given distance is defined ⁇ (ri).
  • the (I i ,Q i ) contribution from a given solid angle at the sensor may be obtained. It is useful to also define a polar coordinate system in I/Q space.
  • the magnitude of the IQ vector is defined to be M i , and the angle, ⁇ i , is already defined.
  • Both (I m ,Q m ) may be defined as the measured (raw) I and Q values.
  • One more term (I c ,Q c ) may be added to represent any constant crosstalk (electrical or optical).
  • a background model poses a number of challenges due to the relative lack of information provided by the sensor 112 .
  • some simplifying assumptions may be made. Initially, the mathematics of the model for a single sensor will be addressed followed by the multiple sensors case.
  • the scene may be divided into these two types of objects.
  • the challenge is measuring the distance to the dynamic objects in the scene, such as people entering and leaving. In order to measure these objects successfully, an accurate model for the static objects in the scene is created and their relation to the dynamic object modeled.
  • (I p ,Q p ) are defined to be the signal associated with the object that is being measured.
  • the (I m ,Q m ) and (I c ,Q c ) may continue to be used as the measured (raw) and the crosstalk values, respectively.
  • This model is may be used to produce a distance output. For scenes that have no foreground objects, this model will always over-estimate the distance. Note that this model depends on factory calibration values to be accurate over the lifetime of the device. It may not account for crosstalk added due to smudge/etc.
  • the LED falloff with angle is defined as I( ⁇ ).
  • a single object of fixed width w is assumed to correspond to an angular section ⁇ p at a fixed position.
  • the center position of the object is defined in angular terms as ⁇ p .
  • L total ⁇ - ⁇ ⁇ ⁇ l ⁇ ( ⁇ )
  • L ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) ⁇ ⁇ p - ⁇ ⁇ ⁇ ⁇ p 2 ⁇ p + ⁇ ⁇ ⁇ ⁇ p 2 ⁇ l ⁇ ( ⁇ ) L total
  • R ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) 1 - L ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) .
  • L( ⁇ p , ⁇ p ) represents the fraction of light from the LED that is directed at the solid angle defined by the object of interest
  • L total represents the total light output
  • R( ⁇ p ; ⁇ p ) represents the fraction of total light cast on the background.
  • I p ⁇ p ⁇ L ( ⁇ p , ⁇ p ), cos( ⁇ p ),
  • I bg ⁇ bg ⁇ R ( ⁇ p , ⁇ p ) ⁇ cos( ⁇ bg ),
  • I open ⁇ bg ⁇ cos( ⁇ bg )+ I c
  • I m I p +I bg +I fg +I c
  • I open I bg +I fg +I c .
  • Two more variables may be added that are estimated as compared to the previous case.
  • This model partitions the horizontal field of view into a series of sections 1 . . . S, each of which is modeled as a uniform foreground/uniform background.
  • a superscript s is added to denote the section to which a variable belongs.
  • the background sections assume that an object is in the scene with width w corresponding to an angular section ⁇ p , and angular position ⁇ p . Redefine the R function sectionally to represent the fraction of light cast on the background after occlusion by the object of interest. It may be referred to as R s .
  • I bg s ⁇ bg s ⁇ R s ( ⁇ p , ⁇ p ) ⁇ cos( ⁇ bg s ), and
  • the foreground signal is not changed by an object in the scene, there is no need to model it sectionally.
  • the foreground may occlude the object of interest to varying degrees across sections. This could be modeled in a number of different ways, the cleanest of which would be to associate an ‘occlusion factor’ F s for each foreground section.
  • F s is defined as the fraction of total light output from the LED that illuminates the objects of interest in section s.
  • ⁇ p , ⁇ p , ⁇ p and d are common values between the two sensor measurements, and are specific to the object of interest. There is a well defined relationship between d and ⁇ p —see the example section herein.
  • is a constant sensitivity difference between the two sensors/LEDs, which should be slowly changing over the lifetime of the sensors.
  • I p 1 L 1 ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) ⁇ ⁇ p ⁇ 1 d 2 ⁇ cos ⁇ ( ⁇ p )
  • Q p 1 - L 1 ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) ⁇ ⁇ p ⁇ 1 d 2 ⁇ sin ⁇ ( ⁇ p )
  • I p 2 ⁇ ⁇ L 2 ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) ⁇ ⁇ p ⁇ 1 d 2 ⁇ cos ⁇ ( ⁇ p )
  • I p 2 - ⁇ ⁇ L 2 ⁇ ( ⁇ p , ⁇ ⁇ ⁇ ⁇ p ) ⁇ ⁇ p ⁇ 1 d 2 ⁇ sin ⁇ ( ⁇ p ) .
  • the falloff cast by the 10 degree LED sensor 112 was imaged against a white wall. Its horizontally projected falloff is approximately gaussian with a standard deviation of roughly 12 degrees.
  • the prototype was placed about 3.5 ft above the floor in a relatively dark, empty room with a backdrop at 12 ft.
  • the crosstalk was measured with a black felt baffle covering the sensor 112 .
  • the zero phase offset was measured with a reflective baffle.
  • the nominal ‘open’ background was captured.
  • Sensor data was collected with a person standing at 1 ft increments out from the sensor 112 at a 0 degree offset from the LED out to 10 ft.
  • Sensor data was collected in 5 degree increments at a radial distance of 5 ft, from ⁇ 15 degrees to +15 degrees.
  • the felt measurements may be referred to as (I c ,Q c ), as it essentially measures crosstalk.
  • the reflective baffle measurements may be referred to as (I 0 ; Q 0 ) and the open measurements as (I open ;Q open ).
  • the raw measurements with the object of interest in the scene may be referred to as (I m ;Q m ) and the to-be-estimated object of interest signal as (I p ;Q p ).
  • L( ⁇ p , ⁇ p ) was modeled assuming the Gaussian distribution mentioned above, whose specific form becomes:
  • ⁇ O arctan ⁇ ( - Q O I O )
  • ⁇ p arctan ⁇ ( - Q p I p )
  • d p ⁇ ⁇ ( ⁇ O - ⁇ p )
  • ⁇ ⁇ ⁇ p 2 ⁇ arctan ⁇ ( 1 d p ) ;
  • I m I p +(1.0 ⁇ L ( ⁇ p , ⁇ p )) ⁇ ( I open ⁇ I c )+ I c , and
  • multiple sensors may be combined into an integrated position model.
  • multiple proximity sensors may be implemented in one embodiment.
  • multiple LEDs may be used in a TDM manner to provide a desired field of view. Integrating a camera based sensor should allow estimation all parameters of interest.
  • FIG. 5 illustrates a method 300 for utilizing multiple sensors in a tiered manner to change the state of the device.
  • the device may be in a reduce power consumption mode such as a sleep mode, and a controller may receive data from a first sensor (Block 302 ).
  • the received data is processed (Block 304 ) and compared with a threshold (Block 306 ).
  • the comparison with the threshold allows for a determination as to whether the user is present or is likely present (Block 308 ). If the user is not present, data may continue to be received from the first sensor (Block 302 ).
  • a second sensor may be actuated (Block 310 ) and data is received from the second sensor (Block 312 ).
  • the data from the second sensor is processed (Block 314 ) and combined with the data from the first sensor data (Block 316 ).
  • the processing of data from the first and second sensor may include, but is not limited to, performing digital signal processing on the data such as filtering the data, scaling the data, and/or generally conditioning the data so that it is useful for presence determination.
  • the combination of data from the first and second sensors may include storing the data together and/or logically or mathematically combining the data.
  • the data from the first and second sensors is used to compute user presence values and/or probability of user presence scores (Block 318 ).
  • the user presence values and/or probability of user presence scores are compared with thresholds to determine if a user is present (Block 322 ). Further, if the user is determined to be present, other parameters may be determined such as distance and location of the user relative to the device (Block 324 ) and the state of the device may be changed (Block 326 ).
  • the state change may include bringing the device into an awake mode from a sleep mode or other suitable state change.
  • the second determination of user presence (Block 320 ) may be more accurate than the first determination (Block 308 ) based on the additional information provided from the second sensor.
  • additional parameters may be determined based on the combination of data from both the first and second sensors.
  • FIGS. 6-11 illustrate more detailed flowcharts of methods for presence determinations.
  • a flowchart 200 illustrating presence sensing is illustrated.
  • a camera is used to obtain an image (Block 202 ).
  • a light level determination may be made (Block 204 ) and provided to a skin tone detection routine (Block 206 ).
  • the light level determination may be provided to other routines as well, as indicated by arrow 203 .
  • the captured image may be pre-processed (Block 208 ).
  • the preprocessing may include down scaling of the image, changing the color space of the image and/or enhancing the image, for example.
  • Other detector specific preprocessing may also be performed (Blocks 214 , 215 and 217 ).
  • the image may optionally be blurred by preprocessing in Block 214 before being provided to the skin tone detection routine (Block 206 ).
  • preprocessing in Block 215 may include changing color into grayscale before providing the image to the face detection routine (Block 210 ) and/or performing edge detection in the preprocessing of Block 217 before providing the image to the movement detection routine (Block 212 ).
  • the skin tone detection routine, face detection routine and movement detection routine are discussed in greater detail below with reference to FIGS. 7-11 .
  • the results of the skin tone detection routine, face detection routine and movement detection routine may be weighted and combined using fusion and detection logic (Block 216 ) and a user presence classification is determined (Block 218 ).
  • the fusion and detection logic may include the use of neural networks, support vector machines, and/or some other form of probabilistic machine learning based algorithm to arrive at a determination of whether a user is present.
  • FIG. 7 illustrates the skin tone detection routine (Block 206 ) as a flowchart starting with the low light determination (Block 204 ). It should be appreciated that the low light determination may be utilized in a variety of different manners to influence the processing of the image.
  • the low light determination may be provided as a vector to a neural network, while in other embodiments the low light determination may be used to select a particular type of categorizer to be used. That is, if it is determined that the image was not taken in low light, feature vectors may be generated (Block 220 ), and a first pixel classifier is applied (Block 222 ). If the image was captured in low light, a different set of feature vectors may be generated (Block 224 ) and a second per pixel classifier may be applied (Block 226 ).
  • the type of features e.g., color conversion, and so forth may be selectively provided to achieve desired results and may be different depending on the low light determination.
  • the first and second per pixel classifiers may be different based on the low light determination.
  • the first classifier may be a 7-5-2 multilayer perceptron (MLP) feed forward neural network, per pixel classifier, while the second classifier may be a 2-12-2 MLP feed forward neural network per pixel classifier.
  • the classifiers may be implemented in an open kernel with a GPU to help speed up the process.
  • the output from the classifiers may be a probability (e.g., a value between 0 and 1) that indicates a probability that the image includes a skin tone.
  • a morphology filter may optionally be applied to the image (Block 228 ) and an average grey scale level may be calculated (Block 230 ). Further, nonlinear scaling (Block 232 ), a temporal queue filter (Block 234 ) and a clamp (Block 236 ) may be applied before determining a probability of user presence due to skin tone detection (Block 238 ).
  • FIG. 8 illustrates the face detection routine (Block 210 ) as a flowchart starting with applying a face detector (Block 240 ).
  • a face detector Any suitable face detector may be implemented, such as a Viola-Jones cascade face detector, for example, that provides a probabilistic score indicating the likelihood that a face is present.
  • the face presence score is then scaled (Block 242 ) and an intermittent detection flicker filter optionally may be applied (Block 244 ) to smooth the image. It should be appreciated that in some embodiments, such as where the camera is of a relatively good quality, smoothing may be omitted from the process.
  • the flicker filter may include a temporal queue filter (Block 246 ), determination as to whether the normalized score deviation from average is less than a threshold (Block 248 ) and then multiplying an output value with the scaled score (Block 250 ).
  • a temporal queue filter (Block 252 ) and a clamp (Block 254 ) are applied before determining a probability of user presence due to face detection (Block 256 ).
  • FIG. 9 illustrates the motion detection routine 212 as a flowchart starting by collecting multiple frames and, as such, memory may be implemented to store the multiple frames. For example, three frames may be utilized, a current frame and two other frames. In the embodiment illustrated in FIG. 9 , a current frame and two subsequent frames are used.
  • an input frames are delayed by k frames, sequentially, (Blocks 260 , 262 ) and fed forward to be added (Block 264 ) with the output of the second delay (Block 262 ).
  • the output of Block 260 is multiplied by 2 (Block 266 ) and a difference (Block 268 ) between the adder (Block 264 ) and the output of the multiplier (Block 266 ) is determined.
  • a per pixel inner product is then determined (Block 270 ), scaled (block 272 ), and pixels are clamped (Block 274 ).
  • An average gray level is calculated (Block 276 ), nonlinear scaling is performed (Block 278 ), temporal queue filter is applied (Block 280 ) and the clamped to [0,1] (Block 282 ).
  • a probability of user presence due to movement is determined (Block 284 ).
  • Some parameters that may be useful in the motion detection routine may include an auto focus (AF) window statistics or horizontal edges or Sobel/Sharr edges, 2D color histogram data, component histogram data, automatic white balance/auto exposure (AWB/AE) window statistics of color content, and so forth.
  • AF auto focus
  • AVB/AE automatic white balance/auto exposure
  • Some preprocessing steps may be implemented for the motion detection such as Y channel (intensity), gradient magnitude computed with either Sobel or Scharr filters (accumulating the gradient for proper normalization), threshold gradient magnitude (normalization by count of edge pixels), skin-probability in chrominance (Cr, Cb) space, sub-images of any of the foregoing, and so forth.
  • the motion detection may include the ability to computer image centroids.
  • the distance from the centroid for the current frame to the centroid of the previous frame is used as a measure of the amount of motion, and a hard threshold is applied to produce a binary detection or motion.
  • a change in centroid location of either a Y′-intensity, edge gradient magnitude, binary edge, or skin probability image may indicate motion.
  • Sensitivity and robustness tradeoffs may dictate a particular combination of parameters being used.
  • skin probability image may be used with edge-gradient and binary edges may be utilized to provide robustness to lighting changes.
  • the skin probability may be performed with a neural network, as mentioned above or alternatively using an auto white balance color space filters to approximate this functionality.
  • Some embodiments for sensing motion may refer to window statistics of either skin detection of gradient images.
  • One embodiment may look at a change in a global sum. In particular the image is summed over the entire frame to produce a scalar value s[i], where i is current frame index.
  • the motion detection routine may be implemented with a single frame and thus little or no memory may be used, as full frames will not be stored.
  • the image (a single frame) may be divided into windows for which statistics may be calculated. Changes in the statistics for the windows may be used to determine motion and also position of a user.
  • FIG. 10 illustrates possible sets of windows into which the images may be divided for single frame motion detection.
  • FIG. 10 shows single frames divided into both non-overlapping windows and concentric windows for statistical purposes.
  • the luminace of the images (frames at the top of page) and intensity gradient magnitude of the images (frames at the bottom of the page) are considered.
  • an image 300 may be divided into multiple non-overlapping exposure statistics windows 302 .
  • the image 300 may be divided into multiple concentric overlapping exposure statistics windows 304 .
  • the statistics for each window may be determined based on a luminance image (as in image 300 ) or based on an intensity gradient magnitude image 306 .
  • one embodiment includes eight concentrically arranged rectangles, the largest contains the entire frame and the smallest is centered in the image.
  • the sum of the frames is s j [i] for j ⁇ [1,2, . . . 8].
  • the differences are normalized by the height of the strip h j , which is approximately proportional to its area.
  • FIG. 11 is a flow chart illustrating the motion detection routine 212 B using a single frame 300 with non-concentric windows 302 .
  • each statistics window may be provided with a unique analysis pipe. That is, each window may be concurrently processed.
  • one or more windows may be processed sequentially in a common analysis pipe.
  • analysis pipe may refer to the processing steps associated with the statistical analysis of the windows and may include the temporal queue filter.
  • the image 300 may be divided into statistical windows 302 and an average statistic may be calculated for each window (Block 310 ).
  • a temporal queue filter may be applied to the average statistic (Block 312 ) and an excursion value “e” from short term past behavior may be calculated (Block 314 ).
  • the excursion value may be compared with a threshold to determine if it exceed the threshold (Block 316 )
  • a normalized count is kept for each excursion value that exceeds the threshold (Block 318 ) and if the normalized count exceed a voting threshold (Block 320 ), it is determined that motion has been detected (Block 322 ).
  • the excursion value may also be used to generate summary statistics (Block 324 ).
  • the summary statistics may be nonlinearly scaled (Block 326 ) and a probabilistic motion score may be provided (Block 328 ).
  • Block 322 will provide a binary one or a zero output indicating motion has been detected
  • Block 328 will provide a value between zero and one indicating the likelihood that motion has been detected in the image 300 .
  • a neural network, support vector machine (SVM) or other classification system may be utilized in each of the aforementioned routines to make a determination as to presence of a user.
  • a probability value of from each of the routines may be alone enough to make the determination that a user is present, for example, if a value Is above a certain threshold.
  • a combination of the probabilities from each of the routines may be used in determining if a user is present.
  • the output of a routine may not be used as its validity may be questionable. For example, the skin tone detection routine may be unreliable due to the lighting.
  • the probabilities output from the routines may be combined in a weighted manner (e.g., one probability may be given more weight based on the likelihood of it being more accurate than the others).
  • Embodiments described herein may be implemented to reduce power consumption in computing devices, such as notebook computers, desktop computers, and so forth.
  • the computing devices may provide presence sensing functionality even when the device is in a low power state, such as a hibernate or sleep state, so that the device may power up when a user is present and power down or enter a reduced power state when the user leaves.
  • the embodiments may be provided to improve a user experience with computing devices by providing intuitive powering up and powering down operation, as well as security features, among others.
  • a tiered system may be implemented that precludes the use of a main processor and RAM in low power operation.
  • a lowest power state may implement only a camera, an image signal processing (ISP) device and an embedded processor that may calculate a presence value in real-time.
  • ISP image signal processing
  • a face detector chip and RAM may be turned on.
  • the system processor and other resources may be powered on.
  • memory of previous frames is limited to statistics computed by the ISP, and limited to the space available in the embedded processor registers and cache (e.g., 32 k), as RAM may not be available.
  • the presence sensing information e.g., statistics, images, and so forth
  • the presence sensing information are not available outside of the presence sensing routines. That is, for example, an image captured for presence sensing may not be viewed by a user.

Abstract

One embodiment may take the form of a method of operating a computing device to provide presence based functionality. The method may include operating the computing device in a reduced power state and collecting a first set of data from a first sensor. Based on the first set of data, the computing device determines if an object is within a threshold distance of the computing device and, if the object is within the threshold distance, the device activates a secondary sensor to collect a second set of data. Based on the second set of data, the device determines if the object is a person. If the object is a person, the device determines a position of the person relative to the computing device and executes a change of state in the computing device based on the position of the person relative to the computing device. If the object is not a person, the computing device remains in a reduced power state.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • The present application claims priority to U.S. Provisional Patent Application No. 61/485,610, filed May 12, 2011, and entitled, “Presence Sensing,” and U.S. Provisional Patent Application No. 61/504,026, filed Jul. 1, 2011, and entitled, “Presence Sensing,” both of which are incorporated herein by reference in their entirety and for all purposes.
  • TECHNICAL FIELD
  • The present disclosure is generally related to devices having computing capabilities and, more particularly, to sensing the presence of a user in local proximity to the device.
  • BACKGROUND
  • Many computing devices are equipped with power saving features/modes intended to reduce power consumption when a user is not using the devices. Often, these power saving features are implemented though timers that count down a set amount of time from when the user last provides an input to the device. For example, a particular device may be configured to enter a sleep mode, or other mode that consumes less power than a fully operational mode, when a user has not provided input for five minutes.
  • Occasionally, however, a device may enter the power saving features/modes while a user is still using the device. For example, the power saving features may be entered because the user failed to provide input within the time period set for the timer while reading content on the device, viewing a movie, or listening to music. Additionally, recovery from the power saving feature/mode may take time, may even require the user to enter credentials, and generally may be a nuisance to the user.
  • SUMMARY
  • One embodiment may take the form of a method of operating a computing device to provide presence based functionality. The method may include operating the computing device in a reduced power state and collecting a first set of data from a first sensor. Based on the first set of data, the computing device determines if an object is within a threshold distance of the computing device and, if the object is within the threshold distance, the device activates a secondary sensor to collect a second set of data. Based on the second set of data, the device determines if the object is a person. If the object is a person, the device determines a position of the person relative to the computing device and executes a change of state in the computing device based on the position of the person relative to the computing device. If the object is not a person, the computing device remains in a reduced power state.
  • Another embodiment may take the form of a method for determining if a user is in proximity of a computing device. The method includes capturing an image using an image sensor and computing at least one of the following from the captured image: a skin tone detection parameter, a face detection parameter and a movement detection parameter. The method also includes utilizing at least one of the skin tone detection parameter, face detection parameter and the movement detection parameter to make a determination as to whether a user is present and, if it is determined that a user is present, changing a state of the computing device.
  • In still another embodiment, a computing system is provided having a main processor and an image based presence sensor coupled to the main processor. The image based presence sensor includes an image sensor, and a processor coupled to-the image sensor and processor configured to process the image to determine if a user is present in the image. If the processor determines that a user is present in the image, an indication that a user has been determined to be present is sent from the processor to the main processor and the main processor changes a state of the computing system based on the indication.
  • While multiple embodiments are disclosed, still other embodiments of the present invention will become apparent to those skilled in the art from the following Detailed Description. As will be realized, the embodiments are capable of modifications in various aspects, all without departing from the spirit and scope of the embodiments. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates an example computing device having user presence sensing capabilities.
  • FIG. 2 is a block diagram of the computing device of FIG. 1.
  • FIG. 3 is a plot showing presence sensing results when an object of interest is located at different distances from presence sensor.
  • FIG. 4 is another plot showing presence sensing results when the object of interest is offset an angle from the sensor.
  • FIG. 5 is a flowchart illustrating an example method for operating a tiered presence sensor system.
  • FIG. 6 is a flowchart illustrating a method for determining presence of a user.
  • FIG. 7 is a flowchart illustrating skin tone detection routine for use in presence sensing.
  • FIG. 8 is a flowchart illustrating a face recognition routine for use in presence sensing.
  • FIG. 9 is a flowchart illustrating a motion detection routine for use in presence sensing.
  • FIG. 10 illustrates frames being divided into window for single frame motion detection.
  • FIG. 11 is a flowchart illustrating an example method for single frame motion detection.
  • DETAILED DESCRIPTION
  • Generally, the embodiments discussed herein are directed to user presence determination and computing device functionality related thereto. It should be appreciated that a user's experience interacting with computing devices equipped with such functionality may be improved. Further, in some embodiments, power saving and/or power efficiency may be realized through implementation of the embodiments discussed herein.
  • One embodiment may take the form of a computing device that is configured to sense the presence and/or absence of a user and provide an operating state based on the presence and/or absence of the user. In other embodiments, the computing device may calculate and provide a likelihood or probability score of the user being present or not present. In some embodiments multiple parameters may be determined, weighted, and used in conjunction in making a presence determination. This weighted detection can be used for more informed higher level decision making algorithms, or when fusing data from different sensors.
  • For example, in some embodiments, the computing device may be configured to determine when a user arrives or enters into proximity with the computing device and/or a probability that the user is present based on sensor input. In response to a positive determination that the user is present or upon achieving a threshold probability that the user is present, the device may power up, exit a sleep mode, and/or provide some feedback to the user.
  • Moreover, in some embodiments, a system awake may be initiated when it is determined that a user is approaching. The system awake may include a reduced set of routines so that the system is in an operational mode faster than with a conventional power up sequence. For example, the system may power up within a half second rather than six to eight seconds due to the reduced set of routines. In some embodiments, the computing device may be configured to determine when a user moves away from the device or leaves the proximity of the device. In response, the device may enter a power saving mode, such as a display sleep mode, a system sleep mode, activation of a screen saver, and so forth. Further, the system may exit the sleep mode partially in order to speed up the computer wake up time based on sensing the presence of a user.
  • In some embodiments, the device may also be configured to track the user movements (e.g., vector and speed) and, in response to certain movements, provide feedback and/or enter or change a state of operation. For example, movement toward the device may activate more features, such as providing more options/menus in a user interface, whereas movement away from the device may reduce the number of features available to a user, such as reducing the number of menus/options and/or reducing or increasing the size of the options displayed. Additionally or alternatively, the display may zoom in or zoom out based on movement towards or away from the device. In some embodiments, a lateral movement of by the user (e.g., from left to right) may cause a change in a background and/or a screen saver image displayed on the device. Still further, the changing of the image may correspond generally with the sensed motion. For example, the movement from left to right may cause the image to be replaced in a left to right motion with another image.
  • Moreover, in some embodiments, the presence of the user may be used together with the position of the user relative to the device to provide certain functionality. In some embodiments, input and/or output may be based on the position. For example, the device may be configured to provide audio stereo panning (e.g., audio steering) directed to the user's position. Additionally, in some embodiments, microphone steering may be implemented based on the user's position.
  • Further, a plurality of sensors and/or operational states may be implemented in a tiered manner. That is, in a first operational mode a first sensor may be operational. Detection of movement or user presence may result in the activation of a second sensor, and so forth. In some embodiments, the activation of the second sensor may be concurrent with the device entering a second operational mode, while in other embodiments, a second operation mode may not be entered into until a determination is made based upon data retrieved from the second sensor alone or in combination with the data from the first sensor.
  • The presence determination may be made by data collected by one or more sensors. In one embodiment, data from one or more sensor is used to determine if a user is present. For example, a neural net, support vector machine or other suitable classifier or probabilistic determiner may be implemented. In some instances a large set of data points may be collected, classified and stored for using in the presence determination. Furthermore, subsequently acquired data may be added and used for future determinations.
  • Turning to the drawings and referring initially to FIG. 1, a computing device 100 is illustrated. The computing device 100 may generally include one or more sensors 102 that may be utilized for presence sensing. For example, one or more cameras and/or light sensors may be used in the presence sensing. Although cameras and light sensors will be generally discussed herein with respect to the presence sensing, it should be appreciated that other sensor types may be implemented as well, such as ultrasonic sensors, microwave RADAR, and so forth. Moreover, various techniques and wavelengths of light may be implemented. For example, proximity may be determined by focusing and defocusing, using active IR reflected power, active IR structured light; active IR time of flight (2D+depth), active IR time of flight (single pixel sensor), Passive IR (Motion detector), passive IR thermal imaging (2D), and so forth. As such, the particular embodiments described herein are merely presented as examples and are not limiting.
  • FIG. 2 is a block diagram of the computing device 100 of FIG. 1. Generally, the computing device includes a microprocessor/microcontroller 104 to which other components (e.g., sensors). The microprocessor/microcontroller 104 may be implemented as one or more low power microcontrollers and as a point of data fusion for the data coming from sensors (e.g., camera, proximity, and so forth) as well as for the high-level user present or not-present decision making. In some embodiments, the user presence determination and data related thereto may be externalized and isolated from the main operation of the device. That is, the user presence system provides security and privacy by isolating the presence sensing data from the main computer processing unit (CPU) 105, the operating system, and so forth.
  • A variety of suitable sensors may provide input/data to the microprocessor/microcontroller 104. Specifically, a camera based sensor 106 may be communicatively coupled with the microprocessor 104. Any suitable camera based sensor may be implemented and a variety of different techniques may be utilized. For example, camera sensors available from ST Microelectronics may be used. The camera based sensor may include a full image camera 108 that provides face detection capabilities with an integrated processor 110. That is, the sensor may have an embedded microprocessor 110 and may be capable of estimating face position and distance. Additionally, the sensor may be used for determining distances of objects. The camera 108 also provides a windowed histogram information from the AGC system, which may be useful for motion detection.
  • Further, the camera 108 may have a horizontal field of view up to or greater than 120 degrees and a vertical field of view up to or greater than 120 degrees. In some embodiments lenses such as fish eye lenses may be used to achieve field of views having angle greater than 120 degrees. In one embodiment the horizontal field of view may be between 75-95 degrees (e.g., approximately 85 degrees) and the vertical field of view may be between 40-80 degrees (e.g., approximately 60 degrees). Faces may be detected at distances up to 20 feet or more. In one embodiment, faces may be detected at approximately 6-14 feet. Face position data may be available at between approximately 0.6-1 Hz and AGC data may be available at the full frame rate, approximately 10-30 Hz.
  • Generally, the images captured by the camera based sensor 106 and related raw information may not be available outside of the camera based sensor. Rather, information as to whether a face is detected within the functional range of the sensor, the position of the face and/or movement of the face within that range may be provided. In some embodiments, the camera sensor may provide a binary output indicating that a user is or is not present. Additionally, if a user is present, the position of the user relative to the device may be output by the camera based sensor, for example in x-y coordinates. Moreover, the sensor may be configured to indicate the number of faces that are present (e.g., indicate the number of people present), among other things.
  • In some embodiments, the camera based sensor 106 may be implemented independent of other sensors to achieve desired operational characteristics for a device. In some embodiments, the camera based sensor may be configured to operate and provide output in a tiered manner. For example, in a first state the camera based sensor may sense for user presence. If a user is present, then it may enter a second state and determine how many people are present. Subsequently, or concurrently, it may determine the location of the people who are present. As the camera moves from a state of operation, it provides an output which may be used by the device to change the state of the device, as will be discussed in greater detail below.
  • Some embodiments may include using a main camera 103 to capture images. The main camera 103 may be a system camera used for video and still image capture by the user of the device and, in some embodiments it may be a separate camera from the camera of the camera based sensor (e.g., there are multiple cameras in the system), while in other embodiments the main camera output may be used by the camera based sensor in lieu of the camera based sensor 106 having a dedicated camera. In one embodiment, the main camera output may be provided to an image processor 107 for use by a user as well as to a micro-controller of the camera based sensor 106 for user presence detection. There may be different options on how the image processor and user-detect co-processor communicate and make the data from the main camera available to a user. For example, when a user is not present the output from the main camera may primarily be processed by the micro-controller for the presence sensing determination. In this state, the data from the camera may generally not be made available to other components of the system. When a user is present, the output from the main camera may be provided to the image processor 107. However for the image data/information to be available, the user may be required to access a camera based application (e.g., video chat application, image capture program, or the like). Otherwise, the image data from the camera may not generally be accessible.
  • It should be appreciated that there may be many different configurations that allow for the desired presence sensing using one or more cameras as well as the conventional camera and image processing functionality. For example, in one embodiment, the main camera output may be routed to a single chip that combines the normal image processing functions and user presence detection functions. In other embodiments, the video output from the camera may be streamed to a host for processing by a central processing unit.
  • A second sensor, such as a proximity sensor 112, may also be connected to the microprocessor 104. In some embodiments, a controller 114, a multiplexer 116 and an array of light emitting diodes 118 may be operated in conjunction with the proximity sensor 112. In particular, the controller 114 may be configured to control the operation of the multiplexer 116 and the LEDs 118 in a time division multiplexed (TDM) manner. A suitable filter may be implemented to obtain a desirable response with the TDM alternating of the LEDs. In other embodiments, a mechanical device (e.g., micro electrical-mechanical device) may be used to multiplex one or more LEDs to cover discrete fields of view.
  • The LEDs 118 may operate in any suitable range of wavelengths and, in one example, may operate in the near infrared region of the electromagnetic spectrum. Each of the LEDs (LED1-LEDN) may be directed to a particular field of view. In some embodiments, each LED 118 may be directed to a discrete field of view, while in other embodiments the field of view of adjacent LEDs may overlap. In some embodiments, the array of LEDs 118 may distribute the LEDs about a bezel of the computing device. In other embodiments, the LED array 118 may be configured in a row (e.g., across a curved portion of a display screen bezel) with the LEDs directionally positioned to cover different field of view.
  • In one embodiment, an average value of the field of views (e.g., value indicating proximity) may be obtained and used to determine whether an object is in proximity with the device 100. If the average value exceeds a threshold value, it may indicate that an object is within proximity of the device 100. With the use of the array of LEDs 118, the proximity sensor may be able to more accurately detect proximity over a broad field of view. As each LED is directed to discrete field of view, the position of an object may also be determined using the proximity sensor 112. As such, in other embodiments, a change in the proximity value from a presumed empty scene may be determined. A largest change (or some rank) may be looked at across the various sensors and this value may be compared with a threshold to determine proximity and/or location.
  • In some embodiments, the camera based sensor 106 and the proximity sensor 112 may be utilized in conjunction with the microprocessor 104 to make a determination as to whether a user is in proximity of the computing device 100. A tiered sensing system may be implemented to provide power savings, to improve a user experience or provide a particular desired user experience, among other purposes and/or functions. In particular, the tiered sensing system may include operating a first sensor to initially determine the presence of a user within a threshold distance of the computing device to provide power savings. In some embodiments, the threshold distance may be within 2-10 feet (e.g., five feet) of the device 100. Additionally, in some embodiments, data collected from the first sensor may be used to determine a relative position of the user.
  • In the tiered system, if a user is present, a second sensor may be activated. Data from the second sensor, alone or in combination with data from the first sensor, may be used to further identify the user/person and/or the position of the user. The data from both the first and Second sensors may be used together to make determinations as to what functions to perform and or what the user is doing. For example, it may be determined how close the user is to the device; if the user is facing the device, if the user is moving away/toward the device; and so forth. Further the data may be used to identify the user (e.g., as a credentialed user).
  • A state of the computing device 100 may change based on the determination that a user is present. For example, if the user is approaching the device, the display may awake, the system may awake, and so forth. If the user is moving left to right, a displayed image may change and may generally move corresponding to the movement of the user. Further, if multiple users are present (as determined based on discerning the presence of multiple faces), the device 100 may be powered to a secure state and may require entry of user credentials to fully access the device.
  • The presence determination may be based on multiple factors that are utilized in a neural network, support vector machine (SVM) or other machine learning based classifier or probabilistic decision system. For example, skin tone/color, presence and movement can be utilized in a weighted manner with a neural net to make a presence determination. As discussed above, based on the presence determination, the device 100 may enter/change operational states.
  • It should be appreciated that the selection of a particular sensor for use will be dependant upon a wide variety of factors, including functionality desired and power consumption limitations, for example. As such, in some embodiments, the camera based sensor 106 may be implemented as a first tier sensor, while in other embodiments; a proximity sensor 112 may be used as a first tier sensor. A more detailed description for implementing a proximity sensor such as the proximity sensor 112 is provided below.
  • The sensor 112 may chop light at some suitable frequency and measure the phase shift of the returned reflected light signal. The LED 118 outputs may be square waves or other waveforms, and the sensor 112 uses and I/Q demodulation scheme. The light arriving from the sensor 112 is mixed with a sine wave and a cosine wave, giving and I (in-phase) component and a Q (quadrature) component. The sine/cosine waves are synchronized with the LED modulation. These are the ‘raw’ outputs from the sensors, if there is a different internal method for measurement, it may be converted to this scheme. Without loss of generality it may be assumed a period of 2π, and that the integration takes place over that period. In practice, a fixed period may be used and will be integrating over some large multiple of the period. These differences result in a fixed scale factor, which may be ignored. The basic measured components are:
  • s(t) Input signal to sensor

  • i(t)=sin(ts(t)

  • q(t)=cos(ts(t)

  • I=∫ 0 2·π i(t)dt

  • Q=∫ 0 2·π q(t)dt
  • If measuring an object at constant (radial) distance from the sensor 112 that takes up the entire field of view, a square wave input signal of the same frequency with phase offset φ and magnitude A, results in I and Q components:

  • I=M·∫ φ π+φ·sin(t)dt

  • I=M·(−cos(π+φ)+cos(φ))

  • I=2·M·cos(φ).

  • And:

  • Q=M·∫ φ π+φ·cos(t)dt

  • Q=M·(sin(π+φ)−sin(φ))

  • Q=−M·sin(φ).
  • The value φ may then be found as:
  • φ = arctan ( - Q I ) .
  • Then M may be reconstructed as:

  • 2·M=√{square root over (I 2 +Q 2)}
  • Supposing there are two objects (A and B) in the sensor's field of view, each of which is at a constant distance, the phase shifts associated with these distances may be denoted as φ and ψ. The magnitude of the reflected signals may be defined to be A and B. The incoming light signals are additive in this case and so is integration, so I is:

  • I=A·∫ φ π+φ sin(t)dt+B·∫ φ π+φ sin(t)dt

  • I=2·(A·cos(φ)+B·cos(φ)).
  • Similarly for Q:

  • Q=A·∫ φ π+φ cos(t)dt+B·∫ φ π+φ cos(t)dt

  • Q=−2·(A·sin(φ)+B·sin(φ)).
  • Light sources whose intensity does not vary with time will give zero contribution to the I and Q components. This property provides good ambient light rejection. It also provides cancellation due to phase offset from objects at different distances. Using a one/zero square wave demodulation, this information may be retained but with worse ambient light rejection. This demodulation scheme would lead to slightly different math, but the end results would be similar. For the following, the factor of two in front of I/Q will be dropped as it gets absorbed in the other scale factors.
  • A few simplifications may be made and a basic model is proposed for the sensor output as a function of objects in the scene. The discrete case will be developed since it is more amenable to implementation, although other cases may be implemented as well. The LED/Sensor field of view may be partitioned into N sections indexed from 1 to N. Each of these sections has a solid angle of Ωi. Further, each of these solid angles has a fixed reflectance ρi, and is at a fixed radial distance ri. Also, the output from the LED is constant across a given solid angle with emitted intensity per steradian Ii. The phase shift for a given distance is defined φ(ri).
  • From this model, the (Ii,Qi) contribution from a given solid angle at the sensor may be obtained. It is useful to also define a polar coordinate system in I/Q space. The magnitude of the IQ vector is defined to be Mi, and the angle, φi, is already defined.
  • M i = l i · Ω i · ρ i r i 2 I i = · M i · cos ( φ ( r i ) ) Q i = - · M i · sin ( φ ( r i ) )
  • Both (Im,Qm) may be defined as the measured (raw) I and Q values. One more term (Ic,Qc) may be added to represent any constant crosstalk (electrical or optical). Finally:
  • I m = I c + i = 1 N I i Q m = Q c + i = 1 N Q i
  • Generally, to determine if a user is proximately located to a device it may be beneficial to understand the environment in which the device is located. This may help reduce false positives and more accurately determine when a user enters or exits the proximity of the device 100. However, creating a background model poses a number of challenges due to the relative lack of information provided by the sensor 112. In order to define a useful model some simplifying assumptions may be made. Initially, the mathematics of the model for a single sensor will be addressed followed by the multiple sensors case.
  • Fundamentally, there are two types of objects that affect the distance measurement provided by certain proximity sensors, such as sensor 112. There are those objects which can not be occluded by the person, and there those are objects which can be occluded by the person. The former will be referred to as ‘foreground’ objects and the latter as ‘background’ objects. Of course, an object could fall into both categories depending on how it is positioned relative to the person. For now, the scene may be divided into these two types of objects. Generally, the challenge is measuring the distance to the dynamic objects in the scene, such as people entering and leaving. In order to measure these objects successfully, an accurate model for the static objects in the scene is created and their relation to the dynamic object modeled.
  • Initially, (Ip,Qp) are defined to be the signal associated with the object that is being measured. The (Im,Qm) and (Ic,Qc) may continue to be used as the measured (raw) and the crosstalk values, respectively.
  • Empty Scene
  • One model assumes there are no foreground or background objects, and that all of the signal is due to the person in the scene. In its purest form, the factory calibration/crosstalk values may be used:

  • I p =I m −I c

  • Q p =Q m −Q c
  • This model is may be used to produce a distance output. For scenes that have no foreground objects, this model will always over-estimate the distance. Note that this model depends on factory calibration values to be accurate over the lifetime of the device. It may not account for crosstalk added due to smudge/etc.
  • Once a static offset is observed, it is modeled as some combination of foreground and background objects. The choice of how to distribute this static offset strongly affects the estimate of Ip and Qp.
  • Foreground Only
  • One way to account for the static offset is to assume it is all due to foreground objects. Effects such as crosstalk changes due to temperature or smudge fall into this category. Foreground objects, by definition, have a constant contribution to the signal regardless of the presence of a person. In the pure foreground model, the spatial distribution of the foreground objects is not relevant—anything that is not foreground is assumed to be our object of interest. Define (Ifg, Qfg) to be the signal from the foreground. This model implies:

  • I p =I m −I fg −I c

  • Q p =Q m −Q fg −Q c
  • Note that (Ifg+Ic,Qfg+Qc) is the measured sensor reading with no objects of interest in the scene. This is the standard ‘baseline subtraction’ model.
  • Uniform Background with Partial Occlusion
  • For this model, it is assumed that the background is at a uniform distance and has uniform reflectivity. It is further assumed that objects vertically cover the field of view. The LED falloff with angle is defined as I(θ). A single object of fixed width w is assumed to correspond to an angular section Δθp at a fixed position. The center position of the object is defined in angular terms as θp.
  • The general model is discussed above. For this model, area is purely a function of width, incident light is defined by I(θ), and distance/reflectance are constant but unknown.
  • For convenience, define:
  • L total = - l ( θ ) , L ( θ p , Δ θ p ) = θ p - Δ θ p 2 θ p + Δ θ p 2 l ( θ ) L total , and R ( θ p , Δ θ p ) = 1 - L ( θ p , Δ θ p ) .
  • L(θp, Δθp) represents the fraction of light from the LED that is directed at the solid angle defined by the object of interest, Ltotal represents the total light output, and R(θp;Δθp) represents the fraction of total light cast on the background.
  • The magnitude of light reaching the sensor from our object of interest is proportional to L(θp;Δθp). We'll define the constant of proportionality to be θp and the phase offset associated with the distance to our object of interest to be φp. This gives:

  • I pp ·Lp, Δθp), cos(φp),

  • Q p=−ρp ·Lp, Δθp)sin(φp).
  • Similarly, the magnitude of light from the background reaching our sensor is proportional to R(θp,Δθp). The constant of proportionality is defined to be ρbg, and the phase associated with the background distance to be φbg. This gives us:

  • I bgbg ·Rp, Δθp)·cos(φbg),

  • Q bg=−ρbg ·Rp, Δθp)·sin(φbg).

  • Upon summing:

  • I m =I p +I bg +I c

  • Q m =Q p +Q bg +Q c.
  • Assuming measurement of:

  • I openbg·cos(φbg)+I c, and

  • Q open=−ρbg·sin(φbg)+Q c.
  • If the angle θp and width w, are known or may be assumed, this system of equations may be solved.
    Uniform Background and Uniform Foreground with Partial Occlusion
  • For this model, start with the ‘Uniform Background with Partial Occlusion’ model, and build upon it, adding a foreground component that is uniform and has no spatially varying effect on the object of interest. Since the foreground components are not spatially varying, and are not affected by the presence of the object of interest, define pfg and φfg to be the magnitude and phase of the foreground object. Now, for the foreground:

  • I fgfg·cos(φfg), and

  • Q fg=−ρfg·sin(φfg).
  • This can simply add into the previous model to get:

  • I m =I p +I bg +I fg +I c, and

  • Q m =Q p +Q bg +I fg +Q c.
  • Assuming that in the empty scene it can be measured:

  • I open =I bg +I fg +I c, and

  • Q open =Q bg +I fg +Q c.
  • Two more variables may be added that are estimated as compared to the previous case.
  • Sectioned Background, Uniform Foreground
  • This model partitions the horizontal field of view into a series of sections 1 . . . S, each of which is modeled as a uniform foreground/uniform background. A superscript s is added to denote the section to which a variable belongs. Starting with the background sections, assume that an object is in the scene with width w corresponding to an angular section Δθp, and angular position θp. Redefine the R function sectionally to represent the fraction of light cast on the background after occlusion by the object of interest. It may be referred to as Rs.
  • Now define:

  • I bg sbg s ·R sp, Δθp)·cos(φbg s), and

  • Q bg s=−ρbg s ·R sp, Δθp)·sin(φbg s).
  • Since the foreground signal is not changed by an object in the scene, there is no need to model it sectionally. However, the foreground may occlude the object of interest to varying degrees across sections. This could be modeled in a number of different ways, the cleanest of which would be to associate an ‘occlusion factor’ Fs for each foreground section. Also, Ls is defined as the fraction of total light output from the LED that illuminates the objects of interest in section s. Now:
  • I p = ρ p · cos ( φ p ) s = 1 S L s ( θ p , Δ θ p ) · F s , and Q p = - ρ p · sin ( φ p ) s = 1 S L s ( θ p , Δ θ p ) · F s .
  • In the uniform foreground case, Fs is equal to one for all sections and the equations collapse back down to the non-sectioned foreground case. In sum:
  • I m = I p + I fg + I c + s = 1 S I bg s , and Q m = Q p + I fg + Q c + s = 1 S Q bg 2 .
  • Here, two variables are added per section for background, and one variable per section for the foreground occlusion. The occlusion effect from foreground objects may be ignored, and then only the extra background variables are added in.
  • Two Sensors with Overlapping Fields of View
  • Two sensors with an overlapping field of view may be used. Consider only the overlapping portion of the field of view and looking at what sorts of information can be glean in this region, assume that each sensor has its own L(Δp;Δθp), where θp is referenced to a global coordinate system. These may be referred to as L1 and L2, and use superscripts to denote the sensor. Further assume that the two sensors may differ in their sensitivity and LED output, and that this results in a scale factor error, a, for measurements of the same object in the overlapping field of view. Also assume a 1/d2 relationship for distance and signal magnitude from the object of interest. Further assume that the object has a fixed reflectivity ρp and fixed width w.
  • Note that ρp, φp, θp and d are common values between the two sensor measurements, and are specific to the object of interest. There is a well defined relationship between d and φp—see the example section herein. Here, α is a constant sensitivity difference between the two sensors/LEDs, which should be slowly changing over the lifetime of the sensors. With these definitions:
  • I p 1 = L 1 ( θ p , Δ θ p ) · ρ p · 1 d 2 · cos ( φ p ) , Q p 1 = - L 1 ( θ p , Δ θ p ) · ρ p · 1 d 2 · sin ( φ p ) , I p 2 = α · L 2 ( θ p , Δ θ p ) · ρ p · 1 d 2 · cos ( φ p ) , and I p 2 = - α · L 2 ( θ p , Δ θ p ) · ρ p · 1 d 2 · sin ( φ p ) .
  • These equations may be substituted for Ip and Qp into the background only partial occlusion model and generate equations for (Im 1,Qm 1) and (Im 2,Qm 2). There are five unknowns:
    • α
    • ρp
    • φp
    • θp
    • Δθp
  • Additionally, there are four equations, so as long as one of these values is known (or may be assumed), the remaineder could potentially be calculated. It is reasonable to assume that a good initial guess at α, and Δθp may be made. Once another sensor, such as the camera based sensor 106 is provided, for example, direct measurements for θp and φp may be obtained. Unfortunately these equations are non-linear, so some work may still be done to show that a unique solution exists within these constraints. To accomplish this estimation process, any of a number of estimation schemes may be utilized. Examples may include using an extended Kalman filter, sigma-point Kalman filter, or direct estimation.
  • Example Implementation of Background Only Partial Occlusion Model
  • The falloff cast by the 10 degree LED sensor 112 was imaged against a white wall. Its horizontally projected falloff is approximately gaussian with a standard deviation of roughly 12 degrees. The prototype was placed about 3.5 ft above the floor in a relatively dark, empty room with a backdrop at 12 ft.
  • The crosstalk was measured with a black felt baffle covering the sensor 112. The zero phase offset was measured with a reflective baffle. The nominal ‘open’ background was captured. Sensor data was collected with a person standing at 1 ft increments out from the sensor 112 at a 0 degree offset from the LED out to 10 ft. Sensor data was collected in 5 degree increments at a radial distance of 5 ft, from −15 degrees to +15 degrees. The felt measurements may be referred to as (Ic,Qc), as it essentially measures crosstalk. The reflective baffle measurements may be referred to as (I0; Q0) and the open measurements as (Iopen;Qopen). Finally, the raw measurements with the object of interest in the scene may be referred to as (Im;Qm) and the to-be-estimated object of interest signal as (Ip;Qp). L(θp,Δθp) was modeled assuming the Gaussian distribution mentioned above, whose specific form becomes:
  • L ( θ p , Δ θ p ) = 0.5 · ( erf ( 1 12 · ( θ p + Δ θ p 2 ) ) - erf ( 1 12 · ( θ p - Δ θ p 2 ) ) ) ;
  • where “erf” is the error function. Also define:
  • φ O = arctan ( - Q O I O ) φ p = arctan ( - Q p I p ) d p = γ · ( φ O - φ p ) , and Δ θ p = 2 · arctan ( 1 d p ) ;
  • where γ is the conversion from phase delta to distance, and Δθp is calculated assuming person with a width of 2 ft. Now the system of equations may be set up:

  • I m =I p+(1.0−Lp, Δθp))·(I open −I c)+I c, and

  • Q m =Q p+(1.0−Lp, Δθp))·(Q open −Q c)+Q c;
  • where L(θp; Δθp) is expressed using the above equations for Δθp and L(θp;Δθp). Treat θp as a known value, and solve the system of non-linear equations numerically. Results with real data are shown in the plots of FIGS. 3 and 4. In FIG. 3, line 120 represents no correction and line 122 represents corrected data. In FIG. 4, line 130 represents no correction, line 132 represents corrected data, and line 134 represents true distance.
  • With the mathematics for a single sensor 112 with various background models, multiple sensors may be combined into an integrated position model. As mentioned above, multiple proximity sensors may be implemented in one embodiment. In other embodiments, multiple LEDs may be used in a TDM manner to provide a desired field of view. Integrating a camera based sensor should allow estimation all parameters of interest.
  • FIG. 5 illustrates a method 300 for utilizing multiple sensors in a tiered manner to change the state of the device. Initially, the device may be in a reduce power consumption mode such as a sleep mode, and a controller may receive data from a first sensor (Block 302). The received data is processed (Block 304) and compared with a threshold (Block 306). The comparison with the threshold allows for a determination as to whether the user is present or is likely present (Block 308). If the user is not present, data may continue to be received from the first sensor (Block 302). If however, the user is determined to be present or is likely to be present, a second sensor may be actuated (Block 310) and data is received from the second sensor (Block 312). The data from the second sensor is processed (Block 314) and combined with the data from the first sensor data (Block 316). The processing of data from the first and second sensor may include, but is not limited to, performing digital signal processing on the data such as filtering the data, scaling the data, and/or generally conditioning the data so that it is useful for presence determination. Additionally, the combination of data from the first and second sensors may include storing the data together and/or logically or mathematically combining the data.
  • The data from the first and second sensors is used to compute user presence values and/or probability of user presence scores (Block 318). The user presence values and/or probability of user presence scores are compared with thresholds to determine if a user is present (Block 322). Further, if the user is determined to be present, other parameters may be determined such as distance and location of the user relative to the device (Block 324) and the state of the device may be changed (Block 326). The state change may include bringing the device into an awake mode from a sleep mode or other suitable state change.
  • Additionally, it should be appreciated that the determination of other parameters (e.g., distance, location, and so forth) as well as the change in state of the device may occur after a positive determination of user presence based solely on the first sensor data as indicated by the dashed line from Block 308.
  • Further, the second determination of user presence (Block 320) may be more accurate than the first determination (Block 308) based on the additional information provided from the second sensor. Moreover, as mentioned above, additional parameters may be determined based on the combination of data from both the first and second sensors.
  • It should be appreciated that the other embodiments may implement more or fewer steps than the method 300. FIGS. 6-11 illustrate more detailed flowcharts of methods for presence determinations.
  • Turning to FIG. 6, a flowchart 200 illustrating presence sensing is illustrated. Initially, a camera is used to obtain an image (Block 202). A light level determination may be made (Block 204) and provided to a skin tone detection routine (Block 206). Optionally, in some embodiments, the light level determination may be provided to other routines as well, as indicated by arrow 203. Additionally, the captured image may be pre-processed (Block 208). In some cases, the preprocessing may include down scaling of the image, changing the color space of the image and/or enhancing the image, for example. Other detector specific preprocessing may also be performed ( Blocks 214, 215 and 217). For example, the image may optionally be blurred by preprocessing in Block 214 before being provided to the skin tone detection routine (Block 206). Additionally, preprocessing in Block 215 may include changing color into grayscale before providing the image to the face detection routine (Block 210) and/or performing edge detection in the preprocessing of Block 217 before providing the image to the movement detection routine (Block 212). The skin tone detection routine, face detection routine and movement detection routine are discussed in greater detail below with reference to FIGS. 7-11.
  • The results of the skin tone detection routine, face detection routine and movement detection routine may be weighted and combined using fusion and detection logic (Block 216) and a user presence classification is determined (Block 218). The fusion and detection logic may include the use of neural networks, support vector machines, and/or some other form of probabilistic machine learning based algorithm to arrive at a determination of whether a user is present. FIG. 7 illustrates the skin tone detection routine (Block 206) as a flowchart starting with the low light determination (Block 204). It should be appreciated that the low light determination may be utilized in a variety of different manners to influence the processing of the image. For example, in some embodiments, the low light determination may be provided as a vector to a neural network, while in other embodiments the low light determination may be used to select a particular type of categorizer to be used. That is, if it is determined that the image was not taken in low light, feature vectors may be generated (Block 220), and a first pixel classifier is applied (Block 222). If the image was captured in low light, a different set of feature vectors may be generated (Block 224) and a second per pixel classifier may be applied (Block 226). The type of features, e.g., color conversion, and so forth may be selectively provided to achieve desired results and may be different depending on the low light determination. Additionally, the first and second per pixel classifiers may be different based on the low light determination. For example, the first classifier may be a 7-5-2 multilayer perceptron (MLP) feed forward neural network, per pixel classifier, while the second classifier may be a 2-12-2 MLP feed forward neural network per pixel classifier. In some embodiments, the classifiers may be implemented in an open kernel with a GPU to help speed up the process.
  • The output from the classifiers may be a probability (e.g., a value between 0 and 1) that indicates a probability that the image includes a skin tone. A morphology filter may optionally be applied to the image (Block 228) and an average grey scale level may be calculated (Block 230). Further, nonlinear scaling (Block 232), a temporal queue filter (Block 234) and a clamp (Block 236) may be applied before determining a probability of user presence due to skin tone detection (Block 238).
  • FIG. 8 illustrates the face detection routine (Block 210) as a flowchart starting with applying a face detector (Block 240). Any suitable face detector may be implemented, such as a Viola-Jones cascade face detector, for example, that provides a probabilistic score indicating the likelihood that a face is present. The face presence score is then scaled (Block 242) and an intermittent detection flicker filter optionally may be applied (Block 244) to smooth the image. It should be appreciated that in some embodiments, such as where the camera is of a relatively good quality, smoothing may be omitted from the process. The flicker filter may include a temporal queue filter (Block 246), determination as to whether the normalized score deviation from average is less than a threshold (Block 248) and then multiplying an output value with the scaled score (Block 250). A temporal queue filter (Block 252) and a clamp (Block 254) are applied before determining a probability of user presence due to face detection (Block 256).
  • FIG. 9 illustrates the motion detection routine 212 as a flowchart starting by collecting multiple frames and, as such, memory may be implemented to store the multiple frames. For example, three frames may be utilized, a current frame and two other frames. In the embodiment illustrated in FIG. 9, a current frame and two subsequent frames are used. Initially, an input frames are delayed by k frames, sequentially, (Blocks 260, 262) and fed forward to be added (Block 264) with the output of the second delay (Block 262). The output of Block 260 is multiplied by 2 (Block 266) and a difference (Block 268) between the adder (Block 264) and the output of the multiplier (Block 266) is determined. A per pixel inner product is then determined (Block 270), scaled (block 272), and pixels are clamped (Block 274). An average gray level is calculated (Block 276), nonlinear scaling is performed (Block 278), temporal queue filter is applied (Block 280) and the clamped to [0,1] (Block 282). Finally, a probability of user presence due to movement is determined (Block 284).
  • Some parameters that may be useful in the motion detection routine may include an auto focus (AF) window statistics or horizontal edges or Sobel/Sharr edges, 2D color histogram data, component histogram data, automatic white balance/auto exposure (AWB/AE) window statistics of color content, and so forth. Some preprocessing steps may be implemented for the motion detection such as Y channel (intensity), gradient magnitude computed with either Sobel or Scharr filters (accumulating the gradient for proper normalization), threshold gradient magnitude (normalization by count of edge pixels), skin-probability in chrominance (Cr, Cb) space, sub-images of any of the foregoing, and so forth. In some embodiments, the motion detection may include the ability to computer image centroids. The distance from the centroid for the current frame to the centroid of the previous frame is used as a measure of the amount of motion, and a hard threshold is applied to produce a binary detection or motion. Hence, for example, a change in centroid location of either a Y′-intensity, edge gradient magnitude, binary edge, or skin probability image may indicate motion. Sensitivity and robustness tradeoffs may dictate a particular combination of parameters being used. For example, skin probability image may be used with edge-gradient and binary edges may be utilized to provide robustness to lighting changes. The skin probability may be performed with a neural network, as mentioned above or alternatively using an auto white balance color space filters to approximate this functionality.
  • Some embodiments for sensing motion may refer to window statistics of either skin detection of gradient images. One embodiment may look at a change in a global sum. In particular the image is summed over the entire frame to produce a scalar value s[i], where i is current frame index. A queue of the past N values is maintained: S={s[i-1], s[i-2], . . . s[i-N]}. SL,N is denoted as the sequence from s[i-L] to s[i-N], and the extremum of these values is computed as u=max(SL,N) and v=min(SL,N). The amount of motion is determined by the excursion outside of this range: e=max (s[i]−u, v−s[i]). Motion is detected if e exceeds a predetermined threshold.
  • In some embodiments, the motion detection routine may be implemented with a single frame and thus little or no memory may be used, as full frames will not be stored. In some embodiments, the image (a single frame) may be divided into windows for which statistics may be calculated. Changes in the statistics for the windows may be used to determine motion and also position of a user.
  • FIG. 10 illustrates possible sets of windows into which the images may be divided for single frame motion detection. Specifically, FIG. 10 shows single frames divided into both non-overlapping windows and concentric windows for statistical purposes. In each case, the luminace of the images (frames at the top of page) and intensity gradient magnitude of the images (frames at the bottom of the page) are considered. For example, an image 300 may be divided into multiple non-overlapping exposure statistics windows 302. Alternatively, the image 300 may be divided into multiple concentric overlapping exposure statistics windows 304. The statistics for each window may be determined based on a luminance image (as in image 300) or based on an intensity gradient magnitude image 306.
  • The use of the windows provided more robust capture of motion when computing sums of the gradient magnitude. With respect to the overlapping rectangular windows, one embodiment includes eight concentrically arranged rectangles, the largest contains the entire frame and the smallest is centered in the image. Thus, at frame i, the sum of the frames is sj[i] for j ε [1,2, . . . 8]. The sums in the strips of pixels lying between the rectangles is computed as the difference of these sums: dj [i]=sj [i]−sj+1[i]/hj, except for the special case d8[i]=s8 [i]/h8. The differences are normalized by the height of the strip hj, which is approximately proportional to its area.
  • Next, the extrema u, v, (maximum and minimum) of the differences dj over the previous N frames is computed using queues, and the excursions ei=max (dj[i]−u, v−dj[i]). Comparing each excursion ej to a threshold gives an indicator of motion in region j of the frame. Subtle changes in lighting can produce false positive detections. True motion is usually associated with detection in two or three of the eight regions. Hence, in some embodiments, at least two regions detecting motion are required for a determination that motion has been detected in the frame. Additionally, large changes in lighting such as turning on or off the room lights, often results in many regions showing motion detection. Hence, detection may be suppressed if more than three regions have detected motion. These design parameters may be adjusted based on experience or to provide a desired level of sensitivity and robustness
  • FIG. 11 is a flow chart illustrating the motion detection routine 212B using a single frame 300 with non-concentric windows 302. In some embodiments, each statistics window may be provided with a unique analysis pipe. That is, each window may be concurrently processed. In other embodiments, one or more windows may be processed sequentially in a common analysis pipe. As used herein, “analysis pipe” may refer to the processing steps associated with the statistical analysis of the windows and may include the temporal queue filter.
  • As discussed above, the image 300 may be divided into statistical windows 302 and an average statistic may be calculated for each window (Block 310). A temporal queue filter may be applied to the average statistic (Block 312) and an excursion value “e” from short term past behavior may be calculated (Block 314). The excursion value may be compared with a threshold to determine if it exceed the threshold (Block 316) A normalized count is kept for each excursion value that exceeds the threshold (Block 318) and if the normalized count exceed a voting threshold (Block 320), it is determined that motion has been detected (Block 322).
  • The excursion value may also be used to generate summary statistics (Block 324). The summary statistics may be nonlinearly scaled (Block 326) and a probabilistic motion score may be provided (Block 328). Generally, Block 322 will provide a binary one or a zero output indicating motion has been detected, whereas Block 328 will provide a value between zero and one indicating the likelihood that motion has been detected in the image 300.
  • As may be appreciated, a neural network, support vector machine (SVM) or other classification system may be utilized in each of the aforementioned routines to make a determination as to presence of a user. Additionally, a probability value of from each of the routines may be alone enough to make the determination that a user is present, for example, if a value Is above a certain threshold. Moreover, in some embodiments a combination of the probabilities from each of the routines may be used in determining if a user is present. In some embodiments, the output of a routine may not be used as its validity may be questionable. For example, the skin tone detection routine may be unreliable due to the lighting. Moreover, the probabilities output from the routines may be combined in a weighted manner (e.g., one probability may be given more weight based on the likelihood of it being more accurate than the others).
  • Embodiments described herein may be implemented to reduce power consumption in computing devices, such as notebook computers, desktop computers, and so forth. In particular, the computing devices may provide presence sensing functionality even when the device is in a low power state, such as a hibernate or sleep state, so that the device may power up when a user is present and power down or enter a reduced power state when the user leaves. Further, the embodiments may be provided to improve a user experience with computing devices by providing intuitive powering up and powering down operation, as well as security features, among others.
  • A tiered system may be implemented that precludes the use of a main processor and RAM in low power operation. For example, in one embodiment, a lowest power state may implement only a camera, an image signal processing (ISP) device and an embedded processor that may calculate a presence value in real-time. In a next tier, a face detector chip and RAM may be turned on. In the subsequent tier, the system processor and other resources may be powered on.
  • In the examples described above for motion detection, therefore, memory of previous frames is limited to statistics computed by the ISP, and limited to the space available in the embedded processor registers and cache (e.g., 32 k), as RAM may not be available. Additionally, it should be appreciated, that the presence sensing information (e.g., statistics, images, and so forth) are not available outside of the presence sensing routines. That is, for example, an image captured for presence sensing may not be viewed by a user.
  • The foregoing describes some example embodiments for sensing the presence of user. Although the foregoing discussion has presented specific embodiments, persons skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the embodiments. For example, modifications to one or more of the algorithms for presence sensing may be implemented. In one example, hardware limitations may drive the algorithm changes. Accordingly, the specific embodiments described herein should be understood as examples and not limiting the scope thereof.

Claims (21)

1. A method of operating a computing device to provide presence based functionality, the method comprising:
operating the computing device in a reduced power state;
collecting a first set of data from a first sensor;
determining, based on the first set of data, if an object is within a threshold distance of the computing device;
if the object is within the threshold distance, activating a secondary sensor to collect a second set of data;
determining, based on the second set of data, if the object is a person;
if the object is a person:
determining a position of the person relative to the computing device; and
executing a change of state in the computing device based on the position of the person relative to the computing device; and
if the object is not a person, maintaining the computing device in a reduced power state.
2. The method of claim 1 further comprising operating the first sensor in a time division multiplexed manner.
3. The method of claim 1 further comprising using multiple light sources covering discrete/different fields of view.
4. The method of claim 1 further comprising changing modulation frequencies if multiple sensors are in the same room.
5. The method of claim 1 wherein making a change in state comprises at least one of:
changing a display background/screen saver shift, wherein the shift corresponds to movement of the user;
bringing the display to an awake state;
bringing the system to an awake state;
reducing a number of processes when bringing the system to an awake state;
audio steering based on position of user;
microphone steering based on user position; and
modifying the user interface so that if the user is far away a smaller set of user options are provided.
6. The method of claim 5 wherein if the system is already awake and detects that the user moves away from the system, the system is configured to:
change display state to a sleep state; or
change system state to a sleep state.
7. The method of claim 1 further comprising using combined sensor data to determine presence using a neural net or a support vector machine.
8. The method of claim 1 further comprising using combined sensor data to determine presence using a probabilistic determination.
9. The method of claim 1 further comprising using combined sensor data to determine presence using each of a skin tone determination, a presence determination, and a movement determination in a weighted manner to make the presence determination.
10. The method of claim 9, wherein the skin tone determination is weighted less than at least one of the presence determination and the movement determination.
11. The method of claim 1 further comprising using combined sensor data to determine presence using a determined distance of a user from the system.
12. The method of claim 1 further comprising determining a number of faces in proximity of the system for security, wherein security is provided by powering up partially or powering up into a secure state and requesting credentials for further access.
13. A method for determining if a user is in proximity of a computing device, the method comprising:
capturing an image using an image sensor;
computing, using a processor, at least one of the following from the captured image:
a skin tone detection parameter;
a face detection parameter; and
a movement detection parameter;
utilizing at least one of the skin tone detection parameter, face detection parameter and the movement detection parameter to make a determination as to whether a user is present; and
if it is determined that a user is present, changing a state of the computing device.
14. The method of claim 13, wherein at least the face detection and movement detection parameters are calculated and utilized in making the determination as to whether a user is present.
15. The method of claim 13 further comprising:
calculating each of the skin tone detection, face detection, and movement parameters; and
weighting the parameters relative to each other, wherein the skin tone detection parameter is weighted less than the other parameters.
16. The method of claim 13, wherein the movement parameter is calculated using a single frame.
17. The method of claim 16 further comprising:
dividing the image into concentric windows; and
computing statistics for the concentric windows, wherein changes in at least two windows indicate movement in the image.
18. The method of claim 16 further comprising:
dividing the image into non-concentric windows; and
computing statistics for the non-concentric windows, wherein changes in at least two windows indicate movement in the image.
19. A computing system comprising:
a main processor;
a image based presence sensor coupled to the main processor comprising:
an image sensor;
a processor coupled to the image sensor, the processor configured to process the image to determine if a user is present in the image;
wherein if the processor determines that a user is present in the image, an indication that a user has been determined to be present is sent from the processor to the main processor and the main processor changes a state of the computing system based on the indication.
20. The computing system of claim 19, wherein the main processor activates a secondary presence sensor based on the indication that a user is present and data from the image based presence sensor and secondary presence sensor are used together to determine at least one of: if a user is present, where the user is located relative to the computer system, or a distance the user is located from the computer system.
21. The computing system of claim 19, wherein:
if the image based presence sensor determines that a user is present, the image based presence sensor determines at least one of: a number of user's present, a position of the user relative to the computer system, or a distance of the user from the system;
the image based presence sensor provides additional information regarding at least one of: a number of user's present, a position of the user relative to the computer system, or a distance of the user from the system, to the main processor; and
the main processor further changes the state of the computing system based on the additional information.
US13/219,573 2011-05-12 2011-08-26 Presence sensing Abandoned US20120287031A1 (en)

Priority Applications (11)

Application Number Priority Date Filing Date Title
US13/219,573 US20120287031A1 (en) 2011-05-12 2011-08-26 Presence sensing
KR1020167003580A KR101660215B1 (en) 2011-05-12 2012-05-11 Presence sensing
CN201280023139.7A CN103534664B (en) 2011-05-12 2012-05-11 There is sensing
PCT/US2012/037644 WO2012155105A1 (en) 2011-05-12 2012-05-11 Presence sensing
AU2012253292A AU2012253292B2 (en) 2011-05-12 2012-05-11 Presence sensing
US13/469,996 US10372191B2 (en) 2011-05-12 2012-05-11 Presence sensing
EP12722259.4A EP2712431A1 (en) 2011-05-12 2012-05-11 Presence sensing
KR1020137029536A KR20140002034A (en) 2011-05-12 2012-05-11 Presence sensing
JP2014510519A JP5876142B2 (en) 2011-05-12 2012-05-11 Presence sensor
TW101117124A TWI506478B (en) 2011-05-12 2012-05-14 Presence sensing
US15/445,623 US10402624B2 (en) 2011-05-12 2017-02-28 Presence sensing

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201161485610P 2011-05-12 2011-05-12
US201161504026P 2011-07-01 2011-07-01
US13/219,573 US20120287031A1 (en) 2011-05-12 2011-08-26 Presence sensing

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US13/469,996 Continuation-In-Part US10372191B2 (en) 2011-05-12 2012-05-11 Presence sensing
US15/445,623 Continuation US10402624B2 (en) 2011-05-12 2017-02-28 Presence sensing

Publications (1)

Publication Number Publication Date
US20120287031A1 true US20120287031A1 (en) 2012-11-15

Family

ID=47141552

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/219,573 Abandoned US20120287031A1 (en) 2011-05-12 2011-08-26 Presence sensing
US15/445,623 Expired - Fee Related US10402624B2 (en) 2011-05-12 2017-02-28 Presence sensing

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/445,623 Expired - Fee Related US10402624B2 (en) 2011-05-12 2017-02-28 Presence sensing

Country Status (1)

Country Link
US (2) US20120287031A1 (en)

Cited By (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130027548A1 (en) * 2011-07-28 2013-01-31 Apple Inc. Depth perception device and system
US20130057573A1 (en) * 2011-09-02 2013-03-07 DigitalOptics Corporation Europe Limited Smart Display with Dynamic Face-Based User Preference Settings
US20130194172A1 (en) * 2012-01-30 2013-08-01 Cellco Partnership D/B/A Verizon Wireless Disabling automatic display shutoff function using face detection
US20130215218A1 (en) * 2012-02-21 2013-08-22 Sri Rama Prasanna Pavani Optical Target Detection
US20130222611A1 (en) * 2012-02-23 2013-08-29 Pixart Imaging Inc. Electronic apparatus having camera sensing circuit and object sensing circuit used for switching between two different operation modes and related computer-readable medium
US20130229508A1 (en) * 2012-03-01 2013-09-05 Qualcomm Incorporated Gesture Detection Based on Information from Multiple Types of Sensors
US20130238700A1 (en) * 2012-03-09 2013-09-12 Matthew Nicholas Papakipos Dynamic Duty-Cycling of Processor of Mobile Device Based on Operating Condition of Mobile Device
US20130275738A1 (en) * 2012-04-11 2013-10-17 Quanta Computer Inc. Notebook computer
US20130321312A1 (en) * 2012-05-29 2013-12-05 Haruomi HIGASHI Information processing apparatus, information display system and information display method
US20140012528A1 (en) * 2012-07-04 2014-01-09 Research In Motion Limited Estimating a baseline of a proximity sensor
CN103631359A (en) * 2013-11-15 2014-03-12 联想(北京)有限公司 Information processing method and electronic equipment
US8760517B2 (en) 2010-09-27 2014-06-24 Apple Inc. Polarized images for security
WO2014099817A1 (en) * 2012-12-17 2014-06-26 Fluidmaster, Inc. Touchless activation of a toilet
US8787939B2 (en) 2012-03-27 2014-07-22 Facebook, Inc. Dynamic geographic beacons for geographic-positioning-capable devices
US20140232843A1 (en) * 2011-10-27 2014-08-21 Robert Campbell Gain Value of Image Capture Component
CN104122974A (en) * 2013-04-24 2014-10-29 华硕电脑股份有限公司 Mobile communication device with human body recognition function
WO2014178836A1 (en) 2013-04-30 2014-11-06 Hewlett-Packard Development Company, L.P. Depth sensors
CN104281377A (en) * 2013-07-04 2015-01-14 联想(北京)有限公司 Electronic equipment and method and device for controlling screen of electronic equipment
WO2015017931A1 (en) 2013-08-07 2015-02-12 Blackberry Limited Determining the distance of an object to an electronic device
US20150054984A1 (en) * 2012-03-15 2015-02-26 Nec Display Solutions, Ltd. Image display apparatus and image display method
WO2015138095A1 (en) * 2014-03-11 2015-09-17 Google Technology Holdings LLC Display viewing detection
US20150332032A1 (en) * 2014-05-13 2015-11-19 Google Technology Holdings LLC Electronic Device with Method for Controlling Access to Same
WO2015183924A1 (en) * 2014-05-29 2015-12-03 Google Technology Holdings LLC Electronic device and method for controlling access to same
US9245497B2 (en) 2012-11-01 2016-01-26 Google Technology Holdings LLC Systems and methods for configuring the display resolution of an electronic device based on distance and user presbyopia
US9426236B2 (en) 2012-07-10 2016-08-23 Facebook, Inc. Determining a location and area of a place based on distances between the first mean and check in locations
US9428897B2 (en) 2012-12-17 2016-08-30 Fluidmaster, Inc. Touchless activation of a toilet
US9491603B2 (en) * 2012-03-09 2016-11-08 Facebook, Inc. Substantially continuous location logging for geographic-positioning capable devices
CN106445065A (en) * 2015-08-11 2017-02-22 三星电子株式会社 Method for controlling according to state and electronic device thereof
WO2017083178A1 (en) * 2015-11-12 2017-05-18 Microsoft Technology Licensing, Llc Adaptive user presence awareness for smart devices
US20170147888A1 (en) * 2015-11-20 2017-05-25 GM Global Technology Operations LLC Stixel estimation methods and systems
US9678559B1 (en) * 2015-09-18 2017-06-13 Amazon Technologies, Inc. Determining a device state based on user presence detection
US20190141181A1 (en) * 2017-11-07 2019-05-09 Google Llc Sensor Based Component Activation
US10297208B2 (en) * 2012-03-05 2019-05-21 Pixart Imaging Inc. Automatic detection method for backlight control and portable electronic device using the same
US10372191B2 (en) 2011-05-12 2019-08-06 Apple Inc. Presence sensing
CN110192193A (en) * 2017-01-19 2019-08-30 惠普发展公司有限责任合伙企业 Secret protection equipment
US10402624B2 (en) 2011-05-12 2019-09-03 Apple Inc. Presence sensing
USD886245S1 (en) 2018-04-26 2020-06-02 Bradley Fixtures Corporation Dispenser
USD886240S1 (en) 2018-04-26 2020-06-02 Bradley Fixtures Corporation Faucet and soap dispenser set
US20200244861A1 (en) * 2019-01-25 2020-07-30 Pixart Imaging Inc. Light sensor chip, image processing device and operating method thereof
US10817594B2 (en) 2017-09-28 2020-10-27 Apple Inc. Wearable electronic device having a light field camera usable to perform bioauthentication from a dorsal side of a forearm near a wrist
US10923045B1 (en) * 2019-11-26 2021-02-16 Himax Technologies Limited Backlight control device and method

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10657363B2 (en) 2017-10-26 2020-05-19 Motorola Mobility Llc Method and devices for authenticating a user by image, depth, and thermal detection
US10630866B2 (en) 2018-01-28 2020-04-21 Motorola Mobility Llc Electronic devices and methods for blurring and revealing persons appearing in images
US10757323B2 (en) 2018-04-05 2020-08-25 Motorola Mobility Llc Electronic device with image capture command source identification and corresponding methods
US11605242B2 (en) 2018-06-07 2023-03-14 Motorola Mobility Llc Methods and devices for identifying multiple persons within an environment of an electronic device
EP3818693A4 (en) 2018-07-02 2021-10-13 Stowers Institute for Medical Research Facial image recognition using pseudo-images
US11100204B2 (en) 2018-07-19 2021-08-24 Motorola Mobility Llc Methods and devices for granting increasing operational access with increasing authentication factors
DE102018124385B4 (en) * 2018-10-02 2023-02-16 Infineon Technologies Ag display control devices and methods
US11736789B2 (en) * 2021-06-16 2023-08-22 Dell Products L.P. Peripheral camera and information handling system security system and method
US11656532B2 (en) 2021-06-16 2023-05-23 Dell Products L.P. Cylindrical camera dual leaf shutter

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5903350A (en) * 1997-02-03 1999-05-11 Optiphase, Inc. Demodulator and method useful for multiplexed optical sensors
US20020147931A1 (en) * 2001-02-08 2002-10-10 Chu-Kung Liu Computer device for sensing user status and computer system for direct automatic networking
US20050062460A1 (en) * 2003-09-05 2005-03-24 Blake Jame N. Time division multiplexed optical measuring system
US20050071698A1 (en) * 2003-09-30 2005-03-31 Kangas Paul Daniel Apparatus, system, and method for autonomic power adjustment in an electronic device
US20080199049A1 (en) * 2007-02-21 2008-08-21 Daly Scott J Methods and Systems for Display Viewer Motion Compensation Based on User Image Data
US20100328074A1 (en) * 2009-06-30 2010-12-30 Johnson Erik J Human presence detection techniques
US20110262044A1 (en) * 2010-04-21 2011-10-27 Elitegroup Computer Systems Co., Ltd. Energy saving method for electronic device
US20110296163A1 (en) * 2009-02-20 2011-12-01 Koninklijke Philips Electronics N.V. System, method and apparatus for causing a device to enter an active mode
US20120212319A1 (en) * 2011-02-18 2012-08-23 Research In Motion Limited System and method for activating an electronic device using two or more sensors

Family Cites Families (61)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2485773A1 (en) 1980-06-24 1981-12-31 Promocab SYSTEM FOR PROTECTING A ZONE AGAINST HUMAN AGGRESSION
US5537144A (en) 1990-06-11 1996-07-16 Revfo, Inc. Electro-optical display system for visually displaying polarized spatially multiplexed images of 3-D objects for use in stereoscopically viewing the same with high image quality and resolution
US5384588A (en) 1991-05-13 1995-01-24 Telerobotics International, Inc. System for omindirectional image viewing at a remote location without the transmission of control signals to select viewing parameters
DE69717139T2 (en) 1996-12-04 2003-10-02 Matsushita Electric Ind Co Ltd DEVICE FOR RECORDING AND PLAYING BACK HIGH-RESOLUTION AND THREE-DIMENSIONAL IMAGES WITH OPTICAL DISK
US6072894A (en) 1997-10-17 2000-06-06 Payne; John H. Biometric face recognition for applicant screening
US6146332A (en) 1998-07-29 2000-11-14 3416704 Canada Inc. Movement detector
US7006236B2 (en) 2002-05-22 2006-02-28 Canesta, Inc. Method and apparatus for approximating depth of an object's placement onto a monitored region with applications to virtual interface devices
US6775397B1 (en) 2000-02-24 2004-08-10 Nokia Corporation Method and apparatus for user recognition using CCD cameras
US20020158750A1 (en) 2001-04-30 2002-10-31 Almalik Mansour Saleh System, method and portable device for biometric identification
JP2002350555A (en) 2001-05-28 2002-12-04 Yamaha Motor Co Ltd Human presence detector
US6832006B2 (en) 2001-07-23 2004-12-14 Eastman Kodak Company System and method for controlling image compression based on image emphasis
US7136513B2 (en) 2001-11-08 2006-11-14 Pelco Security identification system
US7164781B2 (en) 2002-03-04 2007-01-16 Samsung Electronics Co., Ltd. Method and apparatus of recognizing face using 2nd-order independent component analysis (ICA)/principal component analysis (PCA)
US7369685B2 (en) 2002-04-05 2008-05-06 Identix Corporation Vision-based operating method and system
AU2003228482A1 (en) 2002-04-10 2003-10-27 Kendro Laboratoryb Products, Lp Automated protein crystallization imaging
US7151530B2 (en) 2002-08-20 2006-12-19 Canesta, Inc. System and method for determining an input selected by a user through a virtual interface
US7436569B2 (en) 2003-03-12 2008-10-14 General Photonics Corporation Polarization measurement and self-calibration based on multiple tunable optical polarization rotators
JP2004348349A (en) 2003-05-21 2004-12-09 Masayuki Watabiki Household appliance power controller
US7324824B2 (en) 2003-12-09 2008-01-29 Awarepoint Corporation Wireless network monitoring system
KR100580630B1 (en) 2003-11-19 2006-05-16 삼성전자주식회사 Apparatus and method for discriminating person using infrared rays
US20050221791A1 (en) 2004-04-05 2005-10-06 Sony Ericsson Mobile Communications Ab Sensor screen saver
US8509472B2 (en) 2004-06-24 2013-08-13 Digimarc Corporation Digital watermarking methods, programs and apparatus
EP1672460B1 (en) * 2004-12-15 2009-11-25 STMicroelectronics (Research & Development) Limited Computer user detection apparatus
US9760214B2 (en) 2005-02-23 2017-09-12 Zienon, Llc Method and apparatus for data entry input
JP2006318364A (en) 2005-05-16 2006-11-24 Funai Electric Co Ltd Image processing device
US7806604B2 (en) 2005-10-20 2010-10-05 Honeywell International Inc. Face detection and tracking in a wide field of view
KR100745981B1 (en) 2006-01-13 2007-08-06 삼성전자주식회사 Method and apparatus scalable face recognition based on complementary features
JP2007300363A (en) * 2006-04-28 2007-11-15 Omron Corp Monitor, method, and program
EP2030440B1 (en) 2006-06-06 2011-05-04 Koninklijke Philips Electronics N.V. Scaling an image based on a motion vector
JP2008160351A (en) * 2006-12-22 2008-07-10 Nec Electronics Corp Color motion detection circuit and y/c separation circuit
US7840031B2 (en) 2007-01-12 2010-11-23 International Business Machines Corporation Tracking a range of body movement based on 3D captured image streams of a user
US7929729B2 (en) 2007-04-02 2011-04-19 Industrial Technology Research Institute Image processing methods
US8233094B2 (en) * 2007-05-24 2012-07-31 Aptina Imaging Corporation Methods, systems and apparatuses for motion detection using auto-focus statistics
GB2453163B (en) 2007-09-26 2011-06-29 Christopher Douglas Blair Three-dimensional imaging system
US8564700B2 (en) 2007-12-07 2013-10-22 Panasonic Corporation Image processing device, image processing method, and imaging device using a patterned polarizer
US8600120B2 (en) 2008-01-03 2013-12-03 Apple Inc. Personal computing device control using face detection and recognition
WO2009107935A2 (en) 2008-02-28 2009-09-03 Lg Electronics Inc. Virtual optical input device with feedback and method of controlling the same
US8107721B2 (en) 2008-05-29 2012-01-31 Mitsubishi Electric Research Laboratories, Inc. Method and system for determining poses of semi-specular objects
US8035728B2 (en) * 2008-06-27 2011-10-11 Aptina Imaging Corporation Method and apparatus providing rule-based auto exposure technique preserving scene dynamic range
US20100008900A1 (en) 2008-07-14 2010-01-14 The University Of Hong Kong Annexin ii compositions for treating or monitoring inflammation or immune-mediated disorders
CN101639800A (en) 2008-08-01 2010-02-03 华为技术有限公司 Display method of screen and terminal
EP2315439B1 (en) 2008-08-22 2018-10-17 Saturn Licensing LLC Image display device, control method and computer program
WO2010021375A1 (en) 2008-08-22 2010-02-25 国立大学法人大阪大学 Process for production of protease, protease solution, and solution of pro-form of protease
JP4636149B2 (en) 2008-09-09 2011-02-23 ソニー株式会社 Image data analysis apparatus, image data analysis method, and program
DE102008047413A1 (en) 2008-09-16 2010-04-15 Hella Kgaa Hueck & Co. Motor vehicle object detection system has a stereo camera with two gray image sensors and a mono camera with a color image sensor
KR20100073191A (en) 2008-12-22 2010-07-01 한국전자통신연구원 Method and apparatus for face liveness using range data
US8654179B2 (en) 2008-12-25 2014-02-18 Panasonic Corporation Image processing device and pseudo-3D image creation device
US8254699B1 (en) * 2009-02-02 2012-08-28 Google Inc. Automatic large scale video object recognition
CN102378996B (en) 2009-03-31 2014-05-14 皇家飞利浦电子股份有限公司 Energy efficient cascade of sensors for automatic presence detection
US20100253782A1 (en) 2009-04-07 2010-10-07 Latent Image Technology Ltd. Device and method for automated verification of polarization-variant images
WO2010127488A1 (en) 2009-05-07 2010-11-11 Joemel Caballero System for controlling light emission of television
US8154615B2 (en) 2009-06-30 2012-04-10 Eastman Kodak Company Method and apparatus for image display control according to viewer factors and responses
US8264536B2 (en) 2009-08-25 2012-09-11 Microsoft Corporation Depth-sensitive imaging via polarization-state mapping
KR101648353B1 (en) 2009-09-25 2016-08-17 삼성전자 주식회사 Image sensor having depth sensor
US9357024B2 (en) 2010-08-05 2016-05-31 Qualcomm Incorporated Communication management utilizing destination device user presence probability
US8760517B2 (en) 2010-09-27 2014-06-24 Apple Inc. Polarized images for security
US20120287031A1 (en) 2011-05-12 2012-11-15 Apple Inc. Presence sensing
EP2712431A1 (en) 2011-05-12 2014-04-02 Apple Inc. Presence sensing
US20140196131A1 (en) 2013-01-07 2014-07-10 Salutron, Inc. User authentication based on a wrist vein pattern
JP6854612B2 (en) 2015-10-06 2021-04-07 三星電子株式会社Samsung Electronics Co.,Ltd. Biological information measuring device, biometric information measuring method, and computer-readable recording medium
US10135822B2 (en) 2017-03-21 2018-11-20 YouaretheID, LLC Biometric authentication of individuals utilizing characteristics of bone and blood vessel structures

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5903350A (en) * 1997-02-03 1999-05-11 Optiphase, Inc. Demodulator and method useful for multiplexed optical sensors
US20020147931A1 (en) * 2001-02-08 2002-10-10 Chu-Kung Liu Computer device for sensing user status and computer system for direct automatic networking
US20050062460A1 (en) * 2003-09-05 2005-03-24 Blake Jame N. Time division multiplexed optical measuring system
US20050071698A1 (en) * 2003-09-30 2005-03-31 Kangas Paul Daniel Apparatus, system, and method for autonomic power adjustment in an electronic device
US20080199049A1 (en) * 2007-02-21 2008-08-21 Daly Scott J Methods and Systems for Display Viewer Motion Compensation Based on User Image Data
US20110296163A1 (en) * 2009-02-20 2011-12-01 Koninklijke Philips Electronics N.V. System, method and apparatus for causing a device to enter an active mode
US20100328074A1 (en) * 2009-06-30 2010-12-30 Johnson Erik J Human presence detection techniques
US20110262044A1 (en) * 2010-04-21 2011-10-27 Elitegroup Computer Systems Co., Ltd. Energy saving method for electronic device
US20120212319A1 (en) * 2011-02-18 2012-08-23 Research In Motion Limited System and method for activating an electronic device using two or more sensors

Cited By (68)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9536362B2 (en) 2010-09-27 2017-01-03 Apple Inc. Polarized images for security
US8760517B2 (en) 2010-09-27 2014-06-24 Apple Inc. Polarized images for security
US10402624B2 (en) 2011-05-12 2019-09-03 Apple Inc. Presence sensing
US10372191B2 (en) 2011-05-12 2019-08-06 Apple Inc. Presence sensing
US20130027548A1 (en) * 2011-07-28 2013-01-31 Apple Inc. Depth perception device and system
US20130057573A1 (en) * 2011-09-02 2013-03-07 DigitalOptics Corporation Europe Limited Smart Display with Dynamic Face-Based User Preference Settings
US20140232843A1 (en) * 2011-10-27 2014-08-21 Robert Campbell Gain Value of Image Capture Component
US20130194172A1 (en) * 2012-01-30 2013-08-01 Cellco Partnership D/B/A Verizon Wireless Disabling automatic display shutoff function using face detection
US9245445B2 (en) * 2012-02-21 2016-01-26 Ricoh Co., Ltd. Optical target detection
US20130215218A1 (en) * 2012-02-21 2013-08-22 Sri Rama Prasanna Pavani Optical Target Detection
US8810719B2 (en) * 2012-02-23 2014-08-19 Pixart Imaging Inc. Electronic apparatus having camera sensing circuit and object sensing circuit used for switching between two different operation modes and related computer-readable medium
US20130222611A1 (en) * 2012-02-23 2013-08-29 Pixart Imaging Inc. Electronic apparatus having camera sensing circuit and object sensing circuit used for switching between two different operation modes and related computer-readable medium
US20130229508A1 (en) * 2012-03-01 2013-09-05 Qualcomm Incorporated Gesture Detection Based on Information from Multiple Types of Sensors
US9389690B2 (en) * 2012-03-01 2016-07-12 Qualcomm Incorporated Gesture detection based on information from multiple types of sensors
US10297208B2 (en) * 2012-03-05 2019-05-21 Pixart Imaging Inc. Automatic detection method for backlight control and portable electronic device using the same
US9491603B2 (en) * 2012-03-09 2016-11-08 Facebook, Inc. Substantially continuous location logging for geographic-positioning capable devices
US9246862B2 (en) * 2012-03-09 2016-01-26 Facebook, Inc. Dynamic duty-cycling of processor of mobile device based on operating condition of mobile device
US20130238700A1 (en) * 2012-03-09 2013-09-12 Matthew Nicholas Papakipos Dynamic Duty-Cycling of Processor of Mobile Device Based on Operating Condition of Mobile Device
US20150054984A1 (en) * 2012-03-15 2015-02-26 Nec Display Solutions, Ltd. Image display apparatus and image display method
US8787939B2 (en) 2012-03-27 2014-07-22 Facebook, Inc. Dynamic geographic beacons for geographic-positioning-capable devices
US9622179B2 (en) * 2012-03-27 2017-04-11 Facebook, Inc. Dynamic geographic beacons for geographic-positioning-capable devices
US20170070857A1 (en) * 2012-03-27 2017-03-09 Facebook, Inc. Dynamic Geographic Beacons for Geographic-Positioning-Capable Devices
US10009728B2 (en) * 2012-03-27 2018-06-26 Facebook, Inc. Dynamic geographic beacons for geographic-positioning-capable devices
US20150365901A1 (en) * 2012-03-27 2015-12-17 Facebook, Inc. Dynamic Geographic Beacons for Geographic-Positioning-Capable Devices
US8745368B2 (en) * 2012-04-11 2014-06-03 Quanta Computer Inc. Notebook computer for performing part of power-on self-test according to proximity sensor before displaying image after power switch is triggered
US20130275738A1 (en) * 2012-04-11 2013-10-17 Quanta Computer Inc. Notebook computer
US20130321312A1 (en) * 2012-05-29 2013-12-05 Haruomi HIGASHI Information processing apparatus, information display system and information display method
US9285906B2 (en) * 2012-05-29 2016-03-15 Ricoh Company, Limited Information processing apparatus, information display system and information display method
US9632179B2 (en) * 2012-07-04 2017-04-25 Blackberry Limited Estimating a baseline of a proximity sensor
US20140012528A1 (en) * 2012-07-04 2014-01-09 Research In Motion Limited Estimating a baseline of a proximity sensor
US9426236B2 (en) 2012-07-10 2016-08-23 Facebook, Inc. Determining a location and area of a place based on distances between the first mean and check in locations
US9245497B2 (en) 2012-11-01 2016-01-26 Google Technology Holdings LLC Systems and methods for configuring the display resolution of an electronic device based on distance and user presbyopia
US9626741B2 (en) 2012-11-01 2017-04-18 Google Technology Holdings LLC Systems and methods for configuring the display magnification of an electronic device based on distance and user presbyopia
US9428897B2 (en) 2012-12-17 2016-08-30 Fluidmaster, Inc. Touchless activation of a toilet
WO2014099817A1 (en) * 2012-12-17 2014-06-26 Fluidmaster, Inc. Touchless activation of a toilet
CN104122974A (en) * 2013-04-24 2014-10-29 华硕电脑股份有限公司 Mobile communication device with human body recognition function
WO2014178836A1 (en) 2013-04-30 2014-11-06 Hewlett-Packard Development Company, L.P. Depth sensors
US20160088206A1 (en) * 2013-04-30 2016-03-24 Hewlett-Packard Development Company, L.P. Depth sensors
EP2992403B1 (en) * 2013-04-30 2021-12-22 Hewlett-Packard Development Company, L.P. Depth sensors
CN104281377A (en) * 2013-07-04 2015-01-14 联想(北京)有限公司 Electronic equipment and method and device for controlling screen of electronic equipment
WO2015017931A1 (en) 2013-08-07 2015-02-12 Blackberry Limited Determining the distance of an object to an electronic device
EP3030924A4 (en) * 2013-08-07 2016-07-13 Blackberry Ltd Determining the distance of an object to an electronic device
CN103631359A (en) * 2013-11-15 2014-03-12 联想(北京)有限公司 Information processing method and electronic equipment
WO2015138095A1 (en) * 2014-03-11 2015-09-17 Google Technology Holdings LLC Display viewing detection
CN106462183A (en) * 2014-03-11 2017-02-22 谷歌技术控股有限责任公司 Display viewing detection
US10119864B2 (en) 2014-03-11 2018-11-06 Google Technology Holdings LLC Display viewing detection
US10255417B2 (en) 2014-05-13 2019-04-09 Google Technology Holdings LLC Electronic device with method for controlling access to same
US20150332032A1 (en) * 2014-05-13 2015-11-19 Google Technology Holdings LLC Electronic Device with Method for Controlling Access to Same
US9710629B2 (en) * 2014-05-13 2017-07-18 Google Technology Holdings LLC Electronic device with method for controlling access to same
WO2015183924A1 (en) * 2014-05-29 2015-12-03 Google Technology Holdings LLC Electronic device and method for controlling access to same
CN106445065A (en) * 2015-08-11 2017-02-22 三星电子株式会社 Method for controlling according to state and electronic device thereof
US9678559B1 (en) * 2015-09-18 2017-06-13 Amazon Technologies, Inc. Determining a device state based on user presence detection
US10452116B1 (en) * 2015-09-18 2019-10-22 Amazon Technologies, Inc. Determining a device state based on user presence detection
WO2017083178A1 (en) * 2015-11-12 2017-05-18 Microsoft Technology Licensing, Llc Adaptive user presence awareness for smart devices
US10482331B2 (en) * 2015-11-20 2019-11-19 GM Global Technology Operations LLC Stixel estimation methods and systems
US20170147888A1 (en) * 2015-11-20 2017-05-25 GM Global Technology Operations LLC Stixel estimation methods and systems
CN110192193A (en) * 2017-01-19 2019-08-30 惠普发展公司有限责任合伙企业 Secret protection equipment
US10817594B2 (en) 2017-09-28 2020-10-27 Apple Inc. Wearable electronic device having a light field camera usable to perform bioauthentication from a dorsal side of a forearm near a wrist
US11036844B2 (en) 2017-09-28 2021-06-15 Apple Inc. Wearable electronic device having a light field camera
US10484530B2 (en) * 2017-11-07 2019-11-19 Google Llc Sensor based component activation
US20190141181A1 (en) * 2017-11-07 2019-05-09 Google Llc Sensor Based Component Activation
USD886245S1 (en) 2018-04-26 2020-06-02 Bradley Fixtures Corporation Dispenser
USD886240S1 (en) 2018-04-26 2020-06-02 Bradley Fixtures Corporation Faucet and soap dispenser set
USD954226S1 (en) 2018-04-26 2022-06-07 Bradley Fixtures Corporation Faucet and soap dispenser set
USD964522S1 (en) 2018-04-26 2022-09-20 Bradley Fixtures Corporation Dispenser
US10785419B2 (en) * 2019-01-25 2020-09-22 Pixart Imaging Inc. Light sensor chip, image processing device and operating method thereof
US20200244861A1 (en) * 2019-01-25 2020-07-30 Pixart Imaging Inc. Light sensor chip, image processing device and operating method thereof
US10923045B1 (en) * 2019-11-26 2021-02-16 Himax Technologies Limited Backlight control device and method

Also Published As

Publication number Publication date
US20170193282A1 (en) 2017-07-06
US10402624B2 (en) 2019-09-03

Similar Documents

Publication Publication Date Title
US10402624B2 (en) Presence sensing
US10372191B2 (en) Presence sensing
US9658688B2 (en) Automatic view adjustment
US11561621B2 (en) Multi media computing or entertainment system for responding to user presence and activity
US9704250B1 (en) Image optimization techniques using depth planes
US7542592B2 (en) Systems and methods for face detection and recognition using infrared imaging
US9729865B1 (en) Object detection and tracking
JP5567853B2 (en) Image recognition apparatus and method
US9131150B1 (en) Automatic exposure control and illumination for head tracking
US10027883B1 (en) Primary user selection for head tracking
US20150187076A1 (en) System and Methods for Persona Identification Using Combined Probability Maps
JP2008146539A (en) Face authentication device
CN104182721A (en) Image processing system and image processing method capable of improving face identification rate
US20130235227A1 (en) Image capturing device and method thereof and human recognition photograph system
JP6157165B2 (en) Gaze detection device and imaging device
WO2010042068A1 (en) Method and system for object detection and tracking
JP6798609B2 (en) Video analysis device, video analysis method and program
CN103961108A (en) Tumbling detecting method
US10268922B2 (en) Image processing by means of cross-correlation
US11381743B1 (en) Region of interest capture for electronic devices
Kitajima et al. Privacy‐aware face detection using biological signals in camera images
Wang et al. Shape-based pedestrian/bicyclist detection via onboard stereo vision
Tachos et al. Appearance and Depth for Rapid Human Activity Recognition in Real Applications.
Chiperi et al. Human tracking using multiple views
Ju et al. Hybrid AdaBoost for Multiuser Priority Analysis

Legal Events

Date Code Title Description
AS Assignment

Owner name: APPLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VALKO, EDWARD ALLEN;WALDON, MATTHEW C.;MERWE, RUDOLPH VAN DER;AND OTHERS;SIGNING DATES FROM 20110825 TO 20110826;REEL/FRAME:026865/0689

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION