Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20140325459 A1
Publication typeApplication
Application numberUS 14/324,476
Publication date30 Oct 2014
Filing date7 Jul 2014
Priority date6 Feb 2004
Also published asCA2554917A1, CA2554917C, EP1711886A1, US8819596, US20050210419, WO2005076114A1
Publication number14324476, 324476, US 2014/0325459 A1, US 2014/325459 A1, US 20140325459 A1, US 20140325459A1, US 2014325459 A1, US 2014325459A1, US-A1-20140325459, US-A1-2014325459, US2014/0325459A1, US2014/325459A1, US20140325459 A1, US20140325459A1, US2014325459 A1, US2014325459A1
InventorsJuha Kela, Panu Korpipaa, Jani Mantyjarvi, Heikki Keranen, Tapani Rantakokko, Esko-Juhani Malm, Sanna Kallio, Jussi Holopainen, Jari Kangas, Samuli Silanto
Original AssigneeNokia Corporation
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Gesture control system
US 20140325459 A1
Abstract
A control system is provided based on the use of gestures and functioning especially in mobile terminals. The gesture control system is provided with a general purpose interface with its commands for applications to be controlled. The processing software of the gesture signals includes a training program trained free-form gestures made by the user being stored in the gesture library, and a recognizing program, which matches a gesture made by the user to the stored gestures and chooses the most similar gesture thereof. Gestures can hence be used as commands for controlling any application configured or programmed to receive the command. One and the same application functions in different models of mobile terminals without matching, and in a certain mobile terminal all applications can be run, which applications use specified interface commands. The application can be e.g. a game or activity being included in basic implementation of a mobile terminal.
Images(9)
Previous page
Next page
Claims(19)
1.-20. (canceled)
21. A method for controlling a game, the method comprising:
receiving an indication of a hand gesture from a first user;
calculating an associated skill level associated with the hand gesture; and
causing provision of information associated with the associated skill level to a second user.
22. The method of claim 21, further comprising:
in response to receiving the indication of the hand gesture from the first user, causing performance of a spell according to the game on the second user.
23. The method of claim 21, wherein the calculated associated skill level is based on an accuracy of the hand gesture relative to a stored hand gesture of a gesture library.
24. The method of claim 21, further comprising:
receiving an indication from the first user identifying the second user as a user to which the hand gesture is directed.
25. The method of claim 21, further comprising:
receiving an indication of an attempted repetition of the hand gesture by the second user; and
calculating an accuracy score of the attempted repetition of the hand gesture.
26. The method of claim 21, further comprising:
in response to receiving the indication of the hand gesture from the first user, causing the game application to displace at least one other application in use by the second user.
27. An apparatus comprising at least one processor and at least one memory including computer program code for controlling a game, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to perform at least:
receiving an indication of a hand gesture from a first user;
calculating an associated skill level associated with the hand gesture; and
causing provision of information associated with the associated skill level to a second user.
28. The apparatus of claim 27, wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to perform at least:
in response to receiving the indication of the hand gesture from the first user, causing performance of a spell according to the game on the second user.
29. The apparatus of claim 27, wherein the calculated associated skill level is based on an accuracy of the hand gesture relative to a stored hand gesture of a gesture library.
30. The apparatus of claim 27, wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to perform at least:
receiving an indication from the first user identifying the second user as a user to which the hand gesture is directed.
31. The apparatus of claim 27, wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to perform at least:
receiving an indication of an attempted repetition of the hand gesture by the second user; and
calculating an accuracy score of the attempted repetition of the hand gesture.
32. The apparatus of claim 27, wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to perform at least:
in response to receiving the indication of the hand gesture from the first user, causing the game application to displace at least one other application in use by the second user.
33. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions for:
receiving an indication of a hand gesture from a first user;
calculating an associated skill level associated with the hand gesture; and
causing provision of information associated with the associated skill level to a second user.
34. The computer program product of claim 33, wherein the computer-executable program code instructions further comprise program code instructions for:
in response to receiving the indication of the hand gesture from the first user, causing performance of a spell according to the game on the second user.
35. The computer program product of claim 33, wherein the calculated associated skill level is based on an accuracy of the hand gesture relative to a stored hand gesture of a gesture library.
36. The computer program product of claim 33, wherein the computer-executable program code instructions further comprise program code instructions for:
receiving an indication from the first user identifying the second user as a user to which the hand gesture is directed.
37. The computer program product of claim 33, wherein the computer-executable program code instructions further comprise program code instructions for:
receiving an indication of an attempted repetition of the hand gesture by the second user; and
calculating an accuracy score of the attempted repetition of the hand gesture.
38. The computer program product of claim 33, wherein the computer-executable program code instructions further comprise program code instructions for:
in response to receiving the indication of the hand gesture from the first user, causing the game application to displace at least one other application in use by the second user.
Description
    CROSS-REFERENCE TO RELATED APPLICATION
  • [0001]
    This application is a divisional application of and claims priority to U.S. application Ser. No. 11/049,638, filed Feb. 1, 2005, which claims priority to Finnish Patent Application Number FI 20040184 filed on Feb. 6, 2004, the entire contents of which are hereby incorporated by reference.
  • TECHNICAL FIELD
  • [0002]
    The invention relates to a control system basing on the use of gestures and functioning especially in mobile terminals. The invention also relates to a mobile terminal comprising the software of a gesture control system.
  • BACKGROUND
  • [0003]
    The gesture control system means a system, by means of which the managing of an application, observable with senses, takes place at least partly by hand motions. The control system comprises motion sensors, which move along with the hand of a person using the application, and the converters and processing programs for the signals generated by the motion sensors. The hand motions, or gestures, then are recognized on grounds of e.g., accelerations occurring in the motions. The application controlled by gestures can be 20 for example a game loaded into a mobile terminal or the controlling program of an external electromechanical device.
  • [0004]
    The “application” means in this description and the claims both an observable process and a program, which directly realizes said process.
  • [0005]
    Recognizing a motion by equipment, as such, is known from before. Recognition systems applying acceleration sensors are disclosed among other documents in the articles “Recognizing Human Motion with Multiple Acceleration Sensors” (Mäntyjärvi & kumpp., IEEE International Conference on Systems, Man and Cybernetics, 2001) and “Recognizing Movements of a Portable Handheld Device Using Symbolic Representation and Coding of Sensor Signals” (Flanagan et al., International Conference on Mobile and Ubiquitous Multimedia, 2002) and in the publication WO 03/001340. The system in accordance with the last-mentioned publication includes also a gesture library and a program analyzing acceleration data and defining, if that data corresponds to a certain three-dimensional gesture.
  • [0006]
    Also the controlling of an application by hand motions is known from before. For example the publication WO 00/63874 discloses a system, in which an application changes the shape of a pattern seen in the computer display and moves the pattern, depending on how a control device, to be held in the hand, is handled. The control device comprises acceleration sensors for three dimensions and a button with a pressure sensor.
  • [0007]
    FIG. 1 presents a simplified diagram showing the interfacing an application to its control system, according to the prior art. The whole system comprises a hardware portion HW and a software portion SW. Regarding the hardware portion there is drawn in FIG. 1 a sensor unit SU, an interface IF of the sensor unit, a memory MEM and a computer bus. The software portion comprises a driver 140 for the interface IF, a processing software 130 for the signals which correspond to the hand motions, an application 110 controllable by the hand motions and an operation system OS of the computer at issue. The driver 140 stores the gesture signals, converted to digital form by the interface IF, to the memory MEM. The signal processing software then analyzes the gesture signals and provides a control to the application. Naturally, the signal processing software has to be matched to the application for the data transfer between them.
  • [0008]
    A flaw in the control systems like the above-mentioned systems is the limited scope of their use: Changing an application to another application presumes amending work in the software of the control system. Likewise transferring the system to another computer of other type requires matching work in the programs.
  • SUMMARY
  • [0009]
    Objects of the invention are to reduce said disadvantages related to the prior 30 art and to extend implementing environment of the gesture control systems to the mobile terminals. A system according to the invention is a gesture control system comprising a sensor unit with motion sensors to be held in a user's hand, a processing software of sensor data and an interface program between said processing software and a controllable application, the processing software comprising a trainer to make gesture models and a recognizer to recognize gestures during use of the application, said interface program forming a general purpose interface to get specified commands from the application and to provide specified responses to the application. A mobile terminal according to the invention comprising a control system to manage an application operating in the mobile terminal by gestures, the control system having a processing software of data gotten from motion sensors and an interface program between said processing software and said application is characterized in that which is specified in the independent claim. Some preferred embodiments of the invention are specified in the other claims.
  • [0010]
    The basic idea of the invention is as follows: The gesture control system is provided with a general purpose interface with its commands for applications to be controlled. The processing software of the gesture signals includes a training program, and the trained free form gestures made by the user are stored in the gesture library. The processing software of the gesture signals also includes a recognizing program, which matches a gesture made by the user to the stored gestures and chooses the most similar gesture thereof. Gestures can hence be used as commands for controlling any application configured or programmed to receive the command. The system is preferably implemented in a mobile terminal.
  • [0011]
    An advantage of the invention is that one and the same application functions in different models of mobile terminals without matching relating to the models. This is due to the above-mentioned general purpose interface. Likewise in a certain mobile terminal can be run all applications, which use specified interface commands. Another advantage of the invention is that in a system according to it new and different gestures can easily be formed and put into use. A further advantage of the invention is that it makes possible the controlling by gestures of all possible activities being included in the basic implementation of a mobile terminal. A further advantage of the invention is that it remarkably expands possibilities to interact with game applications in the mobile terminals.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0012]
    The invention will now be described in detail. Reference will be made to the accompanying drawings wherein
  • [0013]
    FIG. 1 presents as a layer diagram the interfacing an application to its control system, according to the prior art,
  • [0014]
    FIG. 2 presents as a layer diagram the interfacing an application to its control system, according to the invention,
  • [0015]
    FIG. 3 presents the general principle of the function of a system according to the invention,
  • [0016]
    FIG. 4 presents as a flow diagram an example of a gesture training procedure in a system according to the invention,
  • [0017]
    FIG. 5 presents as a flow diagram an example of a gesture recognizing procedure in a system according to the invention,
  • [0018]
    FIG. 6 presents as a sequence diagram an example of a gesture training procedure in a system according to the invention,
  • [0019]
    FIG. 7 presents as a sequence diagram an example of a gesture recognizing procedure utilizing a button, in a system according to the invention,
  • [0020]
    FIG. 8 presents as a sequence diagram an example of a gesture recognizing procedure without a button, in a system according to the invention, and
  • [0021]
    FIGS. 9 a, b present examples of a mobile terminal according to the invention.
  • DETAILED DESCRIPTION
  • [0022]
    FIG. 1 was already discussed in connection with the description of the prior art.
  • [0023]
    FIG. 2 shows a diagram, corresponding to FIG. 1, about the interfacing an application to its control system, according to the invention. The whole system comprises a hardware portion HW and a software portion SW. Regarding the hardware portion there is drawn in the figure, also in this case, a sensor unit SU reacting to the motion, a circuit-based interface IF of the sensor unit, a memory MEM and a computer bus. In addition, there is seen an audio part for possible use of voice messages to the user and/or from the user. The software portion comprises a driver 250 for the interface IF, a sensor interface 240, a processing software 230 for the signals which correspond to the hand motions, an interface 220 between said processing software and an application 210 to be controlled by the hand motions, the actual application 210 and an operation system OS of the device at issue. An optional part in the software SW is audio interface 270 for above-mentioned voice messages. The operation system can be Symbian or Linux, for instance. The processing software for the signals which correspond to the hand motions is more briefly named “Gesture Engine” and different interfaces, or matching programs are named by an abbreviation API (Application Programming Interface). So the driver for the circuit-based interface IF is IF-API, the sensor interface is S-API and the interface between the application and said Gesture Engine is G-API. IF-API together with S-API stores the motion signals, converted to digital form, to the memory MEM. S-API then informs by messages to the Gesture Engine the data at issue, or the sensor data.
  • [0024]
    The sensor unit SU can be included in a host device or it can be external. In the latter case the transfer system between the sensor unit and interface IF can be based e.g., on Bluetooth or infrared technology, or can be wired. The motion sensors proper can be acceleration sensors or gyroscopes reacting to angular velocity or angular acceleration. They can also be magnetometers, or electronic compasses. In one and the same sensor unit there can occur more than one sensor type. When using acceleration sensors, the number of these is at least two, but preferably three to measure acceleration in each of three dimensions. The sensor unit further can include a button to inform the gesture engine the start and completion of a gesture.
  • [0025]
    The interface G-API between the application and the Gesture Engine is, as mentioned, general purpose, or standard-like. The application has an inter-5 face directly to the sensor interface S-API, too. That interface can be used for application control directly, when an actual signal processing is not needed. The interface G-API, the Gesture Engine and the sensor interface S-API constitute a general purpose platform, on which different applications can be connected. Similarly a certain application can without difficulty be installed to dif10 ferent devices, which have the platform in question.
  • [0026]
    FIG. 3 shows the general principle of the function of a system according to the invention. As functional parts there are presented an application 310, the interface thereof 320, Gesture Engine 330 and a sensor interface, or S-API 340. In the Gesture Engine there are seen its main parts Gesture trainer and Gesture recognizer. Both these parts utilize processing modules Preprocessor and Gesture capturer. Certain operation phase starts from a command given by the application to the Gesture Engine. The Gesture Engine gives to the application a response required by the command. Giving a response often presumes information about motions of the motion sensors. For this reason the Gesture Engine gives to S-API a request for sensor data, and S-API answers by informing the data when it can be read in the memory. In FIG. 3 also is seen the sensor unit SU in the hand of a user and an example GES of a hand motion, which causes said data.
  • [0027]
    The set of the specified commands and responses is as follows, for example. In the table the response corresponding a certain command is to the right from that command.
  • [0000]
    CollectTrainingGesture TrainingGestureCollected
    StartGestureTraining GestureTrained
    GestureNotTrained
    StartGestureRecognition GestureRecognized
    GestureNotRecognized
    DetectStill DeviceStill
    EndGesture GestureRecognized
    GetGestureNames GestureNameList
    AbortGesture
    SubscribeGesture
    UnsubscribeGesture
    DeleteGesture
    RenameGesture
  • [0028]
    The meaning of most important commands and responses appears more detailed in the descriptions of FIGS. 6-8.
  • [0029]
    FIG. 4 presents as a flow diagram an example of a gesture training procedure in a system according to the invention. The procedure as such is known from before. It is based on HMM-algorithm (Hidden Markov Model), which is applied on recognizing e.g., speech and graphological text. In step 401 a person carrying out the training has done a free form hand motion, and the sensor interface S-API provides the sensor data generated by the motion to the Gesture Engine. The data consists of samples of the analog sensor signals, the samples converted to a digital form. There are three sample sequences, each of which corresponds to a motion in one dimension. In step 402 the data is normalized. That means increasing the number of samples by interpolation or decreasing the number of samples by decimation, to achieve the sample sequences of a certain length. In step 403 the data is quantized. This happens by means of a codebook CB containing fixed three-dimensional points, which characterize the space of the gestures. Using such a codebook alleviates the computing load. The result is a one-dimensional token sequence, suitable for the HMM. In step 404 it is checked, whether the gesture being under training has been performed so many times as enough. If not, the above-described steps 401-403 are repeated. If yes, initial parameters are defined for the gesture model on grounds of accumulated, quantized data (step 405). In step 406 a model of the gesture at issue is formed by means of an expedient method such as the Baum-Welch algorithm, in accordance with the HMM. The Baum-Welch algorithm calculates, using probability theory, values for three parameters, which represent the gesture. Each parameter is a matrix containing then plurality of numbers. The obtained parameter values 407 are the result of the training, and they are stored to the gesture library LIB. The Baum-Welch algorithm is iterative by nature. The iteration converges the better the smaller the variation in the source data. However that variation has certain optimum value. If it is clearly lower than the optimum value, the recognition program discards too readily gestures made by the user, and if the variation is clearly higher than the optimum value, the result of the later recognitions is unreliable.
  • [0030]
    The result of the training can naturally be tested by trying the gesture recognition directly after the training, according to a procedure described below. An estimate for the reliability is obtained by doing several times gestures, which are similar as in the training phase and “something like that”. A speech synthesizer, if being included in the device, can be connected to that kind of testing so that the device tells by clear speech the name of the gesture, which it has recognized.
  • [0031]
    The codebook CB has been created in a blanket way, by means of a separate specific algorithm, which uses among other things gesture data collected from a number of people. The resulting codebook is serviceable to use for various persons without changes. “Generic code book” means here and in patent claims such a codebook. A generic codebook contributes to that also the trained gestures are user-independent. In addition, if a single trained gesture is transferred from a device to another device, it is immediately serviceable in the new environment. Such gestures can be arranged to be loaded through the Internet, and they can be subjects to a charge.
  • [0032]
    FIG. 5 presents as a flow diagram an example of a gesture recognizing procedure in a system according to the invention. Also this procedure is based on HMM-algorithm and is known from before as such. In step 501 a person using the system has done a hand motion, and the sensor interface S-API provides the sensor data generated by the motion to the Gesture Engine. Also now the data consists for example of three sample sequences, each of which corresponds to a motion in one dimension. In steps 502 and 503 happen the data normalization and quantization as in steps 402 and 403 of FIG. 4. In step 504 the number I of gestures stored in the gesture library is loaded in a counter. In step 505 a reference figure is calculated, which represents likelihood of the gesture being under recognition with the stored gesture pointed by the counter value. The calculation employs the parameter values defined for the stored gesture at issue, and it is run by means of an expedient method such as the Viterbi algorithm, in accordance with the HMM. Afterwards the counter content is decremented by one (step 506). In following step 507 it is checked, whether the counter value is zero. If not yet, the calculation according to step 505 is repeated, regarding now a new stored gesture. If yes, or all stored gestures are gone through, obtained reference figures are evaluated, step 508. In the evaluation the reference figures are compared with each other and with a certain threshold value. In step 509 a decision is made. If some reference figure is found to be distinctly highest, and it further is higher than said threshold value, the gesture is considered to be recognized. The chosen gesture is the stored gesture corresponding to the reference figure in question. If no one of the reference figures is distinctly highest, or all of them are lower than said threshold value, the gesture is not considered to be recognized.
  • [0033]
    FIG. 6 presents as a sequence diagram an example of a gesture training procedure in a system according to the invention. In the diagram the vertical dashed lines refer to functional parts of the software. These are, in the order from left to right, an application, “G” which means here the G-API and the upper level of the Gesture Engine together, the Preprocessor, the Gesture Capturer and the S-API. In the diagram the time proceeds downwards. A vertical beam on a dashed line means a period, when the part of the software at issue is active. The example relates to a system, in which the motion sensor unit is equipped with a button. The state information of the button is brought through the S-API to the Gesture Engine and further to the application. The diagram starts by command CollectTrainingGesture provided by the application to the interface G-API. That command has been preceded by the starting of the training procedure in the application and the pressing the button B. The command contains as parameters the name chosen for the gesture to be trained, the button being used and the gesture timeout, which means the maximum time allowed for the collecting of training data. Said command causes command StartGestureCapture to the Gesture Capturer inside the Gesture Engine, and this command in turn command SetSensorListener to the interface S-API. The last-mentioned command contains among other things information at which rate the Gesture Engine will get data packets. Afterwards the S-API provides to the Gesture Capturer the samples of the sensor signals in data packets, in the header of which the size of the packet at a given time is mentioned. That transfer continues, until the button is released. In this case the application provides to the G-API command EndGesture, which results in command StopGestureCapture to the Gesture Capturer inside the Gesture Engine and command SetSensorListener to the S-API. The parameters included in the latter command cause the sampling to stop. The Gesture Capturer gives to its internal module command FindActualGesture, which module cleans the captured raw data deleting, according to certain criteria, from it the parts, which probably do not belong to the actual gesture. That kind of irrelevant data parts come from the small and unintended hand motions before and after the actual gesture. When the irrelevant data is deleted, the Gesture Capturer provides to the upper level of the Gesture Engine response GestureCaptured, by which the actual gesture data is informed. The Gesture Engine then provides to the Preprocessor command Normalize (actual gesture data), which starts the processing of the actual gesture data. The Preprocessor carries out the data normalization and quantization. The latter is started by command ExtractFeatures given to an internal module of the Preprocessor. The Preprocessor then provides to the upper level of the Gesture Engine notice FeaturesExtracted. The data yielded by deleting irrelevant data, normalizing and quantizing is here and in claims called “feature data” in distinction from said raw data. Subsequently the Gesture Engine provides, by the G-API, to the application response TrainingGestureCollected. As a parameter of this response it is mentioned, which repetition turn of the gesture to be trained is in question.
  • [0034]
    The above-disclosed operation sequence is repeated, when the gesture to be trained is repeated. The number of repetition turns is for instance 2-5. When the repetition is stopped, the application provides to the interface G-API command StartGestureTraining. Parameter of this command is the name of the gesture. The training procedure continues by command Train, given to an internal module of the Gesture Engine. That module carries out the training algorithm, depicted in context of FIG. 4. Finally, the Gesture Engine provides, by the G-API, to the application response Gesture Trained. Parameters of this response are the name of the gesture and calculated values of the gesture parameters.
  • [0035]
    FIG. 7 presents as a sequence diagram an example of a gesture recognizing procedure in a system according to the invention. This example, too, relates to a system, in which the motion sensor unit is equipped with a button, the state information of which is brought through the S-API to the Gesture Engine and further to the application. The diagram starts with command StartGestureRecognition provided by the application to the interface G-API. That command has been preceded by the starting of the recognizing procedure in the application and the pressing of the button B. The command contains as parameters the button being used and the gesture timeout, which means the maximum time allowed for the recognition. Said command causes inside the Gesture Engine command StartGestureCapture to the Gesture Capturer, and this command in turn command SetSensorListener to the interface S-API, as in the training procedure of FIG. 6. Also from that point onwards the operation is equal to FIG. 6 until the Preprocessor provides to the upper level of the Gesture Engine notice FeaturesExtracted. The recognizing procedure then continues so that the Gesture Engine gives to its internal module command Recognize. That module carries out the calculation and evaluation of reference figures, which represent likelihood of the gesture being under recognition with the stored gestures, and makes a decision, as depicted in context of FIG. 5. Subsequently the Gesture Engine provides, by the G-API, to the application response GestureRecognized provided that a gesture indeed has been chosen. As a parameter of that response is the name of the recognized gesture.
  • [0036]
    FIG. 8 presents as a sequence diagram another example of a gesture recognizing procedure in a system according to the invention. This example relates to a system, in which the motion sensor unit is not equipped with a button or it is at least not used. The diagram starts by command DetectStill, provided by the application to the interface G-API. It is in question a still state of the sensor unit. Because the button is absent, said state has to be separately detected, before storing of meaningful gesture signal can be started. Parameters of that command are the still state duration required and the timeout, which means the maximum time allowed for the still state detection. Said command causes inside the Gesture Engine command DetectStill to the Gesture Capturer, and this command in turn command SetSensorListener to the interface S-API. Afterwards the S-API provides to the Gesture Capturer the samples of the sensor signals in data packets. This transfer continues, until an internal module of the Gesture Capturer observes that the sensor unit has been for a certain period in the still state and gives a notice DeviceStillDetected. The Gesture Capturer provides to the Gesture Engine response DeviceStill and to the S-API command SetSensorListener with a parameter causing the sampling to stop. The Gesture Engine in turn provides by the G-API response DeviceStill to the application.
  • [0037]
    The procedure continues by command StartGestureRecognition provided by the application to the interface G-API. Command parameters are the matter that no button is used and the gesture timeout, which means the maximum time allowed for the gesture performing. That command causes inside the Gesture Engine command StartGestureCapture to the Gesture Capturer. Based on this command the Gesture Capturer in this case starts the still state detection function and provides to the S-API command SetSensorListener. After that the S-API again provides to the Gesture Capturer the samples of the sensor signals in data packets, which transfer continues, until an internal module of the Gesture Capturer observes that the sensor unit has been a certain period in the still state. The Gesture Capturer then provides to the S-API command SetSensorListener with a parameter causing the sampling to stop and gives to its internal module command FindActualGesture, which module cleans the captured raw data deleting, according to certain criteria, from it the parts, which probably do not belong to the actual gesture. When the irrelevant data is deleted, the Gesture Capturer provides to the upper level of the Gesture Engine response GestureCaptured, by which the actual gesture data is informed. The Gesture Engine then provides to the Preprocessor command Normalize (actual gesture data), which starts the processing of the actual gesture data. The Preprocessor carries out the data normalization and quantization, and provides to the upper level of the Gesture Engine notice FeaturesExtracted, as in the procedures of FIGS. 6 and 7. The recognizing procedure continues from that point as in FIG. 7, when the Gesture Engine carries out the calculation and evaluation of reference figures, which represent likelihood of the gesture being under recognition with the stored gestures, and makes a gesture decision. Finally the Gesture Engine also in this case provides, by the G-API, to the application response GestureRecognized provided that a gesture indeed has been chosen. As a parameter of that response is the name of the recognized gesture.
  • [0038]
    Also the training procedure can be implemented without a button. In that case the difference with the sequence of FIG. 6 corresponds to the difference between sequences of FIGS. 8 and 7: When the situation presumes, it is waited until the sensor unit is in the still state. The recognizing procedure can also be extended to concern a series of successive gestures. In that case the recognition of the first gesture happens as in FIG. 8. The recognition of the next gesture starts either immediately without any command from the application or on grounds of a command and after a still state again has been detected.
  • [0039]
    FIG. 9 a presents an example of a mobile terminal according to the invention. The mobile terminal MSA has one application or more, controllable by means of gestures. An application can be e.g., an activity being included in the basic implementation of the mobile terminal, such as changing to a certain menu, sending a text message or some camera application. An application also can be a game being originally located in the mobile terminal or loaded later to it. For that kind of applications the mobile terminal is provided with processing software of the gesture signals, which software comprises a trainer for modeling gestures and a recognizer for recognizing gestures. The interface program between the processing software and an application controllable by gestures forms a general purpose interface to get specified commands from the application and to provide specified responses to the application. The sensor unit SUA is placed inside the cover of the mobile terminal. That matter limits the usable applications to be of the type, that do not require watching the display during the gesture control.
  • [0040]
    FIG. 9 b presents another example of a mobile terminal according to the invention. The mobile terminal MSB is from the view point of invention similar to the mobile terminal MSA in FIG. 9 a. The difference is that the sensor unit SUB belonging to the system now is an external device. The data transfer between the sensor unit and the mobile terminal can be based for example on Blue-tooth or infrared technology, or can be wired.
  • [0041]
    As mentioned above, the games form their own group among the applications to be controlled by gestures. A commercial game application can contain the parameter values of the gestures to be used in the game, which values are stored in the gesture library of the Gesture Engine when the game is started. The game may be a spell game by nature, whereupon the hand motions have apparent magical effect on the game events. The gestures can be different by skill level. The tougher gesture a player succeeds to make in a certain situation, the more powerful effect that gesture has. The grading may regard also an individual gesture. Although a player does not succeed to make it well, the Gesture Engine calculates a reference figure and provides it to the application, on condition that the gesture is distinguished from other gestures. The effect on the game is naturally the smaller, the smaller the reference figure is. If a score is calculated for the player, points are granted the more, the tougher gestures he has performed, and the more accurately he has performed those gestures. The game can be played by one or more people. In the latter case the implementation can be a network game such that e.g., two players have the terminals of their own, and the gesture information is transferred to the opponents terminal for instance by using the GPRS (General Packet Radio Service). A player makes e.g., an attack by means of a certain gesture, the opponent sees the attack on his own terminal and defends by trying to make a sufficiently effective counter gesture or a sequence of weaker gestures in a defined time.
  • [0042]
    A game application can also be executed as a background program so that a player makes something else at the same time. The player e.g., has joined a network game of numbers of players. When an action has been directed at his figure, the game application in the terminal of the player in question displaces other applications and shows what is happening.
  • [0043]
    Two persons can also play with a single device. In that case there has to be also two sensor units, and each sensor data packet contains information, a data of which player is in question.
  • [0044]
    A game may also be programmed so that it allows to train and bring new gestures into use. For instance the device is trained in a relatively complicated motion sequence, and the game would simply be such that the players try to repeat the motion sequence, each in turn, and the application gives points. In the description of FIG. 4 it was already explained that user-independent trained gesture models can be transferred from a device to another device through the Internet. For example a network operator can offer a game service such that people having a terminal according to the invention can load trained gesture models therefrom.
  • [0045]
    A system according to the invention is described above. The implementation of the invention in its different points can naturally deviate from what is presented. The commands and responses of the interface G-API according to the invention can be specified in a different way. Working up the sensor data to a form suitable for the HMM can be implemented also by some other process than by data normalization and quantization subsequent thereto, occurring in FIGS. 4 and 5. For both training and recognizing can be used, instead of the whole HMM, some other solution, such as neural network or Bayesian network technology. The starting and completing of a gesture can be informed, instead of a button, e.g., by voice control in the case of devices, which have speech recognition activity. Voice messages can also be used in contrary direction so that an application informs via Gesture Engine the user of certain situations by means of the device's audio part and speaker. For that purpose there is also a command of its own.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4288078 *20 Nov 19798 Sep 1981Lugo Julio IGame apparatus
US4702475 *25 Jul 198627 Oct 1987Innovating Training Products, Inc.Sports technique and reaction training system
US4843568 *11 Apr 198627 Jun 1989Krueger Myron WReal time perception of and response to the actions of an unencumbered participant/user
US5184295 *16 Oct 19892 Feb 1993Mann Ralph VSystem and method for teaching physical skills
US5229756 *14 May 199220 Jul 1993Yamaha CorporationImage control apparatus
US5239464 *9 Dec 199124 Aug 1993Blair Preston EInteractive video system providing repeated switching of multiple tracks of actions sequences
US5252951 *21 Oct 199112 Oct 1993International Business Machines CorporationGraphical user interface with gesture recognition in a multiapplication environment
US5288078 *16 Jul 199222 Feb 1994David G. CapperControl interface apparatus
US5347306 *17 Dec 199313 Sep 1994Mitsubishi Electric Research Laboratories, Inc.Animated electronic meeting place
US5393073 *25 May 199328 Feb 1995Best; Robert M.Talking video games
US5405152 *8 Jun 199311 Apr 1995The Walt Disney CompanyMethod and apparatus for an interactive video game with physical feedback
US5423554 *24 Sep 199313 Jun 1995Metamedia Ventures, Inc.Virtual reality game method and apparatus
US5454043 *30 Jul 199326 Sep 1995Mitsubishi Electric Research Laboratories, Inc.Dynamic and static hand gesture recognition through low-level image analysis
US5563988 *1 Aug 19948 Oct 1996Massachusetts Institute Of TechnologyMethod and system for facilitating wireless, full-body, real-time user interaction with a digitally represented visual environment
US5577981 *4 Aug 199526 Nov 1996Jarvik; RobertVirtual reality exercise machine and computer controlled video system
US5594469 *21 Feb 199514 Jan 1997Mitsubishi Electric Information Technology Center America Inc.Hand gesture machine control system
US5687254 *6 Jun 199411 Nov 1997Xerox CorporationSearching and Matching unrecognized handwriting
US5784504 *24 Jan 199421 Jul 1998International Business Machines CorporationDisambiguating input strokes of a stylus-based input devices for gesture or character recognition
US5795228 *3 Jul 199618 Aug 1998Ridefilm CorporationInteractive computer-based entertainment system
US5947742 *13 Nov 19977 Sep 1999Midori KatayamaMethod for teaching body motions
US6002808 *26 Jul 199614 Dec 1999Mitsubishi Electric Information Technology Center America, Inc.Hand gesture control system
US6072494 *15 Oct 19976 Jun 2000Electric Planet, Inc.Method and apparatus for real-time gesture recognition
US6112021 *19 Dec 199729 Aug 2000Mitsubishi Electric Information Technology Center America, Inc, (Ita)Markov model discriminator using negative examples
US6128003 *22 Dec 19973 Oct 2000Hitachi, Ltd.Hand gesture recognition system and method
US6159100 *23 Apr 199812 Dec 2000Smith; Michael D.Virtual reality game
US6161126 *2 Feb 199912 Dec 2000Immersion CorporationImplementing force feedback over the World Wide Web and other computer networks
US6176782 *22 Dec 199723 Jan 2001Philips Electronics North America Corp.Motion-based command generation technology
US6195104 *23 Dec 199727 Feb 2001Philips Electronics North America Corp.System and method for permitting three-dimensional navigation through a virtual reality environment using camera-based gesture inputs
US6215890 *25 Sep 199810 Apr 2001Matsushita Electric Industrial Co., Ltd.Hand gesture recognizing device
US6239389 *21 Jun 199929 May 2001Synaptics, Inc.Object position detection system and method
US6249606 *19 Feb 199819 Jun 2001Mindmaker, Inc.Method and system for gesture category recognition and training using a feature vector
US6285380 *1 Aug 19974 Sep 2001New York UniversityMethod and system for scripting interactive animated actors
US6304674 *3 Aug 199816 Oct 2001Xerox CorporationSystem and method for recognizing user-specified pen-based gestures using hidden markov models
US6352478 *17 Apr 19985 Mar 2002Creator, Ltd.Techniques and apparatus for entertainment sites, amusement parks and other information and/or entertainment dispensing sites
US6353764 *25 Nov 19985 Mar 2002Matsushita Electric Industrial Co., Ltd.Control method
US6363160 *22 Jan 199926 Mar 2002Intel CorporationInterface using pattern recognition and tracking
US6400996 *1 Feb 19994 Jun 2002Steven M. HoffbergAdaptive pattern recognition based control system and method
US6503086 *25 Apr 20007 Jan 2003Michael M. GolubovBody motion teaching system
US6508706 *21 Jun 200121 Jan 2003David Howard SitrickElectronic interactive gaming apparatus, system and methodology
US6561811 *20 Mar 200213 May 2003Entertainment Science, Inc.Drug abuse prevention computer game
US6681031 *10 Aug 199920 Jan 2004Cybernet Systems CorporationGesture-controlled interfaces for self-service machines and other applications
US6687612 *24 Mar 20033 Feb 2004Navigation Technologies Corp.Method and system using a hand-gesture responsive device for collecting data for a geographic database
US6694044 *1 Sep 200017 Feb 2004Hewlett-Packard Development Company, L.P.Method for motion classification using switching linear dynamic system models
US6694058 *22 Jan 199917 Feb 2004Wincor Nixdorf Gmbh & Co. KgMethod for monitoring the exploitation process of an apparatus and self-service device monitored according to said method
US6819782 *8 Jun 200016 Nov 2004Matsushita Electric Industrial Co., Ltd.Device and method for recognizing hand shape and position, and recording medium having program for carrying out the method recorded thereon
US6944315 *31 Oct 200013 Sep 2005Intel CorporationMethod and apparatus for performing scale-invariant gesture recognition
US7225414 *5 Aug 200329 May 2007Videomining CorporationMethod and system for virtual touch entertainment
US7340077 *18 Feb 20034 Mar 2008Canesta, Inc.Gesture recognition system using depth perceptive sensors
US7454342 *19 Mar 200318 Nov 2008Intel CorporationCoupled hidden Markov model (CHMM) for continuous audiovisual speech recognition
US7665041 *25 Mar 200316 Feb 2010Microsoft CorporationArchitecture for controlling a computer using hand gestures
US8745541 *1 Dec 20033 Jun 2014Microsoft CorporationArchitecture for controlling a computer using hand gestures
US20020022521 *15 May 200121 Feb 2002Konami CorporationGame machine and network system for setting up game environment thereof
US20020036617 *21 Aug 199828 Mar 2002Timothy R. PryorNovel man machine interfaces and applications
US20020082724 *14 Nov 200127 Jun 2002Bernard HennionForce feedback member control method and system
US20020118880 *31 Oct 200129 Aug 2002Che-Bin LiuSystem and method for gesture interface
US20020181773 *28 Mar 20015 Dec 2002Nobuo HigakiGesture recognition system
US20030028498 *7 Jun 20026 Feb 2003Barbara Hayes-RothCustomizable expert agent
US20030055640 *30 Apr 200220 Mar 2003Ramot University Authority For Applied Research & Industrial Development Ltd.System and method for parameter estimation for pattern recognition
US20030076293 *13 Sep 200224 Apr 2003Hans MattssonGesture recognition system
US20030076367 *18 Oct 200224 Apr 2003Paul BenczeRich communication over internet
US20030103653 *4 Dec 20015 Jun 2003Yossi AvniSystem for and method of web signature recognition system based on object map
US20030149803 *31 May 20027 Aug 2003Andrew WilsonSystem and process for controlling electronic components in a ubiquitous computing environment using multimodal integration
US20030156756 *18 Feb 200321 Aug 2003Gokturk Salih BurakGesture recognition system using depth perceptive sensors
US20030170602 *6 Feb 200311 Sep 2003Norihiro HagitaInteraction media device and experience transfer system using interaction media device
US20030185445 *31 Jul 20022 Oct 2003Industrial Technology Research InstituteMethod for extracting and matching gesture features of image
US20030208289 *1 May 20036 Nov 2003Jezekiel Ben-ArieMethod of recognition of human motion, vector sequences and speech
US20040056907 *19 Sep 200325 Mar 2004The Penn State Research FoundationProsody based audio/visual co-analysis for co-verbal gesture recognition
US20110173574 *8 Jan 201014 Jul 2011Microsoft CorporationIn application gesture interpretation
US20130044912 *13 Jul 201221 Feb 2013Qualcomm IncorporatedUse of association of an object detected in an image to obtain information to display to a user
Non-Patent Citations
Reference
1 *TypingMaster Pro Screenshots and descriptions, posted 06/24/2003 retrieved from [https://web.archive.org/web/20030619202824/http://typingmaster.com/index.asp?go=tutor] on [01/11/2016]
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US20150370332 *9 Dec 201324 Dec 2015Sagemcom Broadband SasDevice and method for recognizing gestures for a user control interface
Classifications
U.S. Classification715/863
International ClassificationG06F1/16, G06K9/00, G06F3/00, G06F15/18, G06K9/74, G06K9/62, G06F3/03, G06K9/22, G06F3/0484, G06F
Cooperative ClassificationG06F3/017, G06F1/1694, G06F1/1626, G06F3/0484, G06F2200/1637
Legal Events
DateCodeEventDescription
3 Nov 2014ASAssignment
Owner name: NOKIA CORPORATION, FINLAND
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KELA, JUHA;KORPIPAA, PANU;MANTY JARVI, JANI;AND OTHERS;SIGNING DATES FROM 20050415 TO 20050429;REEL/FRAME:034093/0739
9 Dec 2014ASAssignment
Owner name: NOKIA CORPORATION, FINLAND
Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE THIRD INVENTOR S NAME PREVIOUSLY RECORDED ON REEL 034093 FRAME0739. ASSIGNOR(S) HEREBY CONFIRMS THE THIRD INVENTOR S NAME TO BE JANI MANTYJARVI;ASSIGNORS:KELA, JUHA;KORPIPAA, PANU;MANTYJARVI, JANI;AND OTHERS;SIGNING DATES FROM 20050415 TO 20050429;REEL/FRAME:034545/0556
26 Apr 2015ASAssignment
Owner name: NOKIA TECHNOLOGIES OY, FINLAND
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NOKIA CORPORATION;REEL/FRAME:035495/0924
Effective date: 20150116