WO2016144502A1

WO2016144502A1 - Processing digital ink input subject to monitoring and intervention by an application program

Info

Publication number: WO2016144502A1
Application number: PCT/US2016/018565
Authority: WO
Inventors: Yibo SUN; Krishnan Menon; Connor WEINS; Xiao TU; Olumuyiwa Durojaiye
Original assignee: Microsoft Technology Licensing, Llc
Priority date: 2015-03-12
Filing date: 2016-02-19
Publication date: 2016-09-15
Also published as: US9950542B2; CN107438818B; EP3268849A1; US20160263911A1; CN107438818A

Abstract

A facility for processing ink input is described. In one example facility, the facility receives ink input from an input device. The facility generates and renders ink stroke data structures in response to received ink input in accordance with an ink input processing pipeline. The facility provides to an executing application access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.

Description

PROCESSING DIGITAL INK INPUT SUBJECT TO MONITORING AND INTERVENTION BY AN APPLICATION PROGRAM

BACKGROUND

[0001] Electronic ink is content added to an electronic document based upon a user input technique where the user handwrites, draws, or performs other arbitrary spatial movements. As a result of such user input, visual marks are added to the document that correspond to the path of the movement. For example, in the case of handwriting, handwritten words appear in the document in the same handwriting style that they were executed by the user.

[0002] In some cases, the user performs this input by moving a stylus or his or her finger across the surface of the display device on which the document is displayed. In such cases, the electronic ink generally appears in the document in the positions where the interactions occurred. In some cases, the user performs the inking input by moving a displayed position pointer on the screen, using an input device such as a mouse or touchpad.

[0003] In the case of electronic ink corresponding to handwriting, handwriting recognition techniques can be used to identify text represented by the handwriting, which can be inserted in the document along with or in place of the electronic ink handwriting.

[0004] Ink is generally consumed by application programs that register to receive it. In such a case, it is typical for an ink-aware operating system to process ink-related input events to do some or all of the following automatically on behalf of the registered application: generating ink — that is, creating the data structures representing ink, including strokes interpolated and/or smoothed from a sequence of points along a movement path; rendering ink— that is, causing a visual representation of ink to be displayed in the application's display area; and/or "drying" ink by transferring the data structures that represent it to the control of the application.

SUMMARY

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [0006] A facility for processing ink input is described. In one example facility, the facility receives ink input from an input device. The facility generates and renders ink stroke data structures in response to received ink input in accordance with an ink input processing pipeline. The facility provides to an executing application access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates.

[0008] Figure 2 is a flow diagram showing example acts that may be performed by the facility in some examples to process broad device input.

[0009] Figure 3 is a table diagram showing sample contents of an ink event registration table used by the facility in some examples to store information about applications' registration of ink event handlers.

[0010] Figure 4 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates.

DETAILED DESCRIPTION

[0011] The inventors have observed that the operating system's extensive isolation of conventional electronic ink processing systems from the applications that consume of their ink preclude these applications from implementing a variety of desirable, ink-aware features. For example, when operating in conjunction with conventional ink processing systems, such applications are generally unable to implement features such as allowing a user to invoke certain application functionality using gestures executed with a pen or other inking pointing device, such as crossing out an email message in a list of email messages in order to delete the email message from the list; preventing inking in a certain region of the application's display area that contains visual information that should not be obscured; sending ink strokes to a collaborating user's computer system as they are being composited from individual pen move inputs; implementing customized ink selection, ink erasing, and cursor hovering functionalities; etc.

[0012] In order to enable a fuller range of ink-aware features among applications, the inventors have conceived and reduced to practice a software and/or hardware facility for processing digital ink input subject to monitoring and intervention by an application program ("the facility").

[0013] In various examples, the facility permits an application to intercept information at one or more points in the operating system's ink processing pipeline, including raw device input; ink strokes generated by raw device input for rendering; and types of input not used to generate ink strokes. In some examples, the facility permits the application to register event handlers for events occurring at any of these points in the ink processing pipeline.

[0014] In some examples, the application's event handlers can use their access to events of these types to affect the behavior of the ink processing pipeline. As one example, in some cases an event handler for raw device input can, based upon analysis of this raw device input, instruct the pipeline to abort processing and rendering of an in- progress stroke, such as where a stroke would be improper in the current location of the application display area, or where the application recognizes the raw device input making up the stroke as matching a user intent other than inking a stroke, such as issuing a command to the application via a gesture.

[0015] In some examples, the facility causes the application's event handlers to execute at different levels of priority for different categories of events. In particular, in some examples, the facility causes the application's event handler for raw device input to execute in a high-priority background thread used by the pipeline to process raw device input into strokes, while causing the application's event handlers for stroke and non-stroke of events in a low-priority host thread— such as an application user interface thread—used by the application to transfer completed strokes into its own representation of the document being edited. (This transferring is sometimes referred to as "drying," and any stroke that has not been so dried— such as strokes that are in progress— are sometimes referred to as "wet.") In some examples, the facility monitors execution of application's event handlers in high-priority threads to ensure that the application's event handlers do not abuse the high priority of these threads by doing extensive processing, thus preventing such event handlers from interfering with the pipeline's prompt generation and rendering of ink strokes, which is an important part of providing a good inking experience to users.

[0016] By performing in some or all of these ways, the facility enables a wider range of useful and attractive ink-aware features on the part of applications. [0017] Figure 1 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates. In various examples, these computer systems and other devices 100 can include server computer systems, desktop computer systems, laptop computer systems, tablet computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, electronic kiosk devices, electronic table devices, electronic whiteboard devices, etc. In various examples, the computer systems and devices may include any number of the following: a central processing unit ("CPU") 101 for executing computer programs; a computer memory 102 for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel and device drivers, and one or more applications; a persistent storage device 103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive 104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer- readable medium; and/or a communications subsystem 105 for connecting the computer system to other computer systems and/or other devices to send and/or receive data, such as via the Internet or another wired or wireless network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like.

[0018] In various examples, these computer systems and other devices 100 may further include any number of the following: a display 106 for presenting visual information, such as text, images, icons, documents, menus, etc.; and a touchscreen digitizer 107 for sensing interactions with the display, such as touching the display with one or more fingers, styluses, or other objects. In various examples, the touchscreen digitizer uses one or more available techniques for sensing interactions with the display, such as resistive sensing, surface acoustic wave sensing, surface capacitance sensing, projected capacitance sensing, infrared grid sensing, infrared acrylic projection sensing, optical imaging sensing, dispersive signal sensing, and acoustic pulse recognition sensing. In various examples, the computer systems and other devices 100 include input devices of various other types, such as keyboards, mice, styluses, etc. (not shown).

[0019] While computer systems configured as described above may be used to support the operation of the facility, those skilled in the art will appreciate that the facility may be implemented using devices of various types and configurations, and having various components.

[0020] Figure 2 is a flow diagram showing example acts that may be performed by the facility in some examples to process broad device input. In some examples, the facility performs the acts of Figure 2 in a thread having high execution priority, such as a background thread for generating and rendering ink strokes.

[0021] At 201, the facility receives device input. If the application has registered an input event handler for receiving device input, then the facility continues at 211, else the facility continues at 203. In various examples, the facility provides various mechanisms for an application to register an event handler, such as calling an operating system API, calling an ink system API, storing one or more key/value pairs in the registry, etc. At 203, the facility branches on the type of the input received at 201 : if the input is pen down input - such as input indicating that the user has touched the display with a finger, or stylus, or another object, or activated another type of pointing input device such as by pressing down the button on a mouse - the facility continues at 204; if the input is pen move input reflecting the movement of the pointing device following pen down input and preceding pen up input, then the facility continues at 207; if the input is pen up input reflecting the removal of the finger or other object touching the display, the release of the mouse button, etc., then the facility continues at 208; and if the input is a type of input no named, then the facility continues at 209.

[0022] At 204, where the received input is pen down input, the facility creates and renders an ink stroke as wet ink. Often this new stroke is a single point, or a very short line segment. At 205, if the application has registered a stroke event handler, then the facility continues at 206, else the facility returns to 201 to receive the next device input. At 206, the facility fires a stroke event that is serviced by the application on the low- priority host thread, such as the application's user interface thread. In various examples, the stroke handler can perform processing such as sending strokes to a remote computer system as they are formed, such as sending them to a computer system being used by a collaborating user who can then watch as the present user forms strokes with ink input. After 206, the facility returns to 201.

[0023] At 207, where the input is pen move input, the facility revises and renders the in-progress stroke as wet ink. In some examples, such revision involves extending the stroke to a new point that is the destination of the move input, smoothing the shape of the stroke in light of this extension, etc. After 207, the facility continues at 205.

[0024] At 208, where the input is pen up input, the facility completes and renders the in-progress stroke as wet ink, and causes this wet ink to be dried in the host thread. After 208, the facility continues at 205.

[0025] At 209, where the input is other input, if the application has registered a non- stroke event handler, then the facility continues at 210, else the facility returns to 201. In various examples, this other input can be of various types, such as hovering, erasing, pressing a stylus barrel button, clicking a mouse's right button, etc. At 210, the facility fires a stroke event, serviced by the application on the host thread. In various examples, the non-stroke event handler performs various processing, such as performing custom forms of erasing, selection of ink and other content, hovering, etc. After 210, the facility returns to 201.

[0026] In some examples, the facility fires stroke events and non-stroke events at 206 and 210, respectively, in such a manner that each event is received by the application's event handlers independently. In some examples, however, the facility accumulates a sequence of such events in a queue, where the sequence can be consumed together by the application's event handler. In some examples, the facility's treatment of such events is configurable by the application, such as by specifying whether this queuing should be performed in the application's registration of an event handler, in some subsequent call to the operating system, etc.

[0027] In 211, where the application has registered an input event handler for raw device input events, the facility fires an input event corresponding to the input received at 201. The input event is serviced by the application on the high-priority background thread. In various examples, the input event handler performs various processing, such as detecting a gesture and invoking a corresponding command in response; preventing inking in a forbidden region; etc. As part of servicing the input event, the application's input event handler can determine whether the input event should be marked as "handled," reflecting that the application is claiming the device input that makes up any current in- progress stroke, and that this device input should not be processed by the pipeline. Accordingly, at 212, if the device input is marked as handled by the input event handler, then the facility continues at 213, else the facility continues at 203. At 213, the facility cancels and de-renders the in-process stroke. At 214, the facility receives additional device input. At 215, the facility fires an input event in a manner similar to act 211, that corresponds to the input received at 214. At 216, if the type of the input is pen up input, then the facility returns to 201, else the facility returns to 214.

[0028] Those skilled in the art will appreciate that the acts shown in Figure 2 may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.

[0029] As a further example of the facility's operation, an application can interact with the facility in such a way as to drag selected ink to a new location. In this example, the application's input event handler processes a pen down action and determines that the pen down occurred inside a selected region, such as a selected rectangle. The input event handler calls an input configuration API to notify the operating system that the present input sequence should be directed to the host thread as non-stroke events. In the host thread, the application processes the pen down action and subsequent pen move actions by dragging the selected ink strokes in accordance with the pen move actions.

[0030] Figure 3 is a table diagram showing sample contents of an ink event registration table used by the facility in some examples to store information about applications' registration of ink event handlers. The table 300 is made up of rows, such as rows 301-303, each corresponding to a different event handler registered by an application. Each row is made up of the following columns: a process column 311 containing identifier for the process in which the application that registered the event handler is executing; a type column 312 indicating the category of events - input, stroke, or nonstroke - for which the event handler is registered; and a code pointer column 313 containing an entry point for the event handler, or other information that can be used to invoke the event handler as part of firing an event of the registered category. For example, row 301 indicates that an application executing in process 1 has registered a stroke event handler having entry point XXXXXX. According to the state of the table, this application has not registered an event handler for either input events or non-stroke events. On the other hand, rows 302-303 indicate that an application executing in process 2 has registered event handlers for both input events and non-stroke events, but has not registered an event handler for stroke events. [0031] While Figure 3 shows a table whose contents and organization are designed to make it more comprehensible by a human reader, those skilled in the art will appreciate that actual data structures used by the facility to store this information may differ from the table shown, in that they, for example, may be organized in a different manner; may contain more or less information than shown; may be compressed and/or encrypted; may contain a much larger number of rows than shown; etc.

[0032] Figure 4 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates. The computing system 400 includes a processor 410 configured to execute an application. The computing system further includes an input device interface 420 configured to receive ink input from an input device. The computing system also includes an ink input processing subsystem 430 configured to generate and render ink stroke data structures. It does so in response to ink input received from an input device by the input device interface, in accordance with an ink input processing pipeline. The computing system also includes an ink input processing pipeline access subsystem 440 configured to provide to the application executed by the processor access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline. The computing system further includes an ink input processing control system 450 configured to, in response to receiving from the application executed on the processor an instruction, omitting to generate and render ink stroke data structures with respect to a selected portion of ink input received by the input device interface. The computing system also includes a monitoring subsystem 460 configured to monitor use of processing resources by an event handler that is part of the application executed by the processor.

[0033] In some examples, the facility provides a computing system for processing ink input, comprising: a processor configured to execute an application; an input device interface configured to receive ink input; an ink input processing subsystem configured to generate and render ink stroke data structures in response to ink input received by the input device interface in accordance with an ink input processing pipeline; and an ink input processing pipeline access subsystem configured to provide to the application executed by the processor access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.

[0034] In some examples, the facility provides a method in a computing system for processing ink input in connection with an application program, the method comprising: permitting the application program to register an event handler for input events; permitting the application program to register an event handler for stroke events; receiving input events from an input device; in response to the application registering an event handler for input events, providing access to the received input events to the registered event handler for input events; processing at least a portion of the input events to generate a plurality of ink strokes; and in response to the application registering an event handler for stroke events, providing access to the generated strokes to the registered event handler for stroke events.

[0035] In some examples, the facility provides a computer-readable medium having contents configured to cause a computing system to, in order to handle inking input: in an operating system: invoke an application; receive a stream of inking input events; for each inking input event received by the operating system, before generating ink based on the inking input event, provide to code specified as part of the invoked application access to the inking input event; and for each of at least a first portion of the inking input events received by the operating system, after providing to code specified as part of the invoked application access to the inking input event, generate ink based on the inking input event.

[0036] It will be appreciated by those skilled in the art that the above-described facility may be straightforwardly adapted or extended in various ways. While the foregoing description makes reference to particular examples, the scope of the invention is defined solely by the claims that follow and the elements recited therein.

Claims

1. A computer-readable medium having contents configured to cause a computing system to, in order to handle inking input:

in an operating system:

invoke an application;

receive a stream of inking input events;

for each inking input event received by the operating system, before generating ink based on the inking input event, provide to code specified as part of the invoked application access to the inking input event; and

for each of at least a first portion of the inking input events received by the operating system, after providing to code specified as part of the invoked application access to the inking input event, generate ink based on the inking input event.

2. The computer-readable medium of claim 1 wherein the computer-readable medium has contents configured to further cause a computing system to:

in the operating system,

receive from the code specified as part of the invoked application a direction with respect to a distinguished inking input event to which access was provided, wherein, based at least on receiving the direction, generation of ink based on the distinguished inking input event in the operating system is performed in accordance with the received direction.

3. The computer-readable medium of claim 1 wherein the computer-readable medium has contents configured to further cause a computing system to:

in the operating system,

receive from the code specified as part of the invoked application a direction with respect to a distinguished inking input event to which access was provided, wherein, based at least on receiving the direction, the distinguished inking input event is not among the inking input events for which ink generation is performed in the operating system.

4. The computer-readable medium of claim 1 wherein the computer-readable medium has contents configured to further cause a computing system to:

in the operating system:

for each of at least a second portion of the inking input events received by the operating system, render ink based on the inking input event; and receive from the code specified as part of the invoked application a direction with respect to a distinguished inking input event to which access was provided, wherein, based at least on receiving the direction, the distinguished inking input event is not among the inking input events for which ink rendering is performed in the operating system.

5. A method in a computing system for processing ink input in connection with an application program, the method comprising:

permitting the application program to register an event handler for input events; permitting the application program to register an event handler for stroke events; receiving input events from an input device;

based at least on the application registering an event handler for input events, providing access to the received input events to the registered event handler for input events;

processing at least a portion of the input events to generate a plurality of ink strokes; and

based at least on the application registering the event handler for stroke events, providing access to the generated strokes to the registered event handler for stroke events.

6. The method of claim 5, further comprising

receiving from the application program a direction to cancel a selected ink stroke; and

based at least on receiving the direction, canceling the selected ink stroke.

7. The method of claim 5 wherein providing access to the generated strokes to the registered event handler for stroke events comprises storing information about the generated strokes in a queue to which the registered event handler for stroke events has access.

8. A computing system for processing ink input, comprising:

at least one processor configured to execute an application;

an input device interface configured to receive ink input;

an ink input processing subsystem configured to generate and render ink stroke data structures based at least on ink input received by the input device interface in accordance with an ink input processing pipeline; and an ink input processing pipeline access subsystem configured to provide to the application executed by the at least one processor access to information traversing the ink input processing pipeline at a selected point in the ink input processing pipeline.

9. The computing system of claim 8 wherein the selected point in the ink input processing pipeline is a point in the ink input processing pipeline before generation of ink stroke data structures.

10. The computing system of claim 8 wherein the selected point in the ink input processing pipeline is a point in the ink input processing pipeline beyond generation of ink stroke data structures.

11. The computing system of claim 8 wherein the application executed by the at least one processor comprises an event handler, and the ink input processing pipeline access subsystem is further configured to provide access via the event handler.

12. The computing system of claim 9, further comprising:

an ink input processing control subsystem configured to: receive from the application executed on the at least one processor an instruction, and based at least on the instruction, omit generating and rendering ink stroke data structures with respect to a selected portion of ink input received by the input device interface.

13. The computing system of claim 9 wherein the application executed by the at least one processor comprises an event handler,

the ink input processing pipeline access subsystem is further configured to provide access via the event handler,

and the event handler is executed by the at least one processor in a low-latency thread.

14. The computing system of claim 13 wherein the low-latency thread in which the event handler is executed also executes code for the ink input processing subsystem.

15. The computing system of claim 8 wherein the selected point in the ink input processing pipeline is a point in the ink processing pipeline traversed by ink input of one or more types not used by the ink input processing system to generate and render ink stroke data structures.