WO2009024966A2

WO2009024966A2 - Method for adapting media for viewing on small display screens

Info

Publication number: WO2009024966A2
Application number: PCT/IL2008/001128
Authority: WO
Inventors: Michael Birnboim; David Amselem
Original assignee: Closevu Ltd.
Priority date: 2007-08-21
Filing date: 2008-08-14
Publication date: 2009-02-26
Also published as: WO2009024966A3

Abstract

The present invention discloses devices, methods, and media for adapting media for viewing. A computing device for calculating graphics parameters in media, the device including: a central processor unit for performing computing operations; and a memory for handling data resulting from the computing operations, the memory including program code for: analyzing a surrounding of a graphics area of an input frame; determining whether the surrounding is smooth; upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; determining whether at least two substantially-distinct colors have been recorded in the surrounding color; and upon determining at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from at least two substantially-distinct colors and the recorded graphics-area pixel color for each graphics-area pixel location.

Description

METHODS FOR ADAPTING MEDIA FOR VIEWING ON SMALL DISPLAY SCREENS

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to methods for adapting media for viewing on small display screens, primarily for use by content creators and television stations. Today, an increasing amount of television shows are being shot in HD format

(HDTV) for viewing on large-size display screens. However, the heralded, high quality of HD format does not translate well to mobile devices. The small display screens of cell phones, PDAs, and other electronic mobile devices, make it hard for viewers to read text overlays (e.g. names, scores, and statistics) on the screen, and to see long-distance shots that are typical of HDTV. Table 1 shows screen format and/or resolution for display sizes of several types of devices.

Table 1. Screen format and/or resolution for display sizes of various device types.

Attention must be given to the viewing experience, and hence the programming needs, of small-screen display devices in order to make the experience user-friendly and attractive to mobile TV viewers. Studies show that viewers using hand-held TVs tend to hop from channel to channel more often than users watching regular TV sets. Because of such mobile viewing-patterns, television producers and TV groups need to create fast-paced content offerings of local news, weather, and sports highlights. From a production standpoint, it is highly unlikely that television stations will hire separate crews for shooting video to accommodate small screens due to the associated expense. Instead, television stations will look for easy and affordable tools to resize and fine-tune HD content to make it suitable for small-screen viewing.

Most sporting events are shot in HD for viewing on large TV screens. However, when wide-screen video is "shrunk down" to fit the small screens of mobile devices, the action scenes, big plays, and scores become lost. It is difficult to follow long-distance shots, to see a ball, and to read text overlays of names or numbers. The original video needs to be modified and "tweaked" both for live mobile-TV broadcast, as well as for mobile replays and post-game highlight shows.

The necessary modifications include: (1) zooming in to make scenes larger, turning long shots into close-ups in order to best capture the action or focal point of a sequence;

(2) replacing graphics due to distortion of original graphics of broadcast programs after being compressed for streaming to hand-held devices, due to the fact that graphics displayed on a small screen are hard to read, and/or due to the fact that reframing can cause graphics to be missing, in the wrong place, or partially visible, making the graphics almost unreadable (original graphics need to be stripped from the video source, and replaced with crisp mobile-friendly graphics);

(3) highlighting objects of interest by enabling, for example, a viewer to highlight a ball or a specific player from a mobile device, building interactivity and sponsorship opportunities by providing viewers with single-click graphic enhancements to the video program;

(4) making a smooth, natural camera view by eliminating jerky motions; and (5) ad detection and replacement/modification, due to incompatible ad placement from the source video to the mobile device, by detecting source ads located in a scene (using comparison to an ad database or relying on special ad features), separating ad sequences during a broadcast intermission, and replacing the source ads with small-screen ads, or modifying the source ad to fit better with the medium by adding interactive banners. The process of reformatting content for new platforms and devices can require a lot of time, human effort, and equipment resources. Producers can access the content files of a broadcast station, but require effective methods for integrating current television production into content suitable for mobile TV. Typically, a TV producer is not a professional video editor, but more likely a journalist or a content manager without extensive technical editing skills. Such a producer needs a quick, easy, and simple tool to repurpose content for new platforms, including mobile devices.

It would be desirable to have methods for adapting media for viewing on small display screens.

SUMMARY OF THE INVENTION

It is the purpose of the present invention to provide methods for adapting media for viewing on small display screens.

For the purpose of clarity, several terms which follow are specifically defined for use herein. The term "shot" is used herein to refer to a continuous strip of motion- picture film, composed of a series of frames that runs uninterrupted for a specific period of time. Shots are generally filmed with a single camera and can be of any duration. A shot can be compared to a word with each frame being a letter. The term "cut" is used herein to refer to a sequence of frames that connects one shot to the next shot. There are many types of cuts (e.g. fade in, dissolve, and wipe).

Embodiments of the present invention utilize advanced object-tracking technologies that enable a producer to apply a customized visual style and user- defined preferences to reformat standard-definition (SD) and high-definition (HD) video into compelling small-screen video experiences. A preferred embodiment of the present invention includes production tools configured to help create new content suitable for mobile TV, and/or to easily reformat existing content/programs. Embodiments of the present invention provide post-production tools that aid an operator in adapting video content to formats that are optimally-viewed on small screens. CloseVU Live includes production tools configured for operators, content creators, and TV stations dealing with live or voluminous content. Such production tools adjust and adapt live content to the mobile TV and handheld-device environments. Other users of the present invention include mobile network operators (e.g. telecommunications operators) who are directly involved in producing content or in re-purposing third-party content for mobile use.

Therefore, according to the present invention, there is provided for the first time a computing device for calculating graphics parameters in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) analyzing a surrounding of a graphics area of an input frame; (ii) determining whether the surrounding is smooth; (iii) upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (iv) determining whether at least two substantially-distinct colors have been recorded in the surrounding color; and (v) upon determining at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from at least two substantially-distinct colors and the recorded graphics-area pixel color for each graphics-area pixel location.

According to the present invention, there is provided for the first time a computing device for calculating graphics parameters in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) analyzing a surrounding of a graphics area of an input frame; (ii) determining whether the surrounding is smooth; (iii) upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (iv) for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (v) for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (vi) choosing a representative value from the surrounding computational parameter; and (vii) for each graphics-area pixel, calculating a color value and an alpha value from the recorded surrounding color and the recorded graphics-area pixel color and from a relation between at least one graphics computational parameter and the representative value. According to the present invention, there is provided for the first time a computing device for adapting media for viewing, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames; (ii) for each graphics-area pixel, checking if the alpha value is transparent in the graphics area; (iii) for each graphics-area pixel for which the alpha value is transparent, replacing a pixel value with a calculated-background value obtained from a relation between the color value and the alpha value; and (iv) for each graphics-area pixel for which the alpha value is not transparent, performing a pixel-modification procedure.

Preferably, the program code for performing the pixel-modification procedure includes program code for an in-painting process. ^{~ ~}

Preferably, the program code for performing the pixel-modification procedure includes program code for substituting a combination pixel value from a past-frame pixel value and a future-frame pixel value.

According to the present invention, there is provided for the first time a computing device for detecting graphics disappearance in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) calculating graphics edges of a graphics element; (ii) calculating graphics-area edges of a graphics area in an input frame; (iii) removing missing edges not present in at least one previous frame; and (iv) upon determining that a percentage of the graphics edges that are not present in the graphics-area edges is greater than a pre-determined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a computing device for detecting graphics disappearance in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) calculating, for each pixel in a graphics area in an input frame, a maximum color-value limit and a minimum color-value limit using a pixel alpha value and a graphics-pixel color value; (ii) determining whether each pixel of the input frame is outside the color- value limits; and (iii) upon determining that a pixel amount outside the color- value limits is greater than a pre-determined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a computing device for selective zooming of media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) checking for a scene cut in an input frame of a plurality of frames which define at least one shot; (ii) marking at least one keyframe in at least one shot by designating a keyframe zoom-location and a keyframe zoom-size; and (iii) interpolating a respective zoom-location and a respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein at least one keyframe shot includes at least one keyframe.

Preferably, the memory further includes program code for: (iv) de-interlacing the zoomed portion.

Preferably, the memory further includes program code for: (iv) providing a graphics and an alpha for an input frame of a plurality of frames; and (v) inserting a graphics element into a zoomed portion of the input frame, wherein the graphics element has a graphics zoom-location and a graphics zoom-size based on to the graphics and the alpha and a user input.

Preferably, the respective zoom-location and the respective zoom-size are determined by at least one keyframe and at least one tracked object in at least one keyframe shot.

According to the present invention, there is provided for the first time a computing device for adapting graphics to a zoomed media frame, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) defining a graphics area; (ii) setting a zoom portion of the media frame in a way that excludes the graphics area; and (iii) copying the graphics area to a fixed position relative to the zoomed portion.

According to the present invention, there is provided for the first time a computing device for object tracking and highlighting in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) marking an object to highlight in an input frame; (ii) tracking the object; and (iii) attaching highlighting graphics to the object. According to the present invention, there is provided for the first time a computing device for near-live editing of media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) marking a desired start-frame in a plurality of frames; and (ii) excerpting a desired frame sequence from the plurality of frames, wherein the desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop-frame according to a predefined stop-frame rule, wherein the predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before the desired start-frame, a predetermined time interval after the desired start-frame, a closest scene cut to the desired start- frame, a predetermined scene-cut interval before the desired start-frame, and a predetermined scene-cut interval after the desired start-frame, and wherein the predefined stop-frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after the true start-frame, a predetermined frame interval after the true start- frame, a predetermined scene-cut interval after the true start-frame, a desired stop- frame, a predetermined time interval before the desired stop-frame, a predetermined time interval after the desired stop-frame, a closest scene cut to the desired stop- frame, a predetermined scene-cut interval before the desired stop-frame, and a predetermined scene-cut interval after the desired stop-frame. According to the present invention, there is provided for the first time a computing device for detecting and modifying advertisements in media, the device including: (a) a central processor unit for performing computing operations; and (b) a memory for handling data resulting from the computing operations, the memory including program code for: (i) detecting a frame series, in a plurality of frames, that includes an advertisement; and (ii) modifying the advertisement, in all frames of the frame series, using a procedure selected from the group consisting of: replacing the advertisement with a substitute advertisement, replacing the advertisement with an small-screen-optimized advertisement, and adding at least one interactive banner to the advertisement.

According to the present invention, there is provided for the first time a method for calculating graphics parameters in media, the method including the steps of: (a) analyzing a surrounding of a graphics area of an input frame; (b) determining whether the surrounding is smooth; (c) upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (d) determining whether at least two substantially-distinct colors have been recorded in the surrounding color; and (e) upon determining at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from at least two substantially-distinct colors and the recorded graphics-area pixel color for each graphics-area pixel location.

According to the present invention, there is provided for the first time a method for calculating graphics parameters in media, the method including the steps of: (a) analyzing a surrounding of a graphics area of an input frame; (b) determining whether the surrounding is smooth; (c) upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (d) for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (e) for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (f) choosing a representative value from the surrounding computational parameter; and (g) for each graphics-area pixel, calculating a color value and an alpha value from the recorded surrounding color and the recorded graphics-area pixel color and from a relation between at least one graphics computational parameter and the representative value.

According to the present invention, there is provided for the first time a method for adapting media for viewing, the method including the steps of: (a) providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames; (b) for each graphics-area pixel, checking if the alpha value is transparent in the graphics area; (c) for each graphics-area pixel for which the alpha value is transparent, replacing a pixel value with a calculated-background value obtained from a relation between the color value and the alpha value; and (d) for each graphics-area pixel for which the alpha value is not transparent, performing a pixel- modification procedure.

Preferably, the pixel-modification procedure includes an in-painting process. Preferably, the pixel-modification procedure includes substituting a combination pixel value from a past-frame pixel value and a future-frame pixel value. According to the present invention, there is provided for the first time a method for detecting graphics disappearance in media, the method including the steps of: (a) calculating graphics edges of a graphics element; (b) calculating graphics-area edges of a graphics area in an input frame; (c) removing missing edges not present in at least one previous frame; and (d) upon determining that a percentage of the graphics edges that are not present in the graphics-area edges is greater than a predetermined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a method for detecting graphics disappearance in media, the method including the steps of: (a) calculating, for each pixel in a graphics area in an input frame, a maximum color-value limit and a minimum color-value limit using a pixel alpha value and a graphics-pixel color value; (b) determining whether each pixel of the input frame is outside the color-value limits; and (c) upon determining that a pixel amount outside the color-value limits is greater than a pre-determined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a method for selective zooming of media, the method including the steps of: (a) checking for a scene cut in an input frame of a plurality of frames which define at least one shot; (b) marking at least one keyframe in at least one shot by designating a keyframe zoom-location and a keyframe zoom-size; and (c) interpolating a respective zoom-location and a respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein at least one keyframe shot includes at least one keyframe. Preferably, the method further includes the step of: (d) de-interlacing the zoomed portion. Preferably, the method further includes the steps of: (d) providing a graphics and an alpha for an input frame of a plurality of frames; and (e) inserting a graphics element into a zoomed portion of the input frame, wherein the graphics element has a graphics zoom-location and a graphics zoom-size based on to the graphics and the alpha and a user input.

According to the present invention, there is provided for the first time a method for adapting graphics to a zoomed media frame, the method including the steps of: (a) defining a graphics area; (b) setting a zoom portion of the media frame in a way that excludes the graphics area; and (c) copying the graphics area to a fixed position relative to the zoomed portion.

According to the present invention, there is provided for the first time a method for object tracking and highlighting in media, the method including the steps of: (a) marking an object to highlight in an input frame; (b) tracking the object; and (c) attaching highlighting graphics to the object.

According to the present invention, there is provided for the first time a method for near-live editing of media, the method including the steps of: (a) marking a desired start-frame in a plurality of frames; and (b) excerpting a desired frame sequence from the plurality of frames, wherein the desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop- frame according to a predefined stop-frame rule, wherein the predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before the desired start-frame, a predetermined time interval after the desired start-frame, a closest scene cut to the desired start-frame, a predetermined scene-cut interval before the desired start-frame, and a predetermined scene-cut interval after the desired start-frame, and wherein the predefined stop-frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after the true start-frame, a predetermined frame interval after the true start-frame, a predetermined scene-cut interval after the true start-frame, a desired stop-frame, a predetermined time interval before the desired stop-frame, a predetermined time interval after the desired stop-frame, a closest scene cut to the desired stop-frame, a predetermined scene-cut interval before the desired stop-frame, and a predetermined scene-cut interval after the desired stop-frame.

According to the present invention, there is provided for the first time a method for detecting and modifying advertisements in media, the method including the steps of: (a) detecting a frame series, in a plurality of frames, that includes an advertisement; and (b) modifying the advertisement, in all frames of the frame series, using a procedure selected from the group consisting of: replacing the advertisement with a substitute advertisement, replacing the advertisement with an small-screen- optimized advertisement, and adding at least one interactive banner to the advertisement.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing graphics parameters in media to be discovered, the computer-readable code including: (a) program code for analyzing a surrounding of a graphics area of an input frame; (b) program code for determining whether the surrounding is smooth; (c) program code for, upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (d) program code for determining whether at least two substantially-distinct colors have been recorded in the surrounding color; and (e) program code for, upon determining at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from at least two substantially-distinct colors and the recorded graphics- area pixel color for each graphics-area pixel location.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing graphics parameters in media to be discovered, the computer-readable code including: (a) program code for analyzing a surrounding of a graphics area of an input frame; (b) program code for determining whether the surrounding is smooth; (c) program code for, upon determining the surrounding is smooth, recording surrounding color and graphics-area pixel color; (d) program code for, for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (e) program code for, for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (f) program code for, choosing a representative value from the surrounding computational parameter; and (g) program code for, for each graphics-area pixel, calculating a color value and an alpha value from the recorded surrounding color and the recorded graphics-area pixel color and from a relation between at least one graphics computational parameter and the representative value.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing media to be adapted for viewing, the computer-readable code including: (a) program code for providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames; (b) program code for, for each graphics-area pixel, checking if the alpha value is transparent in the graphics area; (c) program code for, for each graphics-area pixel for which the alpha value is transparent, replacing a pixel value with a calculated-background value obtained from a relation between the color value and the alpha value; and (d) program code for, for each graphics-area pixel for which the alpha value is not transparent, performing a pixel-modification - procedure. Preferably, the program code for performing the pixel-modification procedure includes program code for an in-painting process.

Preferably, the program code for performing the pixel-modification procedure ^~ includes program code for substituting a combination pixel value from a past-frame pixel value and a future-frame pixel value. According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing graphics disappearance in media to be detected, the computer-readable code including: (a) program code for calculating graphics edges of a graphics element; (b) program code for calculating graphics-area edges of a graphics area in an input frame; (c) program code for removing missing edges not present in at least one previous frame; and (d) program code for, upon determining that a percentage of the graphics edges that are not present in the graphics-area edges is greater than a predetermined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing graphics disappearance in media to be detected, the computer-readable code including: (a) program code for calculating, for each pixel in a graphics area in an input frame, a maximum color- value limit and a minimum color- value limit using a pixel alpha value and a graphics-pixel color value; (b) program code for determining whether each pixel of the input frame is outside the color-value limits; and (c) program code for, upon determining that a pixel amount outside the color- value limits is greater than a pre-determined threshold, designating the graphics element as absent.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing media to be adapted selectively zoomed, the computer-readable code including: (a) program code for checking for a scene cut in an input frame of a plurality of frames which define at least one shot; (b) program code for marking at least one keyframe in at least one shot by designating a keyframe zoom-location and a keyframe zoom-size; and (c) program code for interpolating a respective zoom- location and a respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein at least one keyframe shot includes at least one keyframe.

Preferably, the computer-readable code further includes: (d) program code for de-interlacing the zoomed portion.

Preferably, the computer-readable code further includes: (d) program code for providing a graphics and an alpha for an input frame of a plurality of frames; and (e) program code for inserting a graphics element into a zoomed portion of the input frame, wherein the graphics element has a graphics zoom-location and a graphics zoom-size based on to the graphics and the alpha and a user input.

Preferably, the respective zoom-location and the respective zoom-size are determined by at least one keyframe and at least one tracked object in at least one keyframe shot. According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing graphics to be adapted to a zoomed media frame, the computer-readable code including: (a) program code for defining a graphics area; (b) program code for setting a zoom portion of the media frame in a way that excludes the graphics area; and (c) program code for copying the graphics area to a fixed position relative to the zoomed portion.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing objects to be tracked and highlighted in media, the computer-readable code including: (a) program code for marking an object to highlight in an input frame; (b) program code for tracking the object; and (c) program code for attaching highlighting graphics to the object.

According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing media to be near-live edited, the computer-readable code including: (a) program code for marking a desired start-frame in a plurality of frames; and (b) program code for excerpting a desired frame sequence from the plurality of frames, wherein the desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop-frame according to a predefined stop-frame rule, wherein the predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before the desired start- frame, a predetermined time interval after the desired start-frame, a closest scene cut to the desired start-frame, a predetermined scene-cut interval before the desired start- frame, and a predetermined scene-cut interval after the desired start-frame, and wherein the predefined stop-frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after the true start-frame, a predetermined frame interval after the true start-frame, a predetermined scene-cut interval after the true start-frame, a desired stop-frame, a predetermined time interval before the desired stop-frame, a predetermined time interval after the desired stop-frame, a closest scene cut to the desired stop-frame, a predetermined scene-cut interval before the desired stop-frame, and a predetermined scene-cut interval after the desired stop-frame. According to the present invention, there is provided for the first time a computer-readable storage medium having computer-readable code embodied therein for causing advertisements to be detected and modified in media, the computer- readable code including: (a) program code for detecting a frame series, in a plurality 5 of frames, that includes an advertisement; and (b) program code for modifying the advertisement, in all frames of the frame series, using a procedure selected from the group consisting of: replacing the advertisement with a substitute advertisement, replacing the advertisement with an small-screen-optimized advertisement, and adding at least one interactive banner to the advertisement.

10 These and further embodiments will be apparent from the detailed description and examples that follow.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is herein described, by way of example only, with

15 reference to the accompanying drawings, wherein:

Figure 1 is a simplified flowchart of the general process for graphics detection and removal, according to preferred embodiments of the present invention; Figure 2 is a simplified flowchart of the process steps for a method for graphics removal, according to preferred embodiments of the present

20 invention;

Figure 3 is a simplified flowchart of the process steps for an alternative method for graphics removal, according to preferred embodiments of the : --.-- - present invention;

Figure 4 is a simplified flowchart of the process steps for a method for

25 graphics disappearance detection, according to preferred embodiments of the present invention;

Figure 5 is a simplified flowchart of the process steps for an alternative method for graphics disappearance detection, according to preferred embodiments of the present invention;

30 Figure 6 is a simplified flowchart of the process steps for a method for near- live editing, according to preferred embodiments of the present invention; Figure 7 is a simplified flowchart of the process steps for a method for selective zooming, according to preferred embodiments of the present invention;

Figure 8 is a simplified flowchart of the process steps for a method for object highlighting and tracking, according to preferred embodiments of the present invention; and

Figure 9 is a simplified schematic block diagram of a computing device for enabling a Close VU system, according to preferred embodiments of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates to methods for adapting media for viewing on small display screens. The principles and operation for adapting media for viewing on small display screens, according to the present invention, may be better understood with reference to the accompanying description and the drawings.

Some preferred embodiments relate to CloseVU Live and some preferred embodiments relate to CloseVU Producer. CloseVU Live is intended for repurposing video from a live feed, and CloseVU Producer is intended for repurposing video in post-production. CloseVU Live provides several options for real-time image zoom (i.e. reframing/repurposing video from a live feed), when producing the output for a live mobile-TV stream. CloseVU Live creates a time delay between the input and output, and uses the delay to create smooth camera motion between the reframing points (i.e. keyframes) designated by the user. The delay also allows the operator to fix mistakes made in selecting the keyframe locations.

There are two general approaches to utilizing CloseVU Live. The first approach starts with the user interface. The user interface has three primary windows. The working window enables the user to use a mouse or pre-defined keyboard shortcuts (e.g. upper left and medium size) to zoom in on video action, an output area (displaying the current output), and a buffer area (including a series of view ports representing the state of the video feed at different locations in the buffer). Dedicated control panels can also be used for reframing. A tracking system, using a point-and- click function, can be initiated at the discretion of the editor. Shortcut buttons are available, for example, for the following functions: cutting scenes, canceling cuts, image enhancement, and view-port selection. In addition, a panic button enables to user to mark a full-screen refraining, and produce a smooth camera transition to the reframed image. The user interface also provides the user with buttons to select standard zoom configurations (e.g. small zoom and large zoom).

The second approach starts with Close VU Live selecting several reframing- rectangle locations, and presenting the user the different options (i.e. different virtual cameras) in different preview windows. The user can select the desired view with a keyboard, a touch screen, or other input device. For each virtual camera, there is a timeline with thumbnail images in which the refraining window is depicted in the thumbnail. When the user changes the refraining region location, CloseVU Live uses the delay buffer to smooth the transitioning of the "camera" movement.

An option is available, in both approaches described above, for the user to designate a shot transition in the event that the automatic shot-detection misses a shot. The user can also add "trackers" using CloseVU Live in the same way as with CloseVU Producer (i.e. the post-production tool).

The CloseVU Live interface also allows the user to add a scene cut. After the scene cut is selected, CloseVU Live searches for the best candidate for a scene cut in the vicinity of the frame that the user designated. The CloseVU Live interface also allows the user to remove scene cuts automatically detected by CloseVU Live. The user can change the working-window view to different points inside the buffer by clicking one of the view ports, each view port shows the frames inside a different part of the buffer. By clicking on a view port, all the keyframes that occur after the point in time of the view port are cancelled. This allows the user to correct a mistake (e.g. if the user reframes the wrong area by mistake, the user can choose a view port before the keyframe reaches the reframed image, then reframed image will be deleted).

The CloseVU Live main window, which appears when a video is opened, is divided into three areas: (a) a video working window, (b) a preview window, and (c) a timeline/buffer panel.

(a) The video working window in which the operator can bracket and select the desired zoom level and location of the refraining rectangle. (b) The preview window shows how the reformatted video will appear on the small screen of a mobile device. Next to the preview window is a "full-frame" preview window for displaying the current output frame in an un-cropped fashion (with the cropping rectangle displayed in the window).

(c) The timeline/buffer displays a chronological view of the video stream within the buffer. If keyframes were created, the keyframes will be displayed in their location in the buffer, and will advance on the timeline as the video stream progresses in the buffer. Scene cuts, both automatically detected and manually inserted, are also displayed in a similar fashion, but as distinguished icons. At specific locations in the timeline/buffer, thumbnails (i.e. view ports) of the current frame at the location are displayed below the timeline. The user is able to select different locations as working points. The frame in the currently- selected position is the frame displayed in the video working window.

The timeline displays the reframe region in each thumbnail. The CloseVU Producer main window, which appears when a video is opened, is divided into four areas: (a) a video working window, (b) a preview window, (c) a shotline, and (d) a timeline. (a) The video working window is a resizable working space in which the operator can use a mouse to bracket and select the desired zoom level and location for a mobile video. Object-tracking functions are also a feature of the video working window.

(b) The preview window simulates how the reformatted video will appear on a particular mobile phone or podcast. The size of the preview window varies to match the unique parameters of the various mobile devices.

(c) The shotline displays all detected shots and transitions along a vertical track. Each shot is automatically detected using a shot-detection algorithm. The shot appears with a thumbnail and descriptive data (e.g. beginning, end, duration, and index). (d) The timeline displays a chronological view of the selected-shot sequence. If "keyframes" were created, the timeline displays the reframe region in each thumbnail.

The process of reformatting content for new platforms and devices can require a lot of time, human effort, and equipment resources. Embodiments of the present invention leverage the advantages of file-based digital workflows including acquisition (e.g. file-based digital cameras), production (e.g. video-editing platforms), and distribution (e.g. file-based delivery services). Embodiments of the present invention simplify the production of compelling mobile-TV channels created by a small team of producers.

The workflow to generate mobile-TV content can be streamlined to integrate current television production content into suitable mobile-TV content. The simplified workflow scheme for repurposing television production content into suitable mobile- TV content using CloseVU Producer is as follows. (1) The mobile-TV producer retrieves the files from the station's production system via a network or local repository.

(2) CloseVU Producer imports the files from third-party systems using tools for conversion of HD and SD formats such as DV or MPEG2, and various file wrappers. (3) The mobile-TV producer uses CloseVU to resize frames and focus on the action through close-ups, rather than wide-angle or long shots. In addition, the producer can insert specific mobile-friendly logos or "bugs" for branding that will appear sharper and clearer on small screens. (4) The producer completes the mobile-TV product by exporting the file, and optimally compressing the file into appropriate mobile formats for streaming directly to mobile devices.

CloseVU Producer uses many common aspects of CloseVU Live, but in a post-production scheme. The shots and transitions appear in a shotline in CloseVU Producer. Each shot appears with a thumbnail image, and data describing the shot

(e.g. beginning, end, duration, and index). The clip shots, cuts, and transitions are automatically detected. There are several ways to manually edit the shots and transitions including:

(1) designating two or more shots in the shotline, and clicking on the merge option;

(2) designating the in-point and out-point in the timeline, and clicking on create shot/create transition; and

(3) selecting one or more shots, and clicking on the shot-boundary buttons that appear near the shot edges of the shotline (this will move the shot start and end according to the button pressed).

The process of reframing enables easy and fast repurposing of movie clips to a small-screen device. In each shot, the operator designates the desired reframing- region size and location in the keyframe. CloseVU Producer interpolates the size and position of the reframing region in the rest of the frames automatically in a smooth way that emulates natural camera movement. The operator can also designate objects in a frame, and CloseVU Producer will track those objects and adjust the reframing region in a way that the objects will stay inside the reframing region.

There are two windows in the reframing process: the working window and the preview window. In the working window, there are two rectangles that represent the area to zoom in on (i.e. the reframing region). The first rectangle shows the area that is zoomed in on according to the calculations that were made by CloseVU Producer. The second-rectangle location is changed by moving a mouse (or another input mechanism), and the second- size is changed by using a mouse wheel (e.g. the ratio between the rectangle height and width is fixed and defined according to the device intended to receive the content).

When left mouse-button is clicked, the second rectangle indicates where the desired reframing region is in the designated frame. Such frames are called keyframes. Keyframes are marked on a keyframe bar. Editing and navigating between keyframes are performed by using keyframe buttons and text boxes. The user can also create trackers in the clip. The algorithm calculates the reframing-region size and location in each frame. The algorithm works on each shot separately. ❖ The shots that are selected in the shotline are presented in detail on the timeline. The thumbnails of different frames in the shot are shown in the timeline. In each thumbnail, the reframing region corresponding to that frame is indicated. Clicking on the timeline moves the position of the current frame to the appropriate time, and the images in the working and preview windows update accordingly. Pressing "zoom in/out" buttons increase or decrease the span of the timeline, and hence the number of thumbnails. If the timeline is bigger than the screen width, a horizontal scrollbar will appear. On the timeline, there are tick marks representing time. The display can be changed to show different units (e.g. seconds, frames, and time codes).

Selecting items in the image for tracking is performed by a keyboard keystroke, resulting in a rectangle appearing in the working window. The location and size of the rectangle can be controlled by using a mouse with size being changed using a mouse wheel.

Placing the rectangle on the desired object and clicking will designate the object for tracking. Close VU Producer moves the rectangle according to the tracked object location. If the tracked object moves outside the image, or the tracking algorithm estimates that the tracking quality is not good enough, the tracking will stop. Tracking is made only within a single shot. In the timeline, a line from the start of the tracking to the last frame of the tracking-is displayed in the same color as the tracker's rectangle. In a trackers table, a new line representing the tracker is added. Editing the trackers preferences (e.g. start frame, stop frame, disable/enable status, and color) is performed from the tracking table. There is also an option to add multiple trackers to a shot. There are two modes of multiple tracking: a chaining-mode tracker designation and a stacking-mode tracker designation.

❖ Chaining mode - a new tracker that is added will stop all preceding trackers at the new tracking start-frame. The purpose of this mode is to switch tracking objects in a shot. Chaining mode is user-activated via a keyboard keystroke. ❖ Stacking mode - a new tracker that is added will have no effect on other trackers. The reframing algorithm will try to keep all the trackers inside the refraining rectangle. Stacking mode is user-activated via a keyboard keystroke.

Optionally, a camera-motion detection module works separately on each shot. The projective transformation between the frames in the clip is calculated (e.g. using point matching and line matching). The projective transformations define the camera movement.

Optionally, segmentation is performed by calculating the average and standard deviation of pixels with regard to camera-motion pixels. Camera-motion pixels that have values that are outside some function of the standard deviation are considered foreground and the rest is considered background. Segmentation is also useful for filtering the effects of the camera movement when implementing motion vectors, graphics, and tracking algorithms.

Optionally, a panoramic view of the shot is built using data from the camera motion detection module. The user can select one or more items of interest from the clip, and only those items are shown in the panoramic view. To avoid excessive smearing of the item, the item is shown every few frames. If no selection is made, all items are shown. The user can designate a region in the panoramic view in which the region represents the area of interest. CloseVU Producer uses the areas of interest to calculate the refraining region in each frame. Optionally, a content-based compression module, using tracking information and segmentation techniques, allows the clip size to be considerably reduced without losing quality by applying different levels of compression to different objects in a clip.

The methods of removing graphics, according to preferred embodiments of the present invention, provide several features including:

(1) removing graphics (e.g. logos, game clock, and game score) from the input movie;

(2) improving and relocating graphics; and

(3) detecting when graphics have disappeared. Refraining video content is complicated by the presence of graphics (e.g. logos, clocks, and subtitles) in the video. Referring now to the drawings, Figure 1 is a simplified flowchart of the general process for graphics detection and removal, according to preferred embodiments of the present invention. The process starts with defining the graphics area (Step 10). The software can then optionally check if the graphics have disappeared (Step 12). If the graphics have disappeared, then the software does not do anything the frame being checked (Step 14). If the graphics have not disappeared, then the software removes the graphics from the frame by either cropping or using methods A or B (described below) (Step 16). The software then reinserts optimized graphics into the frame (Step 18).

Figure 2 is a simplified flowchart of the process steps for a method for graphics removal, according to preferred embodiments of the present invention. The process starts with defining the graphics area (Step 20), and then moves to the next frame (Step 22). The software checks if the graphics have been calculated (Step 24). If the graphics have not been calculated, the software analyzes the surrounding of graphics area (Step 26). In order to remove graphics, a user marks the area around the graphics (usually a bounding box), then the software starts gathering information about the surroundings of the graphics (a small area surrounding the area that the user marked).

The software first checks if the surrounding is smooth (Step 28). If the surrounding is smooth, then the software records the surrounding color and the graphics-area pixel colors (Step 30). If not, the process returns to Step 22. The software then checks if the colors are close to previously-found colors (Step 32). If so, the process returns to Step 22. If not, the software increases the number of found colors (Step 34). The software then checks if two distinct surrounding colors are detected (Step 36). If so, the software then calculates the graphics and alpha from the two distinct colors (Step 38). If not, the process returns to Step 22. The equation for the graphics is assumed to be P = (l-a)*G+a*B, where P is the pixel color in the image, a is the transparency ratio (α = 0 when graphics are opaque, and a = 1 when graphics are transparent), and B is the pixel color if there were no graphics (background). When the surrounding area has the same color (up to some threshold), it is assumed that the area "behind" the graphics has the same color (i.e. B of each pixel is equal to P of surrounding pixels). Once such a situation occurs for two different colors, two independent equations can be obtained having two variables (A, alpha, and G, graphics) for each pixel, enabling the equations to be solved for the variables.

Once the graphics, G, have been calculated in Step 38, the process returns to Step 22, and continues to Step 24. For each pixel, the software checks if alpha, A, is transparent in the graphics (Step 40). If so, the software replaces the pixel value with the background (surrounding) value (Step 42). If alpha is not transparent in the graphics in Step 40, the software marks the pixel location in a mask {i.e. a list of selected pixels) (Step 46). The software then performs "in-painting" {i.e. well-known algorithms that obtain information from the surrounding pixels), or acquires the pixel value from past/future frames on all pixels that are marked in the mask (Step 44). The process then returns to Step 22.

Another way to calculate A and G is to use image statistics. Figure 3 is a simplified flowchart of the process steps for an alternative method for graphics removal, according to preferred embodiments of the present invention. The process starts with defining the graphics area (Step 50), and then moves to the next frame (Step 52). The software checks if the graphics have been calculated (Step 54). If the graphics have not been calculated, the software analyzes the surrounding of graphics area (Step 56). The software then checks if the surrounding is smooth (Step 58). If the surrounding is smooth, then the software records the surrounding color and the graphics-area pixel colors (Step 60). The software then checks if the colors are close to previously-found colors (Step 62). If not, the software increases the number of found colors (Step 64).

If the surrounding is not smooth in Step 58, or if the colors are close to previously-found colors in Step 62, then for each pixel in the surrounding and in the graphics area, the software calculates the standard deviation or derivative over time S (Step 66). The "near" pixels are assumed to have a similar distribution (over time), allowing alpha to be calculated the using pixel statistics. Also, following Step 64, the process continues to Step 66. There are two alternative statistics that can be used as follows: (1) standard deviation (of pixel values over time):

(2) derivative (of pixel values over time): avg[abs(p_ou,*dp/dt] = Λf(avg[absO,_n*dp/dt], where p_out denotes a pixel outside graphic area while p, and a, denote a pixel and matching alpha within the graphics area.

Once two independent equations are obtained (two colors or one color and alpha), B can be calculated for each pixel (which is the value of the pixel with no graphics). This is the core process, heuristics can also be applied to the results such as smoothing or "thresholding" (e.g. if a is small, it is rounded to 0, if a is close to 1, it is rounded to 1).

The process continues with the software calculating, from a calculated S, for each pixel in the surrounding of the graphics, a surrounding "pixel S" (by averaging or picking the most probable) (Step 68). The software then checks if there is at least one smooth surrounding color, and has a common S been stable for the past few frames (Step 70). If so, then the software calculates, for each pixel in the graphics area, graphics and alpha from the smooth color and the relation between the common S and the pixel S (Step 72), and returns to Step 52. If not, the process returns to Step 52.

Once the graphics, G, have been calculated in Step 72, the process returns to Step 52, and continues to Step 54. For each pixel, the software checks if alpha, A, is transparent in the graphics (Step 74). If so, the software replaces the pixel value with the background (surrounding) value (Step 76). If alpha is not transparent in the graphics in Step 54, the software marks the pixel location in a mask (Step 78). The software then performs in-painting, or acquires the pixel value from past/future frames on all pixels that are marked in the mask (Step 80). The process then returns to Step 52. For pixels where a = 0 (i.e. there is no information about the background), B must be interpolated. This can be performed by in-painting, or by acquiring the pixel value from past/future frames (Step 80). Since the camera motion can be detected, the background pixel can be found from a frame that is not hidden by graphics. In case there are too many opaque pixels and the quality of the graphics removal is not adequate, the original image can be cropped so that the area with the graphics is removed. Once the graphics have been obtained, the graphics can be enlarged and repositioned inside the working window in order to keep the graphics in the same position relative to the zoomed or reframed/repurposed window. The graphics can also be enlarged and smoothed in order to make the graphics have a better appearance after encoding on a small screen.

The software also detects when the graphics have "disappeared" from an original clip. This usually occurs in replays or when cutting into a commercial. When graphics have disappeared, the graphics removal algorithm is paused. Graphics disappearance detection is performed in either one of the two following ways. (1) Edges of the graphics are calculated and compared to edges in the input that are persistent (i.e. edges that do not change for a period of time). If there are not enough matches for a period of time, the graphic are designated as disappeared.

(2) Since the alpha and graphics have been calculated, the legal range of pixel values within the graphics area is known (assuming the graphics exists). In cases in which many pixels are out of the legal range, the graphic are designated as disappeared (i.e. without graphics, each pixel can be in a range of 0-255 in each color channel; whereas with the graphics, this range is limited (for larger values of a, the dependence is stronger). If a large amount of pixels are detected outside the allowed range, the graphic are designated as disappeared.

Figure 4 is a simplified flowchart of the process steps for a method for graphics disappearance detection, according to preferred embodiments of the present invention. The process starts with the software calculating the edges of the graphics, designating the edges as E_g (Step 90), and moving to the next frame (Step 92). The software then calculates the edges in the graphics area of the input frame, and designates the edges as E₁ (Step 94).

The software removes the edges from Ej that were not present in previous N frames (Step 96), and calculates the percentage of edges in E_g that are not in E, (Step 98). The software checks if the percentage is high (i.e. higher than a threshold) (Step

100). If so, then the graphics have disappeared (Step 102), and the process returns to Step 92. If not, then the graphics have not disappeared (Step 104), and the process returns to Step 92.

Figure 5 is a simplified flowchart of the process steps for an alternative method for graphics disappearance detection, according to preferred embodiments of the present invention. The process starts with the software calculating, for each pixel in the graphics, the maximum and minimum color- value limit using the pixel's alpha value and graphics-pixel color value (Step 110). The software then moves to the next frame (Step 112).

The software then checks the pixels of the input frame for pixels that are outside their corresponding color limit (Step 114). The software determines if the amount of pixels outside their corresponding color limit is high (i.e. higher than a threshold) (Step 116). If so, then the graphics have disappeared (Step 118), and the process returns to Step 112. If not, then the graphics have not disappeared (Step 120), and the process returns to Step 112. The software also allows the user of CloseVU Live to edit highlights of a game in real time, and to attach metadata to the highlights. This is referred to as near- live editing. When the operator sees a highlight he/she wants to record (e.g. a goal), the operator presses a record button. The software looks for a scene cut in the buffer (in the immediate past of the frame position), and starts recording from the scene cut. Rules can be defined for starting the recording (e.g. how much time in the past to look for the scene cut, and how many scene cuts to look for). The number of scene cuts to wait before stopping the recording can be defined, or the software can wait for the operator to stop the recording manually (the software will look for the nearest scene cut according to predefined rules). The user can also start/stop a recording by clicking on a scene-cut symbol in the timeline, and the software will start/stop recording at that point. For each recorded clip, the user can add metadata (e.g. type, time, player name, and free text). The result of near-live editing is a series of clips that show the highlights of a game with metadata describing each clip.

Figure 6 is a simplified flowchart of the process steps for a method for near- live editing, according to preferred embodiments of the present invention. The process starts with a user identifying an interesting segment in a clip, and informing the software (Step 130). The software looks for a scene cut in the buffer according to predefined rules (e.g. find the first scene cut in the last 4 seconds) (Step 132). The software stops recording according to a predefined rule (e.g. after 10 seconds or after 5 shots), or via a user input (e.g. user marks that the event is over, software looks for the appropriate scene cut according to predefine rules) (Step 134). Another option is to use the software to control the highlights remotely. The operator watches one or more games simultaneously, and using a control panel he/she can send the start/stop recording indications to the appropriate system running the near-live editing software. For example, the operator watches four games on a split screen (or four screens). When there is a goal in game #3, the operator uses control software to start recording on system #3. The control software sends a signal to the system #3 to start recording. This process is referred to as cluster near-live editing.

Some problems arise when zooming in on parts of an image. One problem is caused by interlaced images; interlaced images looks bad at high zoom. Another problem is caused by blurry images; blurry images look more blurry at high zoom. A de-interlacing algorithm is used to improve image quality. The image-enhancement techniques can be applied either before or after the image has been refrained.

Selective zooming is employed primarily for 2 reasons: (1) to make things bigger, and/or (2) to improve encoding quality. Regarding (1), movies that were intended for normal TV do not have good appearances when viewed on small screens (e.g. mobile phones). The screen is too small and usually its resolution is smaller than a regular TV set. Because of these factors, a relevant area of a clip is cropped and enlarged. Regarding (2), in events recorded in stadiums or arenas, the stadium audience is the hardest part of a clip to encode. When selective zooming is employed by zooming-in on the players, the average area of the crowd is reduced in the output clip, making the clip easier to encode (e.g. higher quality or smaller bandwidth).

The user marks the desired zoom (location and magnification) by moving a rectangle that can change its size (the rectangle width/height ratio is constant and determined by the output-signal desired resolution). A frame where the user has selected a zoom is called a keyframe. The software interpolates between the keyframes so the user does not have to insert data for every frame in the clip.

The software performs an auto-cut detection (the user can add manual cuts or delete auto-detected cuts as well), and uses the data to limit the interpolation. There is no interpolation between keyframes that are present in two different shots. When there are no keyframes in a shot, the minimum zoom (i.e. minimum magnification) is selected. Interpolation is performed:

( 1 ) using the user keyframes; (2) using user keyframes and data from tracking players in which the user marks one or more objects (e.g. players and ball) to track, and the software tries to keep the objects in the frame (and tries to keep the interpolation as smooth as possible); or

(3) interpolation using keyframes, camera motion, and/or tracking in which the software calculates the original camera motion, tracks objects in the clip, and decides how to interpolate between the keyframes based on that data.

When interlaced clips are zoomed, the resulting output clip looks poor - each field becomes bigger, and the image looks jagged. To avoid this from happening, the incoming clip is de-interlaced. In the CloseVU Live application, a delay is created between the input signal and the output signal. The frames are inserted into a FIFO queue. The delay is used for two reasons: (1) to have time to perform the interpolation, and (2) to have a chance to fix mistakes.

Regarding (1), for example, if the user makes a keyframe at frame 1 and then another keyframe at frame 100, then the keyframe at frame 100 affects frames 2-99 because of the interpolation. Obviously, this cannot be rectified if the frames were already sent to the output card. The delay circumvents such a problem. Every frame that is sent to the output card is considered a keyframe because the frame cannot be changed anymore; interpolation is performed only on the keyframes in the buffer. Regarding (2), for example, if there is a 5-second buffer and the user makes a mistake (e.g. marks zoom on the wrong area), the user has five seconds to fix the mistake. The user can move inside the buffer (i.e. "move in time" of the clip), and perform reframing in places that were initially missed. To avoid confusion, the output is shown in a separate window from the input with the zoom marker indicated. Figure 7 is a simplified flowchart of the process steps for a method for selective zooming, according to preferred embodiments of the present invention. The process starts when a frame entering the buffer (Step 140). The software checks for a scene cut (Step 142). The user marks the keyframes on selected frames (Step 144).

The software interpolates the zoom location and size in frames according to the user input and scene cuts (Step 146). The software de-interlace the input video (Step 148), and the clip is zoomed (Step 150). It is typically hard to view small items (e.g. a soccer, tennis, or golf ball) on small screens. Designating small and/or fast-moving objects like balls in sports events makes it easier to "follow" the ball on the small screens mobile devices. Upon selection of a specific displayed object in a video sequence (at the beginning and end) using CloseVU Producer, the operator is able to choose a highlighting color (or other visual effects) to be applied to the object (e.g. trail, virtual zoom, markers, and arrows.

For high-quality results, these effects can be rendered directly on the mobile devices. Figure 8 is a simplified flowchart of the process steps for a method for object highlighting and tracking, according to preferred embodiments of the present invention. The process starts when a user marks an object to highlight (Step 160). The user chooses the highlight type (e.g. trail or graphics enhancement) (Step 162). The software tracks the object, and adds highlight to the object by changing the clip or by saving metadata in a separate file/stream (Step 164).

Figure 9 is a simplified schematic block diagram of a computing device for enabling a CloseVU system, according to preferred embodiments of the present invention. A computing device 170 is shown having a CPU 172 and a memory 174. A

CloseVU system 176, residing in memory 174, enables processing of media frames as described above.

While the invention has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications, and other applications of the invention may be made.

Claims

WHAT IS CLAIMED IS:

1. A computing device for calculating graphics parameters in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(b) a memory for handling data resulting from said computing operations, said memory including program code for:

(i) analyzing a surrounding of a graphics area of an input frame;

(ii) determining whether said surrounding is smooth;

(iii) upon determining said surrounding is smooth, recording surrounding color and graphics-area pixel color;

(iv) determining whether at least two substantially-distinct colors have been recorded in said surrounding color; and

(v) upon determining said at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from said at least two substantially-distinct colors and said recorded graphics-area pixel color for each graphics-area pixel location.

2. A computing device for calculating graphics parameters in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) analyzing a surrounding of a graphics area of an input frame;

(ii) determining whether said surrounding is smooth;

(iii) upon determining said surrounding is smooth, recording surrounding color and graphics-area pixel color; (iv) for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time; (v) for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time;

(vi) choosing a representative value from said surrounding computational parameter; and

(vii) for each graphics-area pixel, calculating a color value and an alpha value from said recorded surrounding color and said recorded graphics-area pixel color and from a relation between said at least one graphics computational parameter and said representative value.

3. A computing device for adapting media for viewing, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames;

(ii) for each graphics-area pixel, checking if said alpha value is transparent in said graphics area;

(iii) for each said graphics-area pixel for which said alpha value is transparent, replacing a pixel value with a calculated- background value obtained from a relation between said color value and said alpha value; and

(iv) for each said graphics-area pixel for which said alpha value is not transparent, performing a pixel-modification procedure.

4. The device of claim 3, wherein said program code for performing said pixel-modification procedure includes program code for an in-painting process.

5. The device of claim 3, wherein said program code for performing said pixel-modification procedure includes said program code for substituting a combination pixel value from a past- frame pixel value and a future-frame pixel value.

6. A computing device for detecting graphics disappearance in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) calculating graphics edges of a graphics element;

(ii) calculating graphics-area edges of a graphics area in an input frame;

(iii) removing missing edges not present in at least one previous frame; and

(iv) upon determining that a percentage of said graphics edges that are not present in said graphics-area^{^} edges is greater than a predetermined threshold, designating said graphics element as absent.

7. A computing device for detecting graphics disappearance in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) calculating, for each pixel in a graphics area in an input frame, a maximum color-value limit and a minimum color-value limit using a pixel alpha value and a graphics-pixel color value;

(ii) determining whether each said pixel of said input frame is outside said color-value limits; and

(iii) upon determining that a pixel amount outside said color-value limits is greater than a pre-determined threshold, designating said graphics element as absent.

8. A computing device for selective zooming of media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) checking for a scene cut in an input frame of a plurality of frames which define at least one shot;

(ii) marking at least one keyframe in said at least one shot by designating a keyframe zoom-location and a keyframe zoom- size; and

(iii) interpolating a respective zoom-location and a respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein said at least one keyframe shot includes said at least one keyframe.

9. The device of claim 8, wherein said memory further includes program code for:

(iv) de-interlacing said zoomed portion.

10. The device of claim 8, wherein said memory further includes program code for:

(iv) providing a graphics and an alpha for an input frame of a plurality of frames; and

(v) inserting a graphics element into a zoomed portion of said input frame, wherein said graphics element has a graphics zoom- location and a graphics zoom-size based on to said graphics and said alpha and a user input.

11. The device of claim 8, wherein said respective zoom-location and said respective zoom-size are determined by said at least one keyframe and at least one tracked object in said at least one keyframe shot.

12. A computing device for adapting graphics to a zoomed media frame, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) defining a graphics area;

(ii) setting a zoom portion of the media frame in a way that excludes said graphics area; and (iii) copying said graphics area to a fixed position relative to said zoomed portion.

13. A computing device for object tracking and highlighting in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) marking an object to highlight in an input frame;

(ii) tracking said object; and

(iii) attaching highlighting graphics to said object.

14. A computing device for near-live editing of media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) marking a desired start-frame in a plurality of frames; and (ii) excerpting a desired frame sequence from said plurality of frames, wherein said desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop-frame according to a predefined stop-frame rule, wherein said predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before said desired start-frame, a predetermined time interval after said desired start-frame, a closest scene cut to said desired start-frame, a predetermined scene-cut interval before said desired start-frame, and a predetermined scene-cut interval after said desired start-frame, and wherein said predefined stop- frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after said true start- frame, a predetermined frame interval after said true start- frame, a predetermined scene-cut interval after said true start- frame, a desired stop-frame, a predetermined time interval before said desired stop-frame, a predetermined time interval after said desired stop-frame, a closest scene cut to said desired stop-frame, a predetermined scene-cut interval before said desired stop-frame, and a predetermined scene-cut interval after said desired stop-frame.

15. A computing device for detecting and modifying advertisements in media, the device comprising:

(a) a central processor unit for performing computing operations; and

(i) detecting a frame series, in a plurality of frames, that includes an advertisement; and

(ii) modifying said advertisement, in all frames of said frame series, using a procedure selected from the group consisting of: replacing said advertisement with a substitute advertisement, replacing said advertisement with an small-screen-optimized advertisement, and adding at least one interactive banner to said advertisement.

16. A method for calculating graphics parameters in media, the method comprising the steps of:

(a) analyzing a surrounding of a graphics area of an input frame;

(b) determining whether said surrounding is smooth;

(c) upon determining said surrounding is smooth, recording surrounding color and graphics-area pixel color;

(d) determining whether at least two substantially-distinct colors have been recorded in said surrounding color; and

(e) upon determining said at least two substantially-distinct colors have been recorded, calculating a color value and an alpha value from said at least two substantially-distinct colors and said recorded graphics- area pixel color for each graphics-area pixel location.

17. A method for calculating graphics parameters in media, the method comprising the steps of:

(a) analyzing a surrounding of a graphics area of an input frame;

(b) determining whether said surrounding is smooth;

(d) for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time;

(e) for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time;

(f) choosing a representative value from said surrounding computational parameter; and

(g) for each graphics-area pixel, calculating a color value and an alpha value from said recorded surrounding color and said recorded graphics-area pixel color and from a relation between said at least one graphics computational parameter and said representative value.

18. A method for adapting media for viewing, the method comprising the steps of:

(a) providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames;

(b) for each graphics-area pixel, checking if said alpha value is transparent in said graphics area;

(c) for each said graphics-area pixel for which said alpha value is transparent, replacing a pixel value with a calculated-background value obtained from a relation between said color value and said alpha value; and

(d) for each said graphics-area pixel for which said alpha value is not transparent, performing a pixel-modification procedure.

19. The method of claim 18, wherein said pixel-modification procedure includes an in-painting process.

20. The method of claim 18, wherein said pixel-modification procedure includes substituting a combination pixel value from a past-frame pixel value and a future-frame pixel value.

21. A method for detecting graphics disappearance in media, the method comprising the steps of:

(a) calculating graphics edges of a graphics element;

(b) calculating graphics-area edges of a graphics area in an input frame;

(c) removing missing edges not present in at least one previous frame; and

(d) upon determining that a percentage of said graphics edges that are not present in said graphics-area edges is greater than a pre-determined threshold, designating said graphics element as absent.

22. A method for detecting graphics disappearance in media, the method comprising the steps of: (a) calculating, for each pixel in a graphics area in an input frame, a maximum color-value limit and a minimum color-value limit using a pixel alpha value and a graphics-pixel color value;

(b) determining whether each said pixel of said input frame is outside said color-value limits; and

(c) upon determining that a pixel amount outside said color-value limits is greater than a pre-determined threshold, designating said graphics element as absent.

23. A method for selective zooming of media, the method comprising the steps of:

(a) checking for a scene cut in an input frame of a plurality of frames which define at least one shot;

(b) marking at least one keyframe in said at least one shot by designating a keyframe zoom-location and a keyframe zoom-size; and

(c) interpolating a respective zoom-location and a respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein said at least one keyframe shot includes said at least one keyframe.

24. The method of claim 23, the method further comprising the step of:

(d) de-interlacing said zoomed portion.

25. The method of claim 23, the method further comprising the steps of:

(d) providing a graphics and an alpha for an input frame of a plurality of frames; and

(e) inserting a graphics element into a zoomed portion of said input frame, wherein said graphics element has a graphics zoom-location and a graphics zoom-size based on to said graphics and said alpha and a user input.

26. The method of claim 23, wherein said respective zoom-location and said respective zoom-size are determined by said at least one keyframe and at least one tracked object in said at least one keyframe shot.

27. A method for adapting graphics to a zoomed media frame, the method comprising the steps of:

(a) defining a graphics area;

(b) setting a zoom portion of the media frame in a way that excludes said graphics area; and

(c) copying said graphics area to a fixed position relative to said zoomed portion.

28. A method for object tracking and highlighting in media, 'the method comprising the steps of:

(a) marking an object to highlight in an input frame;

(b) tracking said object; and

(c) attaching highlighting graphics to said object.

29. A method for near-live editing of media, the method comprising the steps of:

(a) marking a desired start-frame in a plurality of frames; and -

(b) excerpting a desired frame sequence from said plurality of frames, wherein said desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop-frame according to a predefined stop-frame rule, wherein said predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before said desired start-frame, a predetermined time interval after said desired start-frame, a closest scene cut to said desired start-frame, a predetermined scene-cut interval before said desired start-frame, and a predetermined scene-cut interval after said desired start-frame, and wherein said predefined stop-frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after said true start-frame, a predetermined frame interval after said true start-frame, a predetermined scene-cut interval after said true start-frame, a desired stop-frame, a predetermined time interval before said desired stop- frame, a predetermined time interval after said desired stop-frame, a closest scene cut to said desired stop-frame, a predetermined scene-cut interval before said desired stop-frame, and a predetermined scene-cut interval after said desired stop-frame.

30. A method for detecting and modifying advertisements in media, the method comprising the steps of:

(a) detecting a frame series, in a plurality of frames, that includes an advertisement; and

(b) modifying said advertisement, in all frames of said frame series, using a procedure selected from the group consisting of: replacing said advertisement with a substitute advertisement, replacing said advertisement with an small-screen-optimized advertisement, and adding at least one interactive banner to said advertisement.

31. A computer-readable storage medium having computer-readable code embodied therein for causing graphics parameters in media to be discovered, the computer-readable code comprising:

(a) program code for analyzing a surrounding of a graphics area of an input frame;

(b) program code for determining whether said surrounding is smooth;

(c) program code for, upon determining said surrounding is smooth, recording surrounding color and graphics-area pixel color;

(d) program code for determining whether at least two substantially- distinct colors have been recorded in said surrounding color; and

(e) program code for, upon determining said at least two substantially- distinct colors have been recorded, calculating a color value and an alpha value from said at least two substantially-distinct colors and said recorded graphics-area pixel color for each graphics-area pixel location.

32. A computer-readable storage medium having computer-readable code embodied therein for causing graphics parameters in media to be discovered, the computer-readable code comprising:

(b) program code for determining whether said surrounding is smooth;

(d) program code for, for each graphics-area pixel, calculating at least one graphics computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time;

(e) program code for, for each surrounding-area pixel, calculating at least one surrounding computational parameter selected from the group consisting of: a standard deviation over time and a time derivative averaged over time;

(f) program code for, choosing a representative value from said surrounding computational parameter; and

(g) program code for, for each graphics-area pixel, calculating a color value and an alpha value from said recorded surrounding color and said recorded graphics-area pixel color and from a relation between said at least one graphics computational parameter and said representative value.

33. A computer-readable storage medium having computer-readable code embodied therein for causing media to be adapted for viewing, the computer-readable code comprising:

(a) program code for providing a color value and an alpha value for a graphics area of an input frame of a plurality of frames; (b) program code for, for each graphics-area pixel, checking if said alpha value is transparent in said graphics area;

(c) program code for, for each said graphics-area pixel for which said alpha value is transparent, replacing a pixel value with a calculated- background value obtained from a relation between said color value and said alpha value; and

(d) program code for, for each said graphics-area pixel for which said alpha value is not transparent, performing a pixel-modification procedure.

34. The computer-readable storage medium of claim 33, wherein said program code for performing said pixel-modification procedure includes program code for an in-painting process.

35. The computer-readable storage medium of claim 33, wherein said program code for performing said pixel-modification procedure includes program code for substituting a combination pixel value from a past-frame pixel value and a future-frame pixel value.

36. A computer-readable storage medium having computer-readable code embodied therein for causing graphics disappearance in media to be detected, the computer-readable code comprising:

(a) program code for calculating graphics edges of a graphics element;

(b) program code for calculating graphics-area edges of a graphics area in an input frame;

(c) program code for removing missing edges not present in at least one previous frame; and

(d) program code for, upon determining that a percentage of said graphics edges that are not present in said graphics-area edges is greater than a pre-determined threshold, designating said graphics element as absent.

37. A computer-readable storage medium having computer-readable code embodied therein for causing graphics disappearance in media to be detected, the computer-readable code comprising:

(a) program code for calculating, for each pixel in a graphics area in an input frame, a maximum color-value limit and a minimum color- value limit using a pixel alpha value and a graphics-pixel color value;

(b) program code for determining whether each said pixel of said input frame is outside said color- value limits; and

(c) program code for, upon determining that a pixel amount outside said color-value limits is greater than a pre-determined threshold, designating said graphics element as absent.

38. A computer-readable storage medium having computer-readable code embodied therein for causing media to be selectively zoomed, the computer-readable code comprising:

(a) program code for checking for a scene cut in an input frame of a plurality of frames which define at least one shot;

(b) program code for marking at least one keyframe in said at least one shot by designating a keyframe zoom-location and a keyframe zoom- size; and

(c) program code for interpolating a respective zoom-location and a. respective zoom-size for a zoomed portion for all keyframe-shot frames in at least one keyframe shot, wherein said at least one keyframe shot includes said at least one keyframe.

39. The computer-readable storage medium of claim 38, the computer- readable code further comprising:

(d) program code for de-interlacing said zoomed portion.

40. The computer-readable storage medium of claim 38, the computer- readable code further comprising: (d) program code for providing a graphics and an alpha for an input frame of a plurality of frames; and

(e) program code for inserting a graphics element into a zoomed portion of said input frame, wherein said graphics element has a graphics zoom- location and a graphics zoom-size based on to said graphics and said alpha and a user input.

41. The computer-readable storage medium of claim 38, wherein said respective zoom-location and said respective zoom-size are determined by said at least one keyframe and at least one tracked object in said at least one keyframe shot.

42. A computer-readable storage medium having computer-readable code embodied therein for causing graphics to be adapted to a zoomed media frame, the computer-readable code comprising:

(a) program code for defining a graphics area;

(b) program code for setting a zoom portion of the media frame in a way that excludes said graphics area; and

(c) program code for copying said graphics area to a fixed position relative to said zoomed portion.

43. A computer-readable storage medium having computer-readable code embodied therein for causing objects to be tracked and highlighted in media, the computer-readable code comprising:

(a) program code for marking an object to highlight in an input frame;

(b) program code for tracking said object; and

(c) program code for attaching highlighting graphics to said object.

44. A computer-readable storage medium having computer-readable code embodied therein for causing media to be near-live edited, the computer-readable code comprising:

(a) program code for marking a desired start-frame in a plurality of frames; and (b) program code for excerpting a desired frame sequence from said plurality of frames, wherein said desired frame sequence starts at a true start-frame according to a predefined start-frame rule and ends at a true stop-frame according to a predefined stop-frame rule, wherein said predefined start-frame rule is a start-frame rule selected from the group consisting of: a predetermined time interval before said desired start- frame, a predetermined time interval after said desired start-frame, a closest scene cut to said desired start-frame, a predetermined scene-cut interval before said desired start-frame, and a predetermined scene-cut interval after said desired start-frame, and wherein said predefined stop-frame rule is a stop-frame rule selected from the group consisting of: a predetermined time interval after said true start-frame, a predetermined frame interval after said true start-frame, a predetermined scene-cut interval after said true start-frame, a desired stop-frame, a predetermined time interval before said desired stop- frame, a predetermined time interval after said desired stop-frame, a closest scene cut to said desired stop-frame, a predetermined scene-cut interval before said desired stop-frame, and a predetermined scene-cut interval after said desired stop-frame.

45. A computer-readable storage medium having computer-readable code embodied therein for causing advertisements to be detected and modified in media, the computer-readable code comprising:

(a) program code for detecting a frame series, in a plurality of frames, that includes an advertisement; and

(b) program code for modifying said advertisement, in all frames of said frame series, using a procedure selected from the group consisting of: replacing said advertisement with a substitute advertisement, replacing said advertisement with an small-screen-optimized advertisement, and adding at least one interactive banner to said advertisement.