WO2009126258A1 - System and method for enhancing the visibility of an object in a digital picture - Google Patents
System and method for enhancing the visibility of an object in a digital picture Download PDFInfo
- Publication number
- WO2009126258A1 WO2009126258A1 PCT/US2009/002173 US2009002173W WO2009126258A1 WO 2009126258 A1 WO2009126258 A1 WO 2009126258A1 US 2009002173 W US2009002173 W US 2009002173W WO 2009126258 A1 WO2009126258 A1 WO 2009126258A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- digital picture
- video
- enhancing
- visibility
- localization information
- Prior art date
Links
- 230000002708 enhancing effect Effects 0.000 title claims description 31
- 238000000034 method Methods 0.000 title claims description 27
- 230000004807 localization Effects 0.000 claims abstract description 68
- 238000007670 refining Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 8
- 238000013459 approach Methods 0.000 description 20
- 230000015556 catabolic process Effects 0.000 description 7
- 238000006731 degradation reaction Methods 0.000 description 7
- 238000012805 post-processing Methods 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 6
- 238000010276 construction Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000009499 grossing Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 241000270295 Serpentes Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000593 degrading effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009408 flooring Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000009738 saturating Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
Classifications
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration by the use of local operators
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
- A63B24/00—Electric or electronic controls for exercising apparatus of preceding groups; Controlling or monitoring of exercises, sportive games, training or athletic performances
- A63B24/0021—Tracking a path or terminating locations
- A63B2024/0028—Tracking the path of an object, e.g. a ball inside a soccer pitch
- A63B2024/0034—Tracking the path of an object, e.g. a ball inside a soccer pitch during flight
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30221—Sports video; Sports image
- G06T2207/30224—Ball; Puck
Definitions
- the present invention relates, in general, to the transmission of digital pictures and, in particular, to enhancing the visibility of objects of interest in digital pictures, especially digital pictures that are displayed in units that have low resolution, low bit rate video coding.
- the visibility of an object of interest in a digital image is enhanced, given the approximate location and size of the object in the image, or the visibility of the object is enhanced after refinement of the approximate location and size of the object.
- Object enhancement provides at least two benefits. First, object enhancement makes the object easier to see and follow, thereby improving the user experience. Second, object enhancement helps the object sustain less degradation during the encoding (i.e., compression) stage.
- One main application of the present invention is video delivery to handheld devices, such as cell phones and PDA's, but the features, concepts, and implementations of the present invention also may be useful for a variety of other applications, contexts, and environments, including, for example, video over internet protocol (low bit rate, standard definition content).
- video over internet protocol low bit rate, standard definition content
- the present invention provides for highlighting objects of interest in video to improve the subjective visual quality of low resolution, low bit rate video.
- the inventive system and method are able to handle objects of different characteristics and operate in fully-automatic, semi-automatic (i.e., manually assisted), and full manual modes. Enhancement of objects can be performed at a pre-processing stage (i.e., before or in the video encoding stage) or at a postprocessing stage (i.e., after the video decoding stage).
- the visibility of an object in a digital picture is enhanced by providing an input video containing an object, storing information representative of the nature and characteristics of the object, and developing, in response to the video input and the information representative of the nature and characteristics of the object, object localization information that identifies and locates the object.
- An enhanced video of that portion of the input video that contains the object and the region in which the object is located is developed from the input video in response to the object localization information and the enhanced video is encoded.
- Figure 1 is a block diagram of a preferred embodiment of a system for enhancing the visibility of an object in a digital video constructed in accordance with the present invention.
- Figure 2 illustrates approximate object localization provided by the Figure 1 system.
- FIGS 3A through 3D illustrate the work-flow in object enhancement in accordance with the present invention.
- Figure 4 is a flowchart for an object boundary estimation algorithm that can be used to refine object identification information and object location information in accordance with the present invention.
- Figures 5A through 5D illustrate the implementation of the concept of level set estimation of boundaries of arbitrarily shaped objects in accordance with the present invention.
- Figure 6 is a flowchart for an object enlargement algorithm in accordance with the present invention.
- Figures 7A through 7C illustrate three possible sub-divisions of a 16x16 macroblock useful in explaining the refinement of object identification information and object location information during the encoding stage.
- an object enhancing system constructed in accordance with the present invention, may span all the components in a transmitter 10, or the object enhancement component may be in a receiver 20.
- object highlighting may be performed: (1 ) pre-processing where the object is enhanced in transmitter 10 prior to the encoding (i.e., compression) stage; (2) encoding where the region of interest that contains the object is given special treatment in transmitter 10 by the refinement of information about the object and its location; and (3) postprocessing where the object is enhanced in receiver 20 after decoding utilizing side-information about the object and its location transmitted from transmitter 10 through the bitstream as metadata.
- An object enhancing system, constructed in accordance with the present invention can be arranged to provide object highlighting in only one of the stages identified above, or in two of the stages identified above, or in all three stages identified above.
- the Figure 1 system for enhancing the visibility of an object in a digital picture includes means for providing an input video containing an object of interest.
- the source of the digital picture that contains the object, the visibility of which is to be enhanced, can be a television camera of conventional construction and operation and is represented by an arrow 12.
- the Figure 1 system also includes means for storing information representative of the nature and characteristics of the object of interest (e.g., an object template) and developing, in response to the video input and the information representative of the nature and characteristics of the object, object localization information that identifies and locates the object.
- Such means, identified in Figure 1 as an object localization module 14, include means for scanning the input video, on a frame-by-frame basis, to identify the object (i.e., what is the object) and locate that object (i.e., where is the object) in the picture having the nature and characteristics similar to the stored information representative of the nature and characteristics of the object of interest.
- Object localization module 14 can be a unit of conventional construction and operation that scans the digital picture of the input video on a frame-by-frame basis and compares sectors of the digital picture of the input video that are scanned with the stored information representative of the nature and characteristics of the object of interest to identify and locate, by grid coordinates of the digital picture, the object of interest when the information developed from the scan of a particular sector is similar to the stored information representative of the nature and characteristics of the object.
- object localization module 14 implements one or more of the following methods in identifying and locating an object of interest:
- Object tracking The goal of an object tracker is to locate a moving object in a video.
- a tracker estimates the object parameters (e.g. location, size) in the current frame, given the history of the moving object from the previous frames.
- Tracking approaches may be based on, for example, template matching, optical flow, Kalman filters, mean shift analysis, hidden Markov models, and particle filters.
- Object detection The goal in object detection is to detect the presence and location of an object in images or video frames based on prior knowledge about the object.
- Object detection methods generally employ a combination of top-down and bottom-up approaches. In the top-down approach, object detection methods are based on rules derived from human knowledge of the objects being detected. In the bottom-up approach, object detection methods associate objects with low-level structural features or patterns and then locate objects by searching for these features or patterns.
- Object segmentation In this approach, an image or video is decomposed into its constituent "objects," which may include semantic entities or visual structures, such as color patches. This decomposition is commonly based on the motion, color, and texture attributes of the objects. Object segmentation has several applications, including compact video coding, automatic and semi- automatic content-based description, film post-production, and scene interpretation. In particular, segmentation simplifies the object localization problem by providing an object-based description of a scene.
- Figure 2 illustrates approximate object localization provided by object localization module 14.
- a user draws, for example, an ellipse around the region in which the object is located to approximately locate the object.
- the approximate object localization information i.e., the center point, major axis, and minor axis parameters of the ellipse
- object localization module 14 operates in a fully automated mode. In practice, however, some manual assistance might be required to correct errors made by the system, or, at the very least, to define important objects for the system to localize. Enhancing non-object areas can cause the viewer to be distracted and miss the real action. To avoid or minimize this problem, a user can draw, as described above, an ellipse around the object and the system then can track the object from the specified location. If an object is successfully located in a frame, object localization module 14 outputs the corresponding ellipse parameters (i.e., center point, major axis, and minor axis). Ideally, the contour of this bounding ellipse would coincide with that of the object.
- ellipse parameters i.e., center point, major axis, and minor axis.
- the Figure 1 system further includes means, responsive to the video input and the object localization information that is received from object localization module 14 for developing an enhanced video of that portion of the digital picture that contains the object of interest and the region in which the object is located.
- object enhancement module 16 can be a unit of conventional construction and operation that enhances the visibility of the region of the digital picture that contains the object of interest by applying conventional image processing operations to this region.
- the object localization information that is received, on a frame-by-frame basis, from object localization module 14 includes the grid coordinates of a region of predetermined size in which the object of interest is located.
- object enhancement helps in reducing degradation of the object during the encoding stage which follows the enhancement stage and is described below.
- the operation of the Figure 1 system up to this point corresponds to the preprocessing mode of operation referred to above.
- the visibility of the object is improved by applying image processing operations in the region in which the object of interest is located.
- image processing operations can be applied along the object boundary (e.g. edge sharpening), inside the object (e.g. texture enhancement), and possibly even outside the object (e.g. contrast increase, blurring outside the object area).
- edge sharpening e.g. edge sharpening
- texture enhancement e.g. texture enhancement
- contrast increase e.g. contrast increase, blurring outside the object area
- one way to draw more attention to an object is to sharpen the edges inside the object and along the object contour. This makes the details in the object more visible and also makes the object stand out from the background. Furthermore, sharper edges tend to survive encoding better.
- Another possibility is to enlarge the object, for instance by iteratively applying smoothing, sharpening and object refinement operations, not necessarily in that order.
- Figures 3A through 3D illustrate the work-flow in the object enhancement process.
- Figure 3A is a single frame in a soccer video with the object in focus being a soccer ball.
- Figure 3B shows the output of object localization module 14, namely the object localization information of the soccer ball in the frame.
- Figure 3C illustrates a region refinement step, considered in greater detail below, wherein the approximate object location information of Figure 3B is refined to develop a more accurate estimate of the object boundary, namely the light colored line enclosing the ball.
- Figure 3D shows the result after applying object enhancement, in this example the edge sharpening. Note that the soccer ball is sharper in Figure 3D, and thus more visible, than in the original frame of Figure 3A.
- the object also has higher contrast, which generally refers to making the dark colors darker and the light colors lighter.
- refinement of the object localization information might be required when the object localization information only approximates the nature of the object and the location of the object in each frame to avoid enhancing features outside the boundary of the region in which the object is located.
- object localization module 14 The development of the object localization information by object localization module 14 and the delivery of the object localization information to object enhancement module 16 can be fully-automatic as described above. As frames of the input video are received by object localization module 14, the object localization information is updated by the object localization module and the updated object localization information is delivered to object enhancement module 16.
- the development of the object localization information by object localization module 14 and the delivery of the object localization information to object enhancement module 16 also can be semi-automatic. Instead of delivery of the object localization information directly from object localization module 14 to object enhancement module 16, a user, after having available the object localization information, can manually add to the digital picture of the input video markings, such boundary lines, which define the region of predetermined size in which the object is located.
- the development of the object localization information and delivery of the object localization information to object enhancement module 16 also can be fully-manual. In such operation, a user views the digital picture of the input video and manually adds to the digital picture of the input video markings, such boundary lines, which define the region of predetermined size in which the object is located. As a practical matter, fully-manual operation is not recommended for live events coverage.
- the refinement of object localization information involves object boundary estimation, wherein the exact boundary of the object is estimated.
- the estimation of exact boundaries helps in enhancing the object visibility without the side effect of unnatural object appearance and motion and is based on several criteria. Three approaches for object boundary estimation are disclosed.
- the first is an ellipse-based approach that determines or identifies the ellipse that most tightly bounds the object by searching over a range of ellipse parameters.
- the second approach for object boundary estimation is a level-set based search wherein a level-set representation of the object neighborhood is obtained and then a search is conducted for the level-set contour that most likely represents the object boundary.
- a third approach for object boundary estimation involves curve evolution methods, such as contours or snakes, that can be used to shrink or expand a curve with certain constraints, so that it converges to the object boundary. Only the first and second approaches for object boundary estimation are considered in greater detail below.
- object boundary estimation is equivalent to determining the parameters of the ellipse that most tightly bounds the object.
- This approach searches over a range of ellipse parameters around the initial values (i.e., the output of the object localization module 14) and determines the tightness with which each ellipse bounds the object.
- the output of the algorithm, illustrated in Figure 4 is the tightest bounding ellipse.
- the tightness measure of an ellipse is defined to be the average gradient of image intensity along the edge of the ellipse.
- the rationale behind this measure is that the tightest bounding ellipse should follow the object contour closely and the gradient of image intensity is typically high along the object contour (i.e., the edge between object and background).
- the flowchart for the object boundary estimation algorithm is shown in Figure 4.
- the search ranges ( ⁇ x , ⁇ y , ⁇ a, ⁇ b ) for refining the parameters are user-specified.
- the flow chart of Figure 4 begins by computing the average intensity gradient. Then variables are initialized and four nested loops for horizontal centerpoint location, vertical centerpoint location, and the two axes are entered. If the ellipse described by this centerpoint and the two axes produces a better (i.e., larger) average intensity gradient, then this gradient value and this ellipse are noted as being the best so far. Next is looping through all four loops, exiting with the best ellipse.
- the ellipse-based approach may be applied to environments in which the boundary between the object and the background has a uniformly high gradient. However, this approach may also be applied to environments in which the boundary does not have a uniformly high gradient. For example, this approach is also useful even if the object and/or the background has variations in intensity along the object/background boundary.
- the ellipse-based approach produces, in a typical implementation, the description of a best-fit ellipse.
- the description typically includes centerpoint, and major and minor axes.
- An ellipse-based representation can be inadequate for describing objects with arbitrary shapes. Even elliptical objects may appear to be of irregular shape when motion-blurred or partially occluded.
- the level-set representation facilitates the estimation of boundaries of arbitrarily shaped objects.
- Figures 5A through 5D illustrate the concept of the level-set approach for object boundary estimation.
- the intensity image l(x, y) is a continuous intensity surface, such as shown in Figure 5B 1 and not a grid of discrete intensities, such as shown in Figure 5A.
- l(x, y) / ⁇ .
- the closed contours may be described as continuous curves or by a string of discrete pixels that follow the curve.
- Level-sets can be extracted from images by several methods. One of these methods is to apply bilinear interpolation between sets of four pixels at a time in order to convert a discrete intensity grid into an intensity surface, continuous in both space and intensity value. Thereafter, level-sets, such as shown in Figure 5D, are extracted by computing the intersection of the surface with one or more level planes, such as shown in Figure. 5C, (i.e., horizontal planes at specified levels).
- a level-set representation is analogous in many ways to a topographical map.
- the topographical map typically includes closed contours for various values of elevation.
- the image /can be a subimage containing the object whose boundary is to be estimated.
- all the level-set curves i.e., closed contours
- Cy contained in the set L[M) are considered.
- Object boundary estimation is cast as a problem of determining the level-set curve, C * , which best satisfies a number of criteria relevant to the object. These criteria may include, among others, the following variables:
- the criteria may place constraints on these variables based on prior knowledge about the object.
- object boundary estimation using level-sets.
- m ⁇ e i, Sr ⁇ f, a re f, and Xr ⁇ f (*ref. Yrei), be the reference values for the mean intensity, standard deviation of intensities, area, and the center, respectively, of the object. These can be initialized based on prior knowledge about the object, (e.g., object parameters from the object localization module 14, for example, obtained from an ellipse).
- the set of levels, M is then constructed as,
- / min Lm re f - Sred - 0.5
- W Lm r ⁇ f + s ⁇ J + 0.5
- ⁇ / L(W - 4nm) / ⁇ /J
- N is a preset value (e.g., 10).
- LJ denotes an integer flooring operation.
- S a and S x are similarity functions whose output values lie in the range [0, 1], with a higher value indicating a better match between the reference and measured values.
- S a exp( -
- S x exp( -
- C * argmax[s(C,)]).
- the visibility of the object is improved by applying image processing operations in the neighborhood of the object. These operations may be applied along the object boundary (e.g., edge sharpening), inside the object (e.g., texture enhancement), and possibly even outside the object (e.g., contrast increase).
- object boundary e.g., edge sharpening
- texture enhancement e.g., texture enhancement
- contrast increase e.g., contrast increase
- a first is to sharpen the edges inside the object and along its contour.
- a second is to enlarge the object by iteratively applying smoothing, sharpening and boundary estimation operations, not necessarily in that order.
- Other possible methods include the use of morphological filters and object replacement.
- the algorithm for object enhancement by sharpening operates on an object one frame at a time and takes as its input the intensity image l(x, y), and the object parameters (i.e., location, size, etc.) provided by object localization module 14.
- the algorithm comprises three steps as follows:
- the sharpening filter F a is defined as the difference of the Kronecker delta function and the discrete Laplacian operator V a 2
- the parameter ⁇ e [0, 1] controls the shape of the Laplacian operator.
- a 3 x 3 filter kernel is constructed with the center of the kernel being the origin (0, 0).
- An example of such a kernel is shown below:
- Object enhancement by enlargement attempts to extend the contour of an object by iteratively applying smoothing, sharpening and boundary estimation operations, not necessarily in that order.
- the flowchart for a specific embodiment of the object enlargement algorithm is shown in Figure 6.
- the algorithm takes as its input the intensity image /(x, y), and the object parameters provided by object localization module 14.
- a region (subimage J) containing the object with a sufficient margin around the object is isolated and smoothed using a Gaussian filter. This operation spreads the object boundary outward by a few pixels.
- a sharpening operation described previously, is applied to make the edges clearer.
- the boundary estimation algorithm is applied to obtain a new estimate of the object boundary, O. Finally, all the pixels in image /contained by O are replaced by the corresponding pixels in subimage Jsm ⁇ thsharp-
- the smoothing filter G ⁇ is a two-dimensional Gaussian function
- the parameter ⁇ > 0 controls the shape of the Gaussian function, greater values resulting in more smoothing.
- a 3 x 3 filter kernel is constructed with the center of the kernel being the origin (0, 0).
- An example of such a kernel is shown below: 0.0751 0.1238 0.0751
- the Figure 1 system also includes means for encoding the enhanced video output from object enhancement module 16.
- object-aware encoder module 18 can be a module of conventional construction and operation that compresses the enhanced video with minimal degradation to important objects, by giving special treatment to the region of interest that contains the object of interest by, for example, allocating more bits to the region of interest or perform mode decisions that will better preserve the object. In this way, object-aware encoder 18 exploits the enhanced visibility of the object to encode the object with high fidelity.
- object-aware encoder 18 receives the object localization information from object localization module 14, thereby better preserving the enhancement of the region in which the object is located and, consequently, the object. Whether the enhancement is preserved or not, the region in which the object is located is better preserved than without encoding by object-aware encoder 18. However, the enhancement also minimizes object degradation during compression. This optimized enhancement is accomplished by suitably managing encoding decisions and the allocation of resources, such as bits.
- Object-aware encoder 18 can be arranged for making "object-friendly" macroblock (MB) mode decisions, namely those that are less likely to degrade the object.
- MB macroblock
- Such an arrangement for example, can include an object-friendly partitioning of the MB for prediction purposes, such as illustrated by Figures 7 A through 7C.
- Another approach is to force finer quantization, namely more bits, to MBs containing objects. This results in the object getting more bits.
- Yet another approach targets the object itself for additional bits.
- Still another approach uses a weighted distortion metrics during the rate-distortion optimization process, where pixels belonging to the regions of interest would have a higher weight than pixels outside the regions of interest.
- FIG. 7A through 7C there are shown three possible subdivisions of a 16x16 macroblock. Such sub-divisions are part of the mode decision that an encoder makes for determining how to encode the MB.
- One key metric is that if the object takes up a higher percentage of the area of the subdivision, then the object is less likely to be degraded during the encoding. This follows because degrading the object would degrade the quality of a higher portion of the sub-division. So, in Figure 7C, the object makes up only a small portion of each 16x8 sub-division, and, accordingly, this is not considered a good sub-division.
- An object-aware encoder in various implementations knows where the object is located and factors this location information into its mode decision.
- Such an object-aware encoder favors sub-divisions that result in the object occupying a larger portion of the sub-division.
- the goal of object-aware encoder 18 is to help the object suffer as little degradation as possible during the encoding process.
- object localization module 14, object enhancement module 16, and object-aware encoder module 18 are components of transmitter 20 that receives input video of a digital picture containing an object of interest and transmits a compressed video stream with the visibility of the object enhanced.
- the transmission of the compressed video stream is received by receiver 20, such as a cell phone or PDA.
- the Figure 1 system further includes means for decoding the enhanced video in the compressed video stream received by receiver 20.
- Such means identified in Figure 1 as a decoder module 22, can be a module of conventional construction and operation that decompresses the enhanced video with minimal degradation to important objects, by giving special treatment to the region of interest that contains the object of interest by, for example, allocating more bits to the region of interest or perform mode decisions that will better preserve the enhanced visibility of the object.
- the decoded video output from decoder module 22 is conducted to a display component 26, such as the screen of a cell phone or a PDA, for viewing of the digital picture with enhanced visibility of the object.
- the input video can be conducted directly to object-aware encoder module 18, as represented by dotted line 19, and encoded without the visibility of the object enhanced and have the enhancement effected by an object-aware post-processing module 24 in receiver 20.
- This mode of operation of the Figure 1 system is characterized as post-processing in that the visibility of the object is enhanced after the encoding and decoding stages and may be effected by utilizing side-information about the object, for example the location and size of the object, sent through the bitstream as metadata.
- the postprocessing mode of operation has the disadvantage of increased receiver complexity.
- object-aware encoder 18 in transmitter 10 exploits only the object location information when the visibility of the object is enhanced in the receiver.
- one advantage of a transmitter-end object highlighting system i.e., the pre-processing mode of operation
- the pre-processing mode of operation allows using standard video decoders, which facilitates the deployment of the system.
- the implementations that are described may be implemented in, for example, a method or process, an apparatus, or a software program. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation or features discussed may also be implemented in other forms (e.g., an apparatus or a program).
- An apparatus may be implemented in, for example, appropriate hardware, software, and firmware.
- the methods may be implemented in, for example, an apparatus such as, for example, a computer or other processing device. Additionally, the methods may be implemented by instructions being performed by a processing device or other apparatus, and such instructions may be stored on a computer readable medium such as, for example, a CD, or other computer readable storage device, or an integrated circuit.
- implementations may also produce a signal formatted to carry information that may be, for example, stored or transmitted.
- the information may include, for example, instructions for performing a method, or data produced by one of the described implementations.
- a signal may be formatted to carry as data various types of object information (i.e., location, shape), and/or to carry as data encoded image data.
Abstract
Description
Claims
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011503987A JP2011517226A (en) | 2008-04-11 | 2009-04-07 | System and method for enhancing the sharpness of an object in a digital picture |
EP09729220A EP2277142A1 (en) | 2008-04-11 | 2009-04-07 | System and method for enhancing the visibility of an object in a digital picture |
US12/736,408 US20110026606A1 (en) | 1999-03-11 | 2009-04-07 | System and method for enhancing the visibility of an object in a digital picture |
BRPI0911189A BRPI0911189A2 (en) | 2008-04-11 | 2009-04-07 | system and method for improving the visibility of an object in a digital image |
CN200980112778.9A CN101999138A (en) | 2008-04-11 | 2009-04-07 | System and method for enhancing the visibility of an object in a digital picture |
CA2720947A CA2720947A1 (en) | 2008-04-11 | 2009-04-07 | System and method for enhancing the visibility of an object in a digital picture |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12384408P | 2008-04-11 | 2008-04-11 | |
US60/123,844 | 2008-04-11 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2009126258A1 true WO2009126258A1 (en) | 2009-10-15 |
WO2009126258A9 WO2009126258A9 (en) | 2009-12-17 |
Family
ID=40848271
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/002173 WO2009126258A1 (en) | 1999-03-11 | 2009-04-07 | System and method for enhancing the visibility of an object in a digital picture |
Country Status (6)
Country | Link |
---|---|
EP (1) | EP2277142A1 (en) |
JP (1) | JP2011517226A (en) |
CN (1) | CN101999138A (en) |
BR (1) | BRPI0911189A2 (en) |
CA (1) | CA2720947A1 (en) |
WO (1) | WO2009126258A1 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20120114263A (en) * | 2009-12-14 | 2012-10-16 | 톰슨 라이센싱 | Object-aware video encoding strategies |
CN103004213A (en) * | 2010-06-08 | 2013-03-27 | 杜比实验室特许公司 | Tone and gamut mapping methods and apparatus |
US8665286B2 (en) | 2010-08-12 | 2014-03-04 | Telefonaktiebolaget Lm Ericsson (Publ) | Composition of digital images for perceptibility thereof |
US10022544B2 (en) | 2013-07-22 | 2018-07-17 | National Ict Australia Limited | Vision enhancement apparatus for a vision impaired user |
US10583290B2 (en) | 2014-09-10 | 2020-03-10 | National Ict Australia Limited | Enhancing vision for a vision impaired user |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110173752B (en) * | 2018-06-19 | 2021-04-13 | 安徽新大陆特种涂料有限责任公司 | Intelligent humidifying type warmer |
CN111028243A (en) * | 2019-11-29 | 2020-04-17 | 上海交通大学 | Method, system and device for segmenting neuroblastoma of children based on CT scanning image |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5625715A (en) * | 1990-09-07 | 1997-04-29 | U.S. Philips Corporation | Method and apparatus for encoding pictures including a moving object |
US6466275B1 (en) * | 1999-04-16 | 2002-10-15 | Sportvision, Inc. | Enhancing a video of an event at a remote location using data acquired at the event |
US20050036704A1 (en) * | 2003-08-13 | 2005-02-17 | Adriana Dumitras | Pre-processing method and system for data reduction of video sequences and bit rate reduction of compressed video sequences using spatial filtering |
WO2007045001A1 (en) * | 2005-10-21 | 2007-04-26 | Mobilkom Austria Aktiengesellschaft | Preprocessing of game video sequences for transmission over mobile networks |
US20070198906A1 (en) * | 2006-02-13 | 2007-08-23 | Snell & Wilcox Limited | Sport Action Coding |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2002207992A (en) * | 2001-01-12 | 2002-07-26 | Hitachi Ltd | Method and device for processing image |
US6757434B2 (en) * | 2002-11-12 | 2004-06-29 | Nokia Corporation | Region-of-interest tracking method and device for wavelet-based video coding |
JP4468734B2 (en) * | 2004-04-27 | 2010-05-26 | オリンパス株式会社 | Video signal processing apparatus and video signal processing program |
JP2006013722A (en) * | 2004-06-23 | 2006-01-12 | Matsushita Electric Ind Co Ltd | Unit and method for processing image |
JP4703449B2 (en) * | 2006-03-23 | 2011-06-15 | 三洋電機株式会社 | Encoding method |
WO2008039217A1 (en) * | 2006-09-29 | 2008-04-03 | Thomson Licensing | Dynamic state estimation |
-
2009
- 2009-04-07 CN CN200980112778.9A patent/CN101999138A/en active Pending
- 2009-04-07 BR BRPI0911189A patent/BRPI0911189A2/en not_active IP Right Cessation
- 2009-04-07 WO PCT/US2009/002173 patent/WO2009126258A1/en active Application Filing
- 2009-04-07 JP JP2011503987A patent/JP2011517226A/en active Pending
- 2009-04-07 CA CA2720947A patent/CA2720947A1/en not_active Abandoned
- 2009-04-07 EP EP09729220A patent/EP2277142A1/en not_active Withdrawn
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5625715A (en) * | 1990-09-07 | 1997-04-29 | U.S. Philips Corporation | Method and apparatus for encoding pictures including a moving object |
US6466275B1 (en) * | 1999-04-16 | 2002-10-15 | Sportvision, Inc. | Enhancing a video of an event at a remote location using data acquired at the event |
US20050036704A1 (en) * | 2003-08-13 | 2005-02-17 | Adriana Dumitras | Pre-processing method and system for data reduction of video sequences and bit rate reduction of compressed video sequences using spatial filtering |
WO2007045001A1 (en) * | 2005-10-21 | 2007-04-26 | Mobilkom Austria Aktiengesellschaft | Preprocessing of game video sequences for transmission over mobile networks |
US20070198906A1 (en) * | 2006-02-13 | 2007-08-23 | Snell & Wilcox Limited | Sport Action Coding |
Non-Patent Citations (1)
Title |
---|
See also references of EP2277142A1 * |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20120114263A (en) * | 2009-12-14 | 2012-10-16 | 톰슨 라이센싱 | Object-aware video encoding strategies |
JP2013513998A (en) * | 2009-12-14 | 2013-04-22 | トムソン ライセンシング | Object recognition video coding strategy |
US9118912B2 (en) | 2009-12-14 | 2015-08-25 | Thomson Licensing | Object-aware video encoding strategies |
CN103004213A (en) * | 2010-06-08 | 2013-03-27 | 杜比实验室特许公司 | Tone and gamut mapping methods and apparatus |
US20130076763A1 (en) * | 2010-06-08 | 2013-03-28 | Dolby Laboratories Licensing Corporation | Tone and Gamut Mapping Methods and Apparatus |
JP2013527732A (en) * | 2010-06-08 | 2013-06-27 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Gradation and color gamut mapping method and apparatus |
US8928686B2 (en) | 2010-06-08 | 2015-01-06 | Dolby Laboratories Licensing Corporation | Tone and gamut mapping methods and apparatus |
CN103004213B (en) * | 2010-06-08 | 2016-04-20 | 杜比实验室特许公司 | Tone and method of color gamut mapping of color and device |
US9728117B2 (en) | 2010-06-08 | 2017-08-08 | Dolby Laboratories Licensing Corporation | Tone and gamut mapping methods and apparatus |
US8665286B2 (en) | 2010-08-12 | 2014-03-04 | Telefonaktiebolaget Lm Ericsson (Publ) | Composition of digital images for perceptibility thereof |
US10022544B2 (en) | 2013-07-22 | 2018-07-17 | National Ict Australia Limited | Vision enhancement apparatus for a vision impaired user |
US10583290B2 (en) | 2014-09-10 | 2020-03-10 | National Ict Australia Limited | Enhancing vision for a vision impaired user |
Also Published As
Publication number | Publication date |
---|---|
EP2277142A1 (en) | 2011-01-26 |
CN101999138A (en) | 2011-03-30 |
JP2011517226A (en) | 2011-05-26 |
BRPI0911189A2 (en) | 2018-05-22 |
WO2009126258A9 (en) | 2009-12-17 |
CA2720947A1 (en) | 2009-10-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20110026607A1 (en) | System and method for enhancing the visibility of an object in a digital picture | |
Rao et al. | A Survey of Video Enhancement Techniques. | |
Emberton et al. | Hierarchical rank-based veiling light estimation for underwater dehazing. | |
US20190180454A1 (en) | Detecting motion dragging artifacts for dynamic adjustment of frame rate conversion settings | |
US8774512B2 (en) | Filling holes in depth maps | |
US20030053692A1 (en) | Method of and apparatus for segmenting a pixellated image | |
US7085401B2 (en) | Automatic object extraction | |
US8290264B2 (en) | Image processing method and apparatus | |
WO2009126258A9 (en) | System and method for enhancing the visibility of an object in a digital picture | |
US20110026606A1 (en) | System and method for enhancing the visibility of an object in a digital picture | |
JP4118688B2 (en) | System and method for enhancement based on segmentation of video images | |
CN111445424B (en) | Image processing method, device, equipment and medium for processing mobile terminal video | |
CN109784164B (en) | Foreground identification method and device, electronic equipment and storage medium | |
US9743062B2 (en) | Method and device for retargeting a 3D content | |
CN107886518B (en) | Picture detection method and device, electronic equipment and readable storage medium | |
US20230343017A1 (en) | Virtual viewport generation method and apparatus, rendering and decoding methods and apparatuses, device and storage medium | |
US20230131418A1 (en) | Two-dimensional (2d) feature database generation | |
Guo et al. | Progressive Domain Translation Defogging network for real-world fog images | |
Ancuti et al. | Single image restoration of outdoor scenes | |
Yan et al. | A point light source interference removal method for image dehazing | |
CN113452996B (en) | Video coding and decoding method and device | |
CN112991188B (en) | Image processing method and device, storage medium and electronic equipment | |
US20240087185A1 (en) | Virtual view drawing method, rendering method, and decoding method | |
Couto | Inpainting-based Image Coding: A Patch-driven Approach | |
CN115423817A (en) | Image segmentation method, device, electronic device and medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200980112778.9 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09729220 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 6998/DELNP/2010 Country of ref document: IN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 12736408 Country of ref document: US Ref document number: 2011503987 Country of ref document: JP Ref document number: 2009729220 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2720947 Country of ref document: CA |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: PI0911189 Country of ref document: BR Kind code of ref document: A2 Effective date: 20101006 |