WO2009087952A1 - Methods and apparatus for inter-layer image prediction parameter determination - Google Patents
Methods and apparatus for inter-layer image prediction parameter determination Download PDFInfo
- Publication number
- WO2009087952A1 WO2009087952A1 PCT/JP2008/073966 JP2008073966W WO2009087952A1 WO 2009087952 A1 WO2009087952 A1 WO 2009087952A1 JP 2008073966 W JP2008073966 W JP 2008073966W WO 2009087952 A1 WO2009087952 A1 WO 2009087952A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- hdr
- predicted
- ldr
- parameter value
- prediction parameter
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 239000011229 interlayer Substances 0.000 title description 25
- 238000013507 mapping Methods 0.000 claims description 103
- 239000013598 vector Substances 0.000 claims description 20
- 241000023320 Luma <angiosperm> Species 0.000 claims description 12
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 claims description 12
- 238000004590 computer program Methods 0.000 abstract description 4
- 239000010410 layer Substances 0.000 description 113
- 230000006870 function Effects 0.000 description 29
- 230000008569 process Effects 0.000 description 21
- 238000012937 correction Methods 0.000 description 15
- 230000007246 mechanism Effects 0.000 description 9
- 230000001419 dependent effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000011664 signaling Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 102100020970 ATP-binding cassette sub-family D member 2 Human genes 0.000 description 1
- 101000783774 Homo sapiens ATP-binding cassette sub-family D member 2 Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/223—Analysis of motion using block-matching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Definitions
- Embodiments of the present invention comprise methods and systems for inter-layer image prediction.
- LDR low dynamic range
- HDR high dynamic range
- the base layer of a video bitstream may contain a representation of the video sequence at a reduced bit-depth.
- the base layer may contain an eight-bit representation of the sequence
- the enhancement layer of the bitstream may contain a ten-bit representation.
- more than two layers may be used.
- an eight-bit version may represent the eight most significant bits of the higher bit-depth sequence .
- the higher bit-depth version may therefore be predicted by multiplying (or equivalently scaling) the decoded lower bit-depth data to the higher bit-depth.
- the eight-bit data may be decoded and subsequently scaled by a factor of four to predict the ten-bit data. This scaling may be done in either the intensity or transform domain, depending on the application.
- High dynamic range video coding may be considered a more general case of bit-depth scalability.
- the base layer and enhancement layer may contain data represented with different bit-depths. However, the base layer may not be constrained to represent the most significant bits of the enhancement-layer data.
- the base-layer data may contain a lower bit-depth representation of the high dynamic range sequence, and this lower bit-depth may not always correspond to the most significant bits of the corresponding higher bit- depth representation.
- Some embodiments of the present invention comprise methods and systems for prediction of images comprising multiple dynamic range layers. Some embodiments of the present invention comprise methods and systems for determining a high dynamic range (HDR) prediction parameter which may be used to predict HDR image data from low dynamic range (LDR) image data. In some embodiments of the present invention, an HDR prediction parameter may be predicted based on analysis of decoded LDR image data. In some embodiments of the present invention, an HDR prediction parameter may be predicted from the HDR prediction parameters corresponding to previously transmitted image frames.
- HDR high dynamic range
- LDR low dynamic range
- FIG. IA is a chart showing an exemplary embodiment of the present invention comprising prediction with a scaled and offset LDR image element
- Fig. I B is a chart showing an exemplary embodiment of the present invention comprising scaling and offsetting decoded image elements for HDR prediction;
- Fig. 2 is a chart showing an exemplary embodiment of the present invention comprising conversion to an alternative color space
- Fig. 3 is a chart showing an exemplary embodiment of the present invention comprising scaling an LDR image element according to HDR bitstream data
- Fig. 4 is a chart showing an exemplary embodiment of the present invention comprising scaling and applying an offset to an LDR image element according to HDR bitstream data;
- Fig. 5 is a chart showing an exemplary embodiment of the present invention comprising scaling LDR transform coefficients for HDR prediction
- Fig. 6 is a chart showing an exemplary embodiment of the present invention comprising applying an offset to LDR transform coefficients for HDR prediction;
- Fig. 7 is a chart showing an exemplary embodiment of the present invention comprising scaling LDR transform coefficients and applying an offset to LDR transform coefficients for HDR prediction;
- Fig. 8 is a chart showing an exemplary embodiment of the present invention comprising scaling and applying an offset to color-transformed image elements for HDR prediction
- Fig. 9 is a chart showing an exemplary embodiment of the present invention comprising separate scaling and offset operations for luminance and chrominance elements
- Fig. 10 is a chart showing an exemplary embodiment of the present invention comprising using a scale mapping parameter to predict a scale parameter value
- Fig. 1 1 is a chart showing an exemplary embodiment of the present invention comprising using an offset mapping parameter to predict an offset parameter value
- Fig. 12 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted scale parameter value in accordance with a residual value received as part of the HDR enhancement-layer bitstream;
- Fig. 13 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted offset parameter value in accordance with a residual value received as part of an HDR enhancement-layer bitstream;
- Fig. 14 is a chart showing an exemplary embodiment of the present invention comprising a scale correspondence indicator for predicting a scale parameter value
- Fig. 15 is a chart showing an exemplary embodiment of the present invention comprising an offset correspondence indicator for predicting an offset parameter value
- Fig. 16 is a chart showing an exemplary embodiment of the present invention comprising using an HDR prediction parameter correspondence indicator to index stored HDR prediction parameter values from previous image frames to predict a current HDR prediction parameter value;
- Fig. 17 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted HDR prediction parameter value in accordance with a residual value received as part of an HDR enhancement-layer bitstream;
- Fig. 18 is a chart showing an exemplary embodiment of the present invention comprising correcting a motion vector correspondence value in accordance with a residual value received as part of an HDR enhancement-layer bitstream.
- Some embodiments of the present invention comprise systems and methods for using a low dynamic range (LDR) video sequence to predict a high dynamic range (HDR) version of the image data. This may be referred to as inter-layer prediction in this application.
- Some embodiments of the present invention comprise a spatially-varying inter-layer prediction mechanism for HDR video coding.
- Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that operates in the color spaces used for video compression and transmission. Some embodiments use gamma-corrected color spaces. Exemplary embodiments may use xvYCC and YCbCr color spaces.
- Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that may be disabled spatially.
- Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that is multiplication free. Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that may be used in a single-loop decoder. Some embodiments may also be incorporated into multi-loop designs. Some embodiments of the present invention comprise an inter-layer prediction technique for high-dynamic range video coding. Some aspects of some embodiments comprise elements described in U . S . Patent Publication No. 2007- 0223813 A l (Application No. 1 1 / 362 , 571 ) , filed on Feb. 24 , 2006, by Christopher Andrew Segall, which is hereby incorporated herein by reference. Some embodiments of the present invention comprise a method for projecting decoded low dynamic range data to the high dynamic range coding space. This process may be referred to as inter-layer prediction.
- a high dynamic range (HDR) image may be received 100.
- a corresponding low dynamic range (LDR) image or a plurality of LDR images may also be received, or, alternatively, created
- the LDR image may be created 101 through a tone-scale operation, a conversion function or some other method.
- the LDR image may then be predicted, transformed, quantized and encoded 102 , as is well known in the art.
- the LDR image may be transformed using a discrete cosine transform (DCT) , a wavelet transform or by other common transformation methods.
- DCT discrete cosine transform
- the prediction, transformation, quantization and encoding processes may then be substantially reversed 103 to provide a decoded LDR image as would be decoded at a typical decoder.
- a de-quantization process is lossy and therefore does not produce an exact copy of the originally encoded image.
- Other processes may also affect the reproduction of the original LDR image .
- the decoded LDR image may be processed by one or more of the following methods: color conversion, scaling 104 and offsetting 105.
- the decoded, processed LDR image may now be used to create 106 a residual HDR image. This may be performed by subtracting the decoded, processed LDR image from the original HDR image . Other methods may also be used.
- the residual HDR image may then be transformed, quantized and encoded 107, or otherwise prepared, for transmission to a destination.
- this step may comprise embedding the encoded residual HDR image into an HDR, or enhancement-layer, bitstream.
- Information related to the color conversion, scaling and offset operations may also be encoded and embedded 108 in the HDR, or enhancement-layer, bitstream.
- the HDR/ enhancement-layer bitstream may then be transmitted 109 to a destination.
- An LDR/ base-layer bitstream may also be transmitted 1 10 to the destination.
- the LDR/ base-layer bitstream may also comprise a transformed, quantized and encoded LDR image.
- a decoder receiving the LDR/ base-layer bitstream may then decode the LDR/ base-layer image.
- a decoder receiving the LDR/ base-layer bitstream and the HDR/ enhancement- layer bitstream may decode both the LDR/ base-layer image and the HDR/ enhancement-layer image.
- Embodiments of the present invention comprise methods and systems for encoding and decoding images in this framework and similar scenarios.
- a base-layer decoder may receive base-layer data, such as from a base-layer bitstream 2.
- the base-layer decoder may- decode 6 a base-layer block, or other image element, and represent it in the spatial domain.
- Some embodiments may comprise full decoding of the block, including a prediction process followed by residual refinement.
- Some embodiments may comprise reconstruction of the residual only.
- the spatial information in the base layer may be used to predict the high dynamic range signal.
- Some embodiments may comprise scaling 7 the base-layer information.
- Some embodiments may also comprise adding an offset 8 to the base-layer information.
- Some embodiments may comprise both scaling 7 and adding an offset 8.
- scaling 7 and/ or adding an offset 8 may be used to predict 9 an enhancement layer, such as a higher dynamic range (HDR) layer.
- scaling 7 and offset 8 data may be extracted from an enhancement-layer bitstream 4.
- subsequent refinement may be decoded from the enhancement-layer bitstream 4.
- Scaling and offset operations may be performed according to instructions and/ or data received from an enhancement bitstream 1 1. This converted, scaled and/ or offset data may then be converted 16 back to the coding color space. Once converted back to the coding color space, the scaled and/ or offset data may be used to predict 17 an enhancement layer, such as a higher dynamic range (HDR) layer.
- HDR higher dynamic range
- a decoder may receive base-layer data 10 from which a block, or other image element, may be decoded 12 into spatial image data. This spatial image data may then be converted 13 to an alternative color space. This converted data may then be scaled 14 and/ or offset 15. Scaling and offset operations may be performed according to instructions and/ or data received from an enhancement bitstream 1 1. This converted, scaled and/ or offset data may then be converted 16 back to the coding color space. Once converted back to the coding color space, the scaled and/ or offset data may be used to predict
- an enhancement layer such as a higher dynamic range (HDR) layer.
- HDR higher dynamic range
- LDR/ base-layer image data may be received 30 and corresponding HDR/ enhancement-layer data also may be received 31.
- An LDR/ base-layer block, or other image element then may be decoded 32 from the LDR/ base-layer data.
- the decoded LDR/ base-layer image element then may be scaled 33. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to, or a function of, image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the scaled, decoded LDR/ base-layer image may then be used to predict 34 a corresponding HDR block or other image element.
- the scaled, decoded LDR/ base-layer image element may be added to a corresponding decoded residual image element to form an HDR/ enhancement-layer (residually-corrected) image element.
- LDR/ base-layer image data may be received 40 and corresponding HDR/ enhancement-layer data also may be received 4 1.
- An LDR/ base-layer block, or other image element then may be decoded 42 from the LDR/ base-layer image data.
- the decoded LDR/ base-layer image element may be then scaled 43. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data.
- An offset may then be added 44 to the scaled LDR image element.
- Offset data may be carried in the corresponding HDR/ enhancement-layer data. Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the scaled, offset and decoded LDR/base-layer image may then be used to predict 45 a corresponding HDR block or other image element.
- the scaled, offset and decoded LDR/ base-layer image element may be added to a corresponding decoded residual image element to form an HDR/ enhancement-layer image element.
- Some embodiments of the present invention may be described with reference to Figure 5.
- an LDR/ base-layer image comprising LDR transform coefficients may be received 50, and corresponding HDR/ enhancement-layer data also may be received 51 .
- the LDR/ base-layer image transform coefficients may then be scaled 52. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of LDR transform coefficients may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data.
- LDR/ base-layer transform coefficients may then be used to predict 53 transform coefficients for a corresponding HDR block or other image element.
- an LDR/ base-layer image comprising LDR transform coefficients may be received 60, and corresponding HDR/ enhancement-layer data also may be received 61 .
- the LDR/ base-layer image transform coefficients may then be offset 62. Offset data may be carried in the corresponding HDR/ enhancement-layer data 61. Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the offset LDR/ base-layer transform coefficients may then be used to predict 63 transform coefficients for a corresponding HDR block or other image element.
- an LDR/ base-layer image comprising LDR transform coefficients may be received 70, and corresponding HDR/ enhancement-layer data also may be received 71.
- the LDR/ base-layer image transform coefficients may then be scaled 72. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of
- LDR transform coefficients may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the scaled LDR/ base-layer image transform coefficients may then be offset 73. Offset data may be carried in the corresponding
- HDR/ enhancement-layer data 7 1 may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the scaled, offset LDR/ base-layer transform coefficients may then be used to predict 74 transform coefficients for a corresponding HDR block or other image element.
- LDR/ base-layer image data may be received 80, and corresponding HDR/ enhancement-layer data also may be received 81 .
- An LDR/ base-layer block, or image element may then be decoded 82 from the LDR/ base layer image data.
- the decoded LDR/ base-layer image element may then be converted 83 or transformed to an alternative color format or color space.
- the LDR image element may be scaled 84. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data.
- an offset may then be added 85 to the scaled, color-converted LDR image element. Offset data may be carried in the corresponding HDR/ enhancement-layer data.
- Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
- the scaled and/ or offset and color-converted LDR/ base- layer image may then be converted back 86 to the coding color space.
- This scaled and/or offset, coding-color-space LDR/ base-layer image may then be used to predict 87 a corresponding HDR block or image element.
- LDR/ base-layer image data may be received 90, and corresponding HDR/ enhancement-layer data may be also received 91 .
- An LDR/ base-layer block, or image element may then be decoded 92 from the LDR/ base-layer image data.
- the decoded LDR/ base-layer image may comprise separable luminance and chrominance values.
- luminance values may be scaled 93 in relation to their spatial position in the image. Other factors may also affect the luminance value scaling operation. In some embodiments, these luminance values may be offset 94.
- the offset operation may also be related to the spatial position of the luminance value .
- the chrominance values of the decoded LDR/ base-layer image may be scaled 95. This chrominance scaling may also be related to the spatial position of the chrominance value .
- chrominance values may also be offset 96.
- the chrominance value offset may be related to a luminance offset, a chrominance value or scaling factor and/ or a spatial position of the chrominance value. Other factors may also affect the chrominance offset. Once the luminance and chrominance values are scaled and/ or offset, they may be used to predict 97 a corresponding HDR/ enhancement-layer image element.
- the inter- layer prediction process may be controlled at a fine granularity.
- the scaling and offset factors may vary on a 4x4 block basis. That is, for every 4x4 block in the image, an encoder may signal the appropriate scaling and offset factor. Additionally, an encoder may enable and disable inter-layer prediction on a block-by-block basis .
- the high dynamic range image may be predicted from the low dynamic range image in a portion of the frame while predicted with alternative mechanisms in other spatial regions.
- intra-frame and inter- frame prediction mechanisms may be used in these other spatial regions.
- Some embodiments of the present invention comprise inter-layer prediction methods that are multiplication free .
- HDR and LDR are, respectively, the high dynamic range and low dynamic range version of the image sequence
- x and y denote the spatial location within the image frame
- cii is a binary indicator that belongs to the set ⁇ - 1 ,0, 1 ⁇ .
- inter-layer prediction embodiments comprise an offset in the inter-layer prediction process. Some embodiments may comprise a process described in equation 2 :
- Offset(x,y) is the offset value.
- the offset value may be signaled with the scaling values. Alternatively, it may be signaled as part of a residual refinement process.
- control of the prediction process may be enabled at fine granularity.
- the inter-layer prediction process may vary the scaling and offset parameters on a similar block grid. In some embodiments, this may be achieved by sending scaling and/ or offset information from the encoder to the decoder within an enhancement bitstream.
- the scaling factors may be transmitted differentially. That is, the scale factor may be predicted from previously received scale factors. Then, a correction may be transmitted in the bitstream. Some embodiments may predict the scale factor from the upper- or left-most neighbor to the current block. Alternatively, some embodiments may predict the scale factor as the minimum value of the upper- or left-most neighbor.
- the encoder may signal the correction value as a function of the upper and left-most neighbors.
- the encoder and decoder may use a specific context or state for signaling when the neighbors have the same scale factor. An alternative state may be used when the neighbors have different scale factors.
- Some embodiments of the present invention comprise high dynamic range video coding where the scale factor is the same throughout an image region.
- high-level information may also be transmitted from the encoder to the decoder. This high-level information may disable the transmission of scaling and/ or offset parameters on a block-by-block or region-by-region basis. For the case that transmission of the parameters is disabled, the high level information may comprise the scaling and/ or offset information to be used. In some embodiments, this high-level signaling may occur on a macroblock, slice, picture or sequence basis.
- the inter- layer prediction process operates on intensity data. That is, the information may be decoded and converted to the spatial domain by reversing any transform used for signaling.
- the scaling and offset operations may be directly applied in the transform domain.
- the transform coefficients may be de- quantized and then scaled by scale factors.
- transform coefficients may be processed differently depending on their frequency characteristics. For example, in some embodiments, the scaling operation may be applied solely to the AC coefficients while the offset operation may affect the DC component.
- different scaling and offset operations may be signaled for different coefficients or coefficient types.
- Some embodiments of the present invention may comprise a video codec that may adaptively switch between transform domain and spatial domain prediction mechanisms. In some embodiments, this switch may be signaled on a sequence, frame or slice basis. In some embodiments, this switch may operate at finer granularity, such as a block or macro-block.
- a color transform may be used prior to inter-layer prediction. This may address the fact that most color spaces used for video coding are not iso-luminant.
- a video codec typically transmits data in the YCbCr color space with code-word mappings defined in International Telecommunication Union, "Parameter Values for the HDTV Standards for Production and International Programme Exchange," ITU-R BT.709-5, April, 2002.
- Some embodiments of the present invention may perform an inter-layer prediction process in a color space closely related to the coding color space.
- YLDR , CbLDR and C ⁇ LDR are the luma and chroma components in the low dynamic range image sequence, respectively. Then, the scaling and offset process may be applied to YLDR to generate YHDR. Finally, the inter-predicted region may be computed with the following equation 4: ⁇ HDR - »
- CbHDR and CTHDR are predictions for the color components in the high dynamic range layer.
- the components may be resampled.
- applications may down-sample the luma component when the chroma components are stored at a lower resolution.
- the chroma components may be up-sampled to match the resolution of the luma component.
- inter- layer prediction may operate directly on the decoded data without employing a color transform.
- the prediction process may be expressed by the following equation 5:
- Y HDR (* > y) Scale(x, y, c) * Y WR (x, y) + Offset(x, y, x),
- C b HDR (* > y) Scale(x, y,c)* Cb LDR (x,y) + Offset (x, y, x),
- C r HDR (* > y) Scale(x, y, c) * Cr WR (x, y) + Offset (x, y, x), (5)
- scaling and offset parameters may now be a function of both spatial location and chroma component. That is, the reconstructed luma and chroma values may be scaled with different scale factors.
- the luma and chroma values may be scaled with the same scale factor but with different offsets. This may be expressed with the following equation 6:
- Y HDR ( x > y) Scale(x, y) * Y LDR (x, y) + Offset (x, y, x),
- c b HDR ( x > y) Scale(x, y) * Cb WR (x, y) + Offset(x, y, x),
- Cr HDR (x,y) Scale(x,y)* Cr LDR (x,y) + Offset(x,y,x), ( 5 )
- the scale factor may not depend on the chroma component.
- the encoder may transmit the offsets within the enhancement- layer bitstream.
- the luma and chroma values may be scaled with the same scale factor, and the offset for the chroma values may be dependent on the offset of the luma values as well as the decoded image data. This relationship may be expressed in the following equation 7:
- Cr HDR (x,y) Scale(x,y)* Cr LDR (x,y) + f(O_ffset(x,y),Cr LDR (x,y),Y LDR (x,y)), ( 7 )
- mapping operation f(' , ' , ') denotes a mapping operation.
- An exemplary mapping operation may be expressed as equation 8:
- ALDR ⁇ x,y denotes an arbitrary color component, such as Cb or Cr.
- the chroma and luma components may be represented on different sampling grids. In some of these embodiments, the chroma and luma data may be resampled to the same resolution. In some embodiments, a different mapping process may be employed. In some exemplary embodiments, the mapping relationship may be expressed as equation 9 :
- Avg( ) denotes the mean operator.
- the mean may be replaced with a summation operation.
- non-linear operations such as the median, min and max operations may be beneficial.
- the mean operator (or an alternative operator) may be performed in a different domain than that of the Offset variable.
- the mean operation may be computed in the transform domain by operating solely on the DC coefficient.
- the mean operation may be computed by analyzing multiple DC coefficients in the luma base layer.
- Some embodiments of the present invention may comprise elements for determining a predicted value for a high dynamic range (HDR) prediction parameter. Some embodiments of the present invention may comprise elements for refining a predicted HDR prediction parameter value.
- Exemplary HDR prediction parameters include scale parameters, offset parameters and other parameters which may be used in predicting HDR image data from low dynamic range (LDR) image data. While the embodiments described in this section may be described in relation to scale parameters and offset parameters, it will be understood by one of ordinary skill in the art, that the methods and systems embodied in these descriptions may be used for determining other HDR prediction parameter values. These descriptions are by way of illustration and not limitation.
- a scale parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted scale parameter. Some of these embodiments may be described in relation to Figure 10.
- LDR image data may be received 120, and a scale mapping parameter may be derived 122 from the LDR image data.
- the scale mapping parameter may be mapped 124 to a scale parameter prediction.
- the mapping 124 may comprise using the scale mapping parameter as an index into a look-up-table of candidate scale parameter values.
- the look-up-table of candidate scale parameter values may be signaled to a decoder within the bitstream, may be stored as a default lookup-table at the decoder or may otherwise be signaled to the decoder.
- the scale mapping parameter may be used in a scale-parameter generating function.
- the scale-parameter generating function may be signaled to a decoder within the bitstream, may be stored as a default generating function at the decoder or may otherwise be signaled to the decoder.
- the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter.
- the scale mapping parameter may index a luminance-scale- parameter look-up-table to determine a predicted luminance scale parameter, and the scale mapping parameter may index a chrominance-scale-parameter look-up-table to determine a predicted chrominance scale parameter.
- two scale mapping parameters may be generated: a luminance scale mapping parameter and a chrominance scale mapping parameter.
- the two different mapping parameters may index into the same look-up-table.
- the luminance scale mapping parameter and the chrominance scale mapping parameter may index into a luminance-scale-parameter look-up-table and a chrominance- scale-parameter look-up-table, respectively.
- the scale mapping parameter may be used in a luminance-scale- parameter generating function to generate a predicted luminance scale parameter.
- the scale mapping parameter may also be used in a chrominance-scale-parameter generating function to generated a predicted chrominance scale parameter.
- two scale mapping parameters may be generated: a luminance scale mapping parameter and a chrominance scale mapping parameter.
- the two different mapping parameters may be used in the same scale-parameter generating function to generate two scale parameters: a predicted luminance scale parameter and a predicted chrominance scale parameter.
- the luminance scale mapping parameter may be used in a luminance-scale-parameter generating function to generate a predicted luminance scale parameter
- the chrominance scale mapping parameter may be used in a chrominance-scale-parameter generated function to generate a predicted chrominance scale parameter.
- an offset parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted offset parameter.
- LDR image data may be received 126, and an offset mapping parameter may be derived 128 from the LDR image data.
- the offset mapping parameter may be mapped 130 to an offset parameter prediction.
- the mapping 130 may comprise using the offset mapping parameter as an index into a look-up-table of candidate offset parameter values.
- the look-up-table of candidate offset parameter values may be signaled to a decoder within the bitstream, may be stored as a default look-up-table at the decoder or may otherwise be signaled to the decoder.
- the offset mapping parameter may be used in an offset-parameter generating function.
- the offset- parameter generating function may be signaled to a decoder within the bitstream, may be stored as a default generating function at the decoder or may otherwise be signaled to the decoder.
- the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter.
- the offset mapping parameter may index a luminance-offset- parameter look-up-table to determine a predicted luminance offset parameter, and the offset mapping parameter may index a chrominance-offset-parameter look-up-table to determine a predicted chrominance offset parameter.
- two offset mapping parameters may be generated: a luminance offset mapping parameter and a chrominance offset mapping parameter. In some of these embodiments, the two different mapping parameters may index into the same look-up-table.
- the luminance offset mapping parameter and the chrominance offset mapping parameter may index into a luminance-offset-parameter look-up-table and a chrominance- offset-parameter look-up-table, respectively.
- the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter
- the offset mapping parameter may be used in a luminance-offset- parameter generating function to generate a predicted luminance offset parameter.
- the offset mapping parameter may also be used in a chrominance-offset-parameter generating function to generated a predicted chrominance offset parameter.
- the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter
- two offset mapping parameters may be generated: a luminance offset mapping parameter and a chrominance offset mapping parameter.
- the two different mapping parameters may be used in the same offset parameter generating function to generate two offset parameters: a predicted luminance offset parameter and a predicted chrominance offset parameter.
- the luminance offset mapping parameter may be used in a luminance-offset-parameter generating function to generate a predicted luminance offset parameter
- the chrominance offset mapping parameter may be used in a chrominance-offset-parameter generated function to generate a predicted chrominance offset parameter.
- a scale parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted scale parameter which may then be corrected according to enhancement data.
- LDR image data may be received 120, and a scale mapping parameter may be derived 122 from the LDR image data.
- the scale mapping parameter may be mapped 124 to a scale parameter prediction.
- Correction data may be received 132 as part of high dynamic range (HDR) image data.
- the predicted scale parameter value determined 124 from the scale mapping parameter may be corrected 134 , also considered refined, in accordance with the HDR image data.
- the correction data may comprise a single correction which may be applied to both the predicted luminance scale parameter and the predicted chrominance scale parameter.
- the correction data may comprise a luminance correction which may be applied to the predicted luminance scale parameter and a chrominance correction which may be applied to the chrominance scale parameter.
- an offset parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted offset parameter which may then be corrected according to enhancement data.
- LDR image data may be received 126
- an offset mapping parameter may be derived 128 from the
- the offset mapping parameter may be mapped 130 to an offset parameter prediction.
- Correction data may be received 136 as part of high dynamic range (HDR) image data.
- the predicted offset parameter value determined 130 from the offset mapping parameter may be corrected 138, also considered refined, in accordance with the HDR image data.
- the correction data may comprise a single correction which may be applied to both the predicted luminance offset parameter and the predicted chrominance offset parameter.
- the correction data may comprise a luminance correction which may be applied to the predicted luminance offset parameter and a chrominance correction which may be applied to the chrominance offset parameter.
- both a scale parameter and a offset parameter may be determined.
- scale/ offset mapping parameter is to be understood to denote a scale mapping parameter, an offset mapping parameter and a parameter which may be used as both a scale and offset mapping parameter.
- a scale/ offset mapping parameter may be determined from LDR image data.
- a scale / offset mapping parameter may be determined for a block of LDR image data.
- the LDR image data within the block may be used to determine the scale / offset mapping parameter.
- select image data within the block may be used to determine the scale / offset mapping parameter.
- image data outside of the block may be used to determine the scale/ offset mapping parameter.
- the LDR data in the region of LDR image data used to determine the scale / offset mapping parameter may be referred to as the LDR parameter data.
- the LDR parameter data may comprise color components.
- the color components may comprise a luminance component and a chrominance component.
- the region size or region shape of LDR image data used as LDR parameter data may vary based on spatial location.
- the scale/ offset mapping parameter may be determined by computing a single value from the LDR parameter data.
- Exemplary values which may computed from the LDR parameter data may be the average of the LDR parameter data, the mean of the LDR parameter data, the summation of the LDR parameter data, the weighted average of the LDR parameter data, the weighted mean of the LDR parameter data, the weighted summation of the LDR parameter data, the median of the LDR parameter data, normalized summation of the LDR parameter data, block transform of the LDR parameter data, and other computations using the LDR parameter data.
- the scale / offset mapping parameter may determined by transforming the LDR parameter data. In some of these alternative embodiments, the scale/ offset mapping parameter may be related to one of the transform coefficients. In other of these alternative embodiments, the scale/ offset mapping parameter may be related to a subset of the transform coefficients.
- mapping parameter and predicted value is to be understood to denote either a scale mapping parameter and a predicted scale value or an offset mapping parameter and a predicted offset value.
- a mapping parameter may be used as an index into a look-up- table of candidate values to determine a predicted value.
- a mapping parameter may be used in a predicted value generating function to generate the predicted value .
- a first mapping parameter may be used as an index into a first look-up-table wherein entries in the first look-up-table correspond to a candidate-value look-up-table, and a second mapping parameter may be used as an index into the candidate-value look-up-table identified by the first mapping parameter.
- a first mapping parameter may be used as an index into a look-up- table of generating functions, and a second mapping parameter may be used in the indexed generating function to generated the predicted value.
- a scale value and/ or an offset value may be determined according to different mappings for different spatial regions of LDR image data.
- the method for mapping the mapping parameter to the predicted value may signaled from an encoder to a decoder.
- a mapping for example, a look-up-table or generating function, may be signaled from the encoder to the decoder.
- the mapping may be signaled on a slice-by-slice, picture-by-picture, sequence-by-sequence or other basis.
- a plurality of mappings may be pre-stored at the decoder, and an indicator selecting one of the pre-stored mappings may be explicitly signaled from the encoder to the decoder on a slice-by-slice , picture-by-picture, sequence-by-sequence or other basis.
- the mapping may be signaled according to the methods and systems described in U. S . Patent Publication No. 2007- 0201560 A l (Application No. 1 1 / 566,657) , filed on December
- the mapping between the mapping parameter and the prediction value may be different depending on the size of the region of LDR image data used for the LDR parameter data. In alternative embodiments, wherein the size of the region of LDR image data used for the LDR parameter data is not constant over an image frame, the mapping between the mapping parameter and the prediction value may be different depending on the size of the region of LDR image data used for the LDR parameter data. In alternative embodiments, wherein the size of the region of LDR image data used for the LDR parameter data is not constant over an image frame, the mapping between the mapping parameter and the prediction value may be different depending on the size of the region of LDR image data used for the LDR parameter data. In alternative embodiments, wherein the size of the region of
- the mapping between the mapping parameter and the prediction value may be the same for different region sizes, and a normalization factor may be applied to the predicted value based on the region size .
- the region size may be signaled explicitly in the enhancement-layer bitstream.
- the region size may be derived from other information in either the base layer or the enhancement layer. Exemplary derivations may be based on the transform size or the prediction mode determined from the bitstream.
- a scale parameter value in a current image frame (first image frame) of an image sequence may be predicted from the scale parameter values corresponding to previously transmitted plurality of image frames.
- LDR image data for a current image frame may be received 140.
- a scale correspondence indicator (first correspondence indicator) may be received 142 , and the scale correspondence indicator may be used to determine 144 a prediction of the current scale parameter value.
- the scale correspondence indicator may relate the current scale parameter value to a corresponding scale parameter value from a previously transmitted image frame.
- an offset parameter value in a current image frame of an image sequence may be predicted from the offset parameter values corresponding to previously transmitted image frames.
- LDR image data for a current image frame may be received 150.
- An offset correspondence indicator may be received 152 , and the offset correspondence indicator may be used to determine 154 a prediction of the current offset parameter value.
- the offset correspondence indicator may relate the current offset parameter value to a corresponding offset parameter value from a previously transmitted image frame.
- HDR prediction parameter values may be stored 160 for each previously decoded image frame .
- Exemplary HDR prediction parameters include offset parameters, scale parameters and other parameters which may be used in predicting HDR image data from LDR image data.
- the HDR prediction parameter values may be stored for a subset of previously decoded image frames.
- An HDR prediction parameter correspondence indicator (second correspondence indicator) may be received 162.
- the HDR prediction parameter correspondence indicator may be decoded from an enhancement-layer bitstream.
- the HDR prediction parameter correspondence indicator may determine 164 the location in the stored parameter values of the value which may be used as the current predicted HDR prediction parameter value.
- a parameter buffer may contain the parameter values for each 4-by-4 block in the previously reconstructed image frame .
- the enhancement-layer bitstream may signal two indices which define the location in the parameter buffer of previous parameter values to use for the parameter prediction. The two indices may be signaled explicitly in some embodiments. In alternative embodiments, the indices may be signaled relative to the location of the current block in the current frame.
- a second predicted HDR prediction parameter value may be refined according to residual information which may be received in and HDR enhancement-layer bitstream.
- HDR prediction parameter values may be stored 160 for each previously decoded image frame.
- Exemplary HDR prediction parameters include offset parameters, scale parameters and other parameters which may be used in predicting HDR image data from LDR image data.
- the offset parameters include offset parameters, scale parameters and other parameters which may be used in predicting HDR image data from LDR image data.
- HDR prediction parameter values may be stored for a subset of previously decoded image frames.
- An HDR prediction parameter correspondence indicator may be received 162.
- the HDR prediction parameter correspondence indicator may be decoded from an enhancement-layer bitstream.
- the HDR prediction parameter correspondence indicator may determine 164 the location in the stored parameter values of the value which may be used as the current predicted HDR prediction parameter value then extracting 166 the parameter value from storage.
- An HDR prediction parameter residual may be received 168.
- the HDR prediction parameter residual may be decoded from an enhancement-layer bitstream.
- the predicted HDR prediction parameter value may be corrected 170 according to the HDR prediction parameter residual.
- an HDR prediction parameter value in a current image frame of an image sequence may be predicted from the HDR prediction parameter values corresponding to previously transmitted image frames
- knowledge of the correspondence between pixel values in the LDR representation of the current, decoded frame and previous, decoded image data may be used to determine the predicted HDR prediction parameter value.
- the correspondence may be determined from parameters transmitted as part of the LDR bitstream, for example, motion vectors.
- the correspondence may be determined by comparison of the LDR pixel data for the current frame and a previous LDR image frame.
- a motion estimation technique may be used to determine the correspondence .
- the correspondence between pixels in a current block and a block in a previously decoded image frame may be described with a motion vector comprising two indices.
- the two indices may correspond to a vertical displacement and a horizontal displacement.
- the motion vector may be used explicitly for predicting the HDR prediction parameter value.
- the predicted HDR prediction parameter value at a pixel may be the HDR prediction parameter value of the corresponding pixel in the previous frame.
- a different predicted HDR prediction parameter value may be obtained at each pixel in a block being decoded.
- a single predicted HDR prediction parameter value may be determined based on the corresponding HDR prediction parameter values in the previous frame.
- Exemplary methods for combining the multiple parameter values may comprise a median computation, a mean computation, an average computation, a weighted mean computation, a weighted average computation, a voting calculation wherein the largest number of referenced parameter values in the previous locations may be used as the prediction and other combinations of the multiple parameter values.
- motion vector correspondence may be refined based on residual information decoded from an enhancement-layer bitstream.
- LDR image data may be received 180, and HDR image data may be received 182.
- Motion vector correspondence may be determined 184.
- motion vector correspondence may be determined 184 from parameters, for example motion vectors, transmitted with the LDR image data.
- motion vector correspondence may be determined 184 through a motion vector correspondence estimation process wherein the pixel values of the LDR data may be examined.
- a motion vector correspondence residual value may be decoded 186 from the HDR image data.
- the motion vector correspondence may be corrected 188 in accordance with the motion vector correspondence residual value.
- the corrected motion vector correspondence may be used to determine 190 a predicted
- a predicted HDR prediction parameter value may be corrected according to transmitted residual values.
- an HDR prediction parameter value residual may be decoded from an enhancement-layer bitstream, and a predicted HDR prediction parameter value may be corrected, or refined, in accordance with the HDR prediction parameter value residual.
- a first predicted HDR prediction parameter value and a second predicted HDR prediction parameter value may be determined.
- the first predicted HDR prediction parameter value may be a predicted scale value
- the second predicted HDR prediction parameter value may be a predicted offset value.
- a scale value residual may be determined, and an offset value may be determined.
- the predicted scale value may be refined, and the predicted offset value may be refined.
- a single residual value may be signaled, and a correction, or refinement, may be made to only one of these predicted values.
- the other predicted value may be adjusted based on the signaled residual value, characteristics of the decoded LDR image data and the other predicted value .
- a flag may be send within the enhancement-layer bitstream from an encoder to a decoder.
- the flag may be sent on a block-by- block, macro-block-by-macro-block, slice-by-slice, frame-by- frame, sequence-by-sequence or other basis.
- the value, or state, of the flag may determine the prediction method to be used for predicting the HDR prediction parameter value .
- Exemplary prediction methods include those described herein.
- This apparatus comprises an LDR image data receiving section, for receiving a plurality of LDR image data; an LDR image data generating section, for generating a first predicted HDR prediction prarameter value based on the plurality of LDR image data; an LDR image data extracting section, for extracting an LDR image value from the plurality of LDR image data; and an HDR image element prediction forming section for forming an HDR image element prediction using the first predicted HDR prediction parameter value and the LDR image value.
- the above apparatus can also include a computer program used for the prediction of images comprising multiple dynamic range layers, on a computer system.
- This computer program is stored on a storage medium such as an optical disc or a magnetic disk.
- the storage medium containing the content data and the computer programs realizing the functions of the content processing device is by no means limited to the optical disc, which may be a CD-ROM (compact disc read-only memory) , MO (magneto-optical disc) , MD (MiniDisc) , or DVD (digital versatile disc) or the magnetic disk which may be a FD (flexible disk) or hard disk.
- Examples of such a storage medium include tapes, such as magnetic tapes and cassette tapes; card storage media, such as IC (integrated circuit) cards and optical cards; and semiconductor memories, such as mask ROMs, EPROMs (erasable programmable ROMs) , EEPROMs (electrically erasable programmable ROMs) , and flash ROMs.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Aspects of the present invention relate to systems and methods for predicting a prediction parameter which may be used in the prediction of high dynamic range image elements from low dynamic range image data. Such systems and methods can comprise receiving a plurality of low dynamic range (LDR) image data; generating a first predicted high dynamic range (HDR) prediction parameter value based on said plurality of LDR image data, extracting an LDR image value from said plurality of LDR image data and forming an HDR image element prediction using said first predicted HDR prediction parameter value and said LDR image value. The methods can be included in an apparatus which can include a computer program stored on a storage medium.
Description
DESCRIPTION
METHODS AND APPARATUS FOR INTER-LAYER IMAGE PREDICTION PARAMETER DETERMINATION
TECHNICAL FIELD Embodiments of the present invention comprise methods and systems for inter-layer image prediction.
BACKGROUND ART
Typically, there are systems and methods for using a low dynamic range (LDR) video sequence to predict a high dynamic range (HDR) . This may be referred to as inter-layer prediction and can include inter-layer prediction mechanism for HDR video coding.
An analogous process to inter-layer prediction for high dynamic range video coding is inter-layer prediction for bit- depth scalability. For bit-depth scalability, the base layer of a video bitstream may contain a representation of the video sequence at a reduced bit-depth. For example, the base layer may contain an eight-bit representation of the sequence, while the enhancement layer of the bitstream may contain a ten-bit representation. In some scenarios, more than two layers may be used. In some scenarios, an eight-bit version may represent the eight most significant bits of the higher
bit-depth sequence . The higher bit-depth version may therefore be predicted by multiplying (or equivalently scaling) the decoded lower bit-depth data to the higher bit-depth. In this specific example, the eight-bit data may be decoded and subsequently scaled by a factor of four to predict the ten-bit data. This scaling may be done in either the intensity or transform domain, depending on the application.
High dynamic range video coding may be considered a more general case of bit-depth scalability. The base layer and enhancement layer may contain data represented with different bit-depths. However, the base layer may not be constrained to represent the most significant bits of the enhancement-layer data. The base-layer data may contain a lower bit-depth representation of the high dynamic range sequence, and this lower bit-depth may not always correspond to the most significant bits of the corresponding higher bit- depth representation.
DISCLOSURE OF THE INVENTION Some embodiments of the present invention comprise methods and systems for prediction of images comprising multiple dynamic range layers. Some embodiments of the present invention comprise methods and systems for determining a high dynamic range (HDR) prediction parameter which may be used to predict HDR image data from low
dynamic range (LDR) image data. In some embodiments of the present invention, an HDR prediction parameter may be predicted based on analysis of decoded LDR image data. In some embodiments of the present invention, an HDR prediction parameter may be predicted from the HDR prediction parameters corresponding to previously transmitted image frames.
The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS Fig. IA is a chart showing an exemplary embodiment of the present invention comprising prediction with a scaled and offset LDR image element;
Fig. I B is a chart showing an exemplary embodiment of the present invention comprising scaling and offsetting decoded image elements for HDR prediction;
Fig. 2 is a chart showing an exemplary embodiment of the present invention comprising conversion to an alternative color space;
Fig. 3 is a chart showing an exemplary embodiment of the present invention comprising scaling an LDR image
element according to HDR bitstream data;
Fig. 4 is a chart showing an exemplary embodiment of the present invention comprising scaling and applying an offset to an LDR image element according to HDR bitstream data;
Fig. 5 is a chart showing an exemplary embodiment of the present invention comprising scaling LDR transform coefficients for HDR prediction;
Fig. 6 is a chart showing an exemplary embodiment of the present invention comprising applying an offset to LDR transform coefficients for HDR prediction;
Fig. 7 is a chart showing an exemplary embodiment of the present invention comprising scaling LDR transform coefficients and applying an offset to LDR transform coefficients for HDR prediction;
Fig. 8 is a chart showing an exemplary embodiment of the present invention comprising scaling and applying an offset to color-transformed image elements for HDR prediction; Fig. 9 is a chart showing an exemplary embodiment of the present invention comprising separate scaling and offset operations for luminance and chrominance elements;
Fig. 10 is a chart showing an exemplary embodiment of the present invention comprising using a scale mapping parameter to predict a scale parameter value;
Fig. 1 1 is a chart showing an exemplary embodiment of the present invention comprising using an offset mapping parameter to predict an offset parameter value;
Fig. 12 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted scale parameter value in accordance with a residual value received as part of the HDR enhancement-layer bitstream;
Fig. 13 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted offset parameter value in accordance with a residual value received as part of an HDR enhancement-layer bitstream;
Fig. 14 is a chart showing an exemplary embodiment of the present invention comprising a scale correspondence indicator for predicting a scale parameter value; Fig. 15 is a chart showing an exemplary embodiment of the present invention comprising an offset correspondence indicator for predicting an offset parameter value;
Fig. 16 is a chart showing an exemplary embodiment of the present invention comprising using an HDR prediction parameter correspondence indicator to index stored HDR prediction parameter values from previous image frames to predict a current HDR prediction parameter value;
Fig. 17 is a chart showing an exemplary embodiment of the present invention comprising correcting a predicted HDR prediction parameter value in accordance with a residual
value received as part of an HDR enhancement-layer bitstream; and
Fig. 18 is a chart showing an exemplary embodiment of the present invention comprising correcting a motion vector correspondence value in accordance with a residual value received as part of an HDR enhancement-layer bitstream.
BEST MODE FOR CARRYING OUT THE INVENTION
Some embodiments of the present invention comprise systems and methods for using a low dynamic range (LDR) video sequence to predict a high dynamic range (HDR) version of the image data. This may be referred to as inter-layer prediction in this application. Some embodiments of the present invention comprise a spatially-varying inter-layer prediction mechanism for HDR video coding. Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that operates in the color spaces used for video compression and transmission. Some embodiments use gamma-corrected color spaces. Exemplary embodiments may use xvYCC and YCbCr color spaces. Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that may be disabled spatially. Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that is multiplication free.
Some embodiments of the present invention comprise an inter-layer prediction mechanism for HDR video coding that may be used in a single-loop decoder. Some embodiments may also be incorporated into multi-loop designs. Some embodiments of the present invention comprise an inter-layer prediction technique for high-dynamic range video coding. Some aspects of some embodiments comprise elements described in U . S . Patent Publication No. 2007- 0223813 A l (Application No. 1 1 / 362 , 571 ) , filed on Feb. 24 , 2006, by Christopher Andrew Segall, which is hereby incorporated herein by reference. Some embodiments of the present invention comprise a method for projecting decoded low dynamic range data to the high dynamic range coding space. This process may be referred to as inter-layer prediction.
Some embodiments of the present invention may be described with reference to Figure IA. In these embodiments, a high dynamic range (HDR) image may be received 100. A corresponding low dynamic range (LDR) image or a plurality of LDR images may also be received, or, alternatively, created
101 from the HDR image. The LDR image may be created 101 through a tone-scale operation, a conversion function or some other method. The LDR image may then be predicted, transformed, quantized and encoded 102 , as is well known in the art. In a few exemplary embodiments, the LDR image may
be transformed using a discrete cosine transform (DCT) , a wavelet transform or by other common transformation methods. The prediction, transformation, quantization and encoding processes may then be substantially reversed 103 to provide a decoded LDR image as would be decoded at a typical decoder. Typically, a de-quantization process is lossy and therefore does not produce an exact copy of the originally encoded image. Other processes may also affect the reproduction of the original LDR image . Regardless, the decoded LDR image may be processed by one or more of the following methods: color conversion, scaling 104 and offsetting 105. The decoded, processed LDR image may now be used to create 106 a residual HDR image. This may be performed by subtracting the decoded, processed LDR image from the original HDR image . Other methods may also be used.
The residual HDR image may then be transformed, quantized and encoded 107, or otherwise prepared, for transmission to a destination. In some embodiments, this step may comprise embedding the encoded residual HDR image into an HDR, or enhancement-layer, bitstream. Information related to the color conversion, scaling and offset operations may also be encoded and embedded 108 in the HDR, or enhancement-layer, bitstream. The HDR/ enhancement-layer bitstream may then be transmitted
109 to a destination. An LDR/ base-layer bitstream may also be transmitted 1 10 to the destination. In some embodiments, the LDR/ base-layer bitstream may also comprise a transformed, quantized and encoded LDR image. A decoder receiving the LDR/ base-layer bitstream may then decode the LDR/ base-layer image. A decoder receiving the LDR/ base-layer bitstream and the HDR/ enhancement- layer bitstream may decode both the LDR/ base-layer image and the HDR/ enhancement-layer image. Embodiments of the present invention comprise methods and systems for encoding and decoding images in this framework and similar scenarios.
Some embodiments of the present invention may be described with reference to Figure I B. In these embodiments, a base-layer decoder may receive base-layer data, such as from a base-layer bitstream 2. The base-layer decoder may- decode 6 a base-layer block, or other image element, and represent it in the spatial domain. Some embodiments may comprise full decoding of the block, including a prediction process followed by residual refinement. Some embodiments may comprise reconstruction of the residual only. In some embodiments, the spatial information in the base layer may be used to predict the high dynamic range signal. Some embodiments may comprise scaling 7 the base-layer information. Some embodiments may also comprise adding an offset 8 to the base-layer information. Some embodiments
may comprise both scaling 7 and adding an offset 8. Once scaling 7 and/ or adding an offset 8 are performed on the decoded base-layer information, that scaled, offset information may be used to predict 9 an enhancement layer, such as a higher dynamic range (HDR) layer. In some embodiments, scaling 7 and offset 8 data may be extracted from an enhancement-layer bitstream 4. In some embodiments, subsequent refinement may be decoded from the enhancement-layer bitstream 4. Some embodiments of the present invention may be described with reference to Figure 2. In these embodiments, a decoder may receive base-layer data 10 from which a block, or other image element, may be decoded 12 into spatial image data. This spatial image data may then be converted 13 to an alternative color space. This converted data may then be scaled 14 and/ or offset 15. Scaling and offset operations may be performed according to instructions and/ or data received from an enhancement bitstream 1 1. This converted, scaled and/ or offset data may then be converted 16 back to the coding color space. Once converted back to the coding color space, the scaled and/ or offset data may be used to predict 17 an enhancement layer, such as a higher dynamic range (HDR) layer.
Some embodiments of the present invention may be described with reference to Figure 2. In these embodiments,
a decoder may receive base-layer data 10 from which a block, or other image element, may be decoded 12 into spatial image data. This spatial image data may then be converted 13 to an alternative color space. This converted data may then be scaled 14 and/ or offset 15. Scaling and offset operations may be performed according to instructions and/ or data received from an enhancement bitstream 1 1. This converted, scaled and/ or offset data may then be converted 16 back to the coding color space. Once converted back to the coding color space, the scaled and/ or offset data may be used to predict
17 an enhancement layer, such as a higher dynamic range (HDR) layer.
Some embodiments of the present invention may be described with reference to Figure 3. In these embodiments, LDR/ base-layer image data may be received 30 and corresponding HDR/ enhancement-layer data also may be received 31. An LDR/ base-layer block, or other image element, then may be decoded 32 from the LDR/ base-layer data. The decoded LDR/ base-layer image element then may be scaled 33. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to, or a function of, image characteristics comprising spatial location, luminance data, chrominance data and other data. The scaled, decoded LDR/ base-layer image may then be used to
predict 34 a corresponding HDR block or other image element. In some embodiments, the scaled, decoded LDR/ base-layer image element may be added to a corresponding decoded residual image element to form an HDR/ enhancement-layer (residually-corrected) image element.
Some embodiments of the present invention may be described with reference to Figure 4. In these embodiments, LDR/ base-layer image data may be received 40 and corresponding HDR/ enhancement-layer data also may be received 4 1. An LDR/ base-layer block, or other image element, then may be decoded 42 from the LDR/ base-layer image data. The decoded LDR/ base-layer image element may be then scaled 43. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data. An offset may then be added 44 to the scaled LDR image element. Offset data may be carried in the corresponding HDR/ enhancement-layer data. Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
The scaled, offset and decoded LDR/base-layer image may then be used to predict 45 a corresponding HDR block or
other image element. In some embodiments, the scaled, offset and decoded LDR/ base-layer image element may be added to a corresponding decoded residual image element to form an HDR/ enhancement-layer image element. Some embodiments of the present invention may be described with reference to Figure 5. In these embodiments, an LDR/ base-layer image comprising LDR transform coefficients may be received 50, and corresponding HDR/ enhancement-layer data also may be received 51 . The LDR/ base-layer image transform coefficients may then be scaled 52. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of LDR transform coefficients may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data. The scaled
LDR/ base-layer transform coefficients may then be used to predict 53 transform coefficients for a corresponding HDR block or other image element.
Some embodiments of the present invention may be described with reference to Figure 6. In these embodiments, an LDR/ base-layer image comprising LDR transform coefficients may be received 60, and corresponding HDR/ enhancement-layer data also may be received 61 . The LDR/ base-layer image transform coefficients may then be offset 62. Offset data may be carried in the corresponding
HDR/ enhancement-layer data 61. Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data. The offset LDR/ base-layer transform coefficients may then be used to predict 63 transform coefficients for a corresponding HDR block or other image element.
Some embodiments of the present invention may be described with reference to Figure 7. In these embodiments, an LDR/ base-layer image comprising LDR transform coefficients may be received 70, and corresponding HDR/ enhancement-layer data also may be received 71. The LDR/ base-layer image transform coefficients may then be scaled 72. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of
LDR transform coefficients may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data. The scaled LDR/ base-layer image transform coefficients may then be offset 73. Offset data may be carried in the corresponding
HDR/ enhancement-layer data 7 1 . Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data. The scaled, offset LDR/ base-layer transform coefficients may then be used to
predict 74 transform coefficients for a corresponding HDR block or other image element.
Some embodiments of the present invention may be described with reference to Figure 8. In these embodiments, LDR/ base-layer image data may be received 80, and corresponding HDR/ enhancement-layer data also may be received 81 . An LDR/ base-layer block, or image element, may then be decoded 82 from the LDR/ base layer image data. The decoded LDR/ base-layer image element may then be converted 83 or transformed to an alternative color format or color space. While in this alternative color space, the LDR image element may be scaled 84. This scaling may be performed according to data embedded in the HDR/ enhancement-layer data. Scaling of individual image elements may be related to or a function of image characteristics comprising spatial location, luminance data, chrominance data and other data. Also, while in the alternative color space, an offset may then be added 85 to the scaled, color-converted LDR image element. Offset data may be carried in the corresponding HDR/ enhancement-layer data.
Offset data may vary between image elements and may be dependent on image characteristics comprising spatial location, luminance data, chrominance data and other data.
The scaled and/ or offset and color-converted LDR/ base- layer image may then be converted back 86 to the coding
color space. This scaled and/or offset, coding-color-space LDR/ base-layer image may then be used to predict 87 a corresponding HDR block or image element.
Some embodiments of the present invention may be described with reference to Figure 9. In these embodiments,
LDR/ base-layer image data may be received 90, and corresponding HDR/ enhancement-layer data may be also received 91 . An LDR/ base-layer block, or image element, may then be decoded 92 from the LDR/ base-layer image data. In these embodiments, the decoded LDR/ base-layer image may comprise separable luminance and chrominance values. In some embodiments, luminance values may be scaled 93 in relation to their spatial position in the image. Other factors may also affect the luminance value scaling operation. In some embodiments, these luminance values may be offset 94.
The offset operation may also be related to the spatial position of the luminance value . In some embodiments, the chrominance values of the decoded LDR/ base-layer image may be scaled 95. This chrominance scaling may also be related to the spatial position of the chrominance value . In some embodiments, chrominance values may also be offset 96. The chrominance value offset may be related to a luminance offset, a chrominance value or scaling factor and/ or a spatial position of the chrominance value. Other factors may also affect the chrominance offset.
Once the luminance and chrominance values are scaled and/ or offset, they may be used to predict 97 a corresponding HDR/ enhancement-layer image element.
In some embodiments of the present invention, the inter- layer prediction process may be controlled at a fine granularity. As a specific example, the scaling and offset factors may vary on a 4x4 block basis. That is, for every 4x4 block in the image, an encoder may signal the appropriate scaling and offset factor. Additionally, an encoder may enable and disable inter-layer prediction on a block-by-block basis .
This allows, for example, the high dynamic range image to be predicted from the low dynamic range image in a portion of the frame while predicted with alternative mechanisms in other spatial regions. Specifically, intra-frame and inter- frame prediction mechanisms may be used in these other spatial regions.
Exemplary Scaling Embodiments
Some embodiments of the present invention comprise inter-layer prediction methods that are multiplication free . In these embodiments, the base-layer data may be decoded, and the decoded samples may be processed with a sequence of binary shifts and adds. In some embodiments, this may be accomplished with a process described by equation 1 :
HDR(x, y) = ∑a, * LDR(x, y) « i, { l )
Vi
where HDR and LDR are, respectively, the high dynamic range and low dynamic range version of the image sequence , x and y denote the spatial location within the image frame, and cii is a binary indicator that belongs to the set {- 1 ,0, 1}. Some embodiments may select I=(0, 1 ,2 ,3}.
Alternative Exemplary Scaling Embodiments Some inter-layer prediction embodiments comprise an offset in the inter-layer prediction process. Some embodiments may comprise a process described in equation 2 :
where Offset(x,y) is the offset value. In some embodiments, the offset value may be signaled with the scaling values. Alternatively, it may be signaled as part of a residual refinement process.
Spatial Adaptivity
In some embodiments, control of the prediction process may be enabled at fine granularity. For example, when the base-layer video codec employs a block-based structure , the
inter-layer prediction process may vary the scaling and offset parameters on a similar block grid. In some embodiments, this may be achieved by sending scaling and/ or offset information from the encoder to the decoder within an enhancement bitstream.
In some signaling embodiments, the scaling factors may be transmitted differentially. That is, the scale factor may be predicted from previously received scale factors. Then, a correction may be transmitted in the bitstream. Some embodiments may predict the scale factor from the upper- or left-most neighbor to the current block. Alternatively, some embodiments may predict the scale factor as the minimum value of the upper- or left-most neighbor.
In addition, in some embodiments, the encoder may signal the correction value as a function of the upper and left-most neighbors. For example, the encoder and decoder may use a specific context or state for signaling when the neighbors have the same scale factor. An alternative state may be used when the neighbors have different scale factors.
High-Level Syntax
Some embodiments of the present invention comprise high dynamic range video coding where the scale factor is the same throughout an image region. To accommodate these cases, high-level information may also be transmitted from
the encoder to the decoder. This high-level information may disable the transmission of scaling and/ or offset parameters on a block-by-block or region-by-region basis. For the case that transmission of the parameters is disabled, the high level information may comprise the scaling and/ or offset information to be used. In some embodiments, this high-level signaling may occur on a macroblock, slice, picture or sequence basis.
Transform Domain Processing
In some embodiments of the present invention, the inter- layer prediction process operates on intensity data. That is, the information may be decoded and converted to the spatial domain by reversing any transform used for signaling. In alternative prediction embodiments, the scaling and offset operations may be directly applied in the transform domain. In these embodiments, the transform coefficients may be de- quantized and then scaled by scale factors. In some embodiments, transform coefficients may be processed differently depending on their frequency characteristics. For example, in some embodiments, the scaling operation may be applied solely to the AC coefficients while the offset operation may affect the DC component. In some embodiments, different scaling and offset operations may be signaled for different coefficients or coefficient types.
Some embodiments of the present invention may comprise a video codec that may adaptively switch between transform domain and spatial domain prediction mechanisms. In some embodiments, this switch may be signaled on a sequence, frame or slice basis. In some embodiments, this switch may operate at finer granularity, such as a block or macro-block.
Color and Color Space Issues An issue in scalable, high dynamic range video coding is the management of color. In some embodiments of the present invention, a color transform may be used prior to inter-layer prediction. This may address the fact that most color spaces used for video coding are not iso-luminant. For example , a video codec typically transmits data in the YCbCr color space with code-word mappings defined in International Telecommunication Union, "Parameter Values for the HDTV Standards for Production and International Programme Exchange," ITU-R BT.709-5, April, 2002. Some embodiments of the present invention may perform an inter-layer prediction process in a color space closely related to the coding color space. In some exemplary embodiments, the color transform may be expressed in the following equation 3:
1 Y LDR = xY LDR '
L _ ^ ^ LDR
* LDR + ^" LDR + ^ ^ LDR
Y LrDR y =
^LDR + ^rLDR + ^" LDR (3)
where YLDR , CbLDR and CΓLDR are the luma and chroma components in the low dynamic range image sequence, respectively. Then, the scaling and offset process may be applied to YLDR to generate YHDR. Finally, the inter-predicted region may be computed with the following equation 4:
^HDR - »
where CbHDR and CTHDR are predictions for the color components in the high dynamic range layer. In some embodiments wherein YLDR , CbLDR and CTLDR may not be represented at the same resolution, the components may be resampled. In some exemplary embodiments, applications may down-sample the luma component when the chroma components are stored at a lower resolution. Alternatively, the chroma components may
be up-sampled to match the resolution of the luma component.
Alternative Color and Color Space Issues
In some embodiments of the present invention, inter- layer prediction may operate directly on the decoded data without employing a color transform. In some exemplary embodiments, the prediction process may be expressed by the following equation 5:
Y HDR (*> y) = Scale(x, y, c) * YWR (x, y) + Offset(x, y, x), Cb HDR (*> y) = Scale(x, y,c)* CbLDR (x,y) + Offset (x, y, x), Cr HDR (*> y) = Scale(x, y, c) * CrWR (x, y) + Offset (x, y, x), (5)
where the scaling and offset parameters may now be a function of both spatial location and chroma component. That is, the reconstructed luma and chroma values may be scaled with different scale factors.
In some exemplary inter-prediction processes, the luma and chroma values may be scaled with the same scale factor but with different offsets. This may be expressed with the following equation 6:
Y HDR (x > y) = Scale(x, y) * YLDR (x, y) + Offset (x, y, x), cb HDR (x > y) = Scale(x, y) * CbWR (x, y) + Offset(x, y, x),
CrHDR(x,y) = Scale(x,y)* CrLDR(x,y) + Offset(x,y,x), (5)
In these embodiments, the scale factor may not depend
on the chroma component. In some embodiments, the encoder may transmit the offsets within the enhancement- layer bitstream.
In other exemplary embodiments of the inter-prediction process, the luma and chroma values may be scaled with the same scale factor, and the offset for the chroma values may be dependent on the offset of the luma values as well as the decoded image data. This relationship may be expressed in the following equation 7:
Y HDR (*, y) = Scale(x, y) * YLDR (x, y) + Offset (x, y), cb HDR (*> y) = Scale(x, y) * Cb WR (x, y) + f (Offset (x, y), Cb WR (x, y), YWR (x, y)),
CrHDR(x,y) = Scale(x,y)* CrLDR(x,y) + f(O_ffset(x,y),CrLDR(x,y),YLDR(x,y)), (7)
where f(' , ' , ') denotes a mapping operation. An exemplary mapping operation may be expressed as equation 8:
f(Offse1(x,y),ALDR(x,y\YLDR(x,y)) = Offset(x,y) (8)
YwR(χ>y) '
where ALDR{x,y) denotes an arbitrary color component, such as Cb or Cr.
In some embodiments, the chroma and luma components may be represented on different sampling grids. In some of these embodiments, the chroma and luma data may be resampled to the same resolution. In some embodiments, a
different mapping process may be employed. In some exemplary embodiments, the mapping relationship may be expressed as equation 9 :
where Avg( ) denotes the mean operator. In another exemplary embodiment, the mean may be replaced with a summation operation. In other embodiments, non-linear operations such as the median, min and max operations may be beneficial.
In some exemplary embodiments, the mean operator (or an alternative operator) may be performed in a different domain than that of the Offset variable. In some exemplary embodiments, the mean operation may be computed in the transform domain by operating solely on the DC coefficient. Similarly, in embodiments wherein the spatial resolutions of the chroma and luma coefficients are not matched , the mean operation may be computed by analyzing multiple DC coefficients in the luma base layer.
High Dynamic Range Prediction Parameter Determination
Some embodiments of the present invention may comprise elements for determining a predicted value for a high dynamic range (HDR) prediction parameter. Some
embodiments of the present invention may comprise elements for refining a predicted HDR prediction parameter value. Exemplary HDR prediction parameters include scale parameters, offset parameters and other parameters which may be used in predicting HDR image data from low dynamic range (LDR) image data. While the embodiments described in this section may be described in relation to scale parameters and offset parameters, it will be understood by one of ordinary skill in the art, that the methods and systems embodied in these descriptions may be used for determining other HDR prediction parameter values. These descriptions are by way of illustration and not limitation.
In some embodiments of the present invention, a scale parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted scale parameter. Some of these embodiments may be described in relation to Figure 10. LDR image data may be received 120, and a scale mapping parameter may be derived 122 from the LDR image data. The scale mapping parameter may be mapped 124 to a scale parameter prediction. In some embodiments, the mapping 124 may comprise using the scale mapping parameter as an index into a look-up-table of candidate scale parameter values. In some of these embodiments of the present invention, the look-up-table of candidate scale parameter values may be signaled to a
decoder within the bitstream, may be stored as a default lookup-table at the decoder or may otherwise be signaled to the decoder. In alternative embodiments, the scale mapping parameter may be used in a scale-parameter generating function. In some of these embodiments of the present invention, the scale-parameter generating function may be signaled to a decoder within the bitstream, may be stored as a default generating function at the decoder or may otherwise be signaled to the decoder. In some embodiments of the present invention, the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter. In some of these embodiments, the scale mapping parameter may index a luminance-scale- parameter look-up-table to determine a predicted luminance scale parameter, and the scale mapping parameter may index a chrominance-scale-parameter look-up-table to determine a predicted chrominance scale parameter. In alternative embodiments, two scale mapping parameters may be generated: a luminance scale mapping parameter and a chrominance scale mapping parameter. In some of these embodiments, the two different mapping parameters may index into the same look-up-table. In alternative embodiments, the luminance scale mapping parameter and the chrominance scale mapping parameter may index into a luminance-scale-parameter look-up-table and a chrominance-
scale-parameter look-up-table, respectively.
In alternative embodiments of the present invention, wherein the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter, the scale mapping parameter may be used in a luminance-scale- parameter generating function to generate a predicted luminance scale parameter. The scale mapping parameter may also be used in a chrominance-scale-parameter generating function to generated a predicted chrominance scale parameter. In still alternative embodiments of the present invention, wherein the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter, two scale mapping parameters may be generated: a luminance scale mapping parameter and a chrominance scale mapping parameter. In some of these alternative embodiments, the two different mapping parameters may be used in the same scale-parameter generating function to generate two scale parameters: a predicted luminance scale parameter and a predicted chrominance scale parameter. In yet alternative embodiments, the luminance scale mapping parameter may be used in a luminance-scale-parameter generating function to generate a predicted luminance scale parameter, and the chrominance scale mapping parameter may be used in a chrominance-scale-parameter generated function to generate a predicted chrominance scale parameter.
In some embodiments of the present invention, an offset parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted offset parameter. These embodiments may be described in relation to Figure 1 1. LDR image data may be received 126, and an offset mapping parameter may be derived 128 from the LDR image data. The offset mapping parameter may be mapped 130 to an offset parameter prediction. In some embodiments, the mapping 130 may comprise using the offset mapping parameter as an index into a look-up-table of candidate offset parameter values. In some of these embodiments of the present invention, the look-up-table of candidate offset parameter values may be signaled to a decoder within the bitstream, may be stored as a default look-up-table at the decoder or may otherwise be signaled to the decoder. In alternative embodiments, the offset mapping parameter may be used in an offset-parameter generating function. In some of these embodiments of the present invention, the offset- parameter generating function may be signaled to a decoder within the bitstream, may be stored as a default generating function at the decoder or may otherwise be signaled to the decoder.
In some embodiments of the present invention, the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter. In some of these embodiments,
the offset mapping parameter may index a luminance-offset- parameter look-up-table to determine a predicted luminance offset parameter, and the offset mapping parameter may index a chrominance-offset-parameter look-up-table to determine a predicted chrominance offset parameter. In alternative embodiments, two offset mapping parameters may be generated: a luminance offset mapping parameter and a chrominance offset mapping parameter. In some of these embodiments, the two different mapping parameters may index into the same look-up-table. In alternative embodiments, the luminance offset mapping parameter and the chrominance offset mapping parameter may index into a luminance-offset-parameter look-up-table and a chrominance- offset-parameter look-up-table, respectively. In alternative embodiments of the present invention, wherein the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter, the offset mapping parameter may be used in a luminance-offset- parameter generating function to generate a predicted luminance offset parameter. The offset mapping parameter may also be used in a chrominance-offset-parameter generating function to generated a predicted chrominance offset parameter. In still alternative embodiments of the present invention, wherein the offset parameter may comprise a luminance offset parameter and a chrominance offset
parameter, two offset mapping parameters may be generated: a luminance offset mapping parameter and a chrominance offset mapping parameter. In some of these alternative embodiments, the two different mapping parameters may be used in the same offset parameter generating function to generate two offset parameters: a predicted luminance offset parameter and a predicted chrominance offset parameter. In yet alternative embodiments, the luminance offset mapping parameter may be used in a luminance-offset-parameter generating function to generate a predicted luminance offset parameter, and the chrominance offset mapping parameter may be used in a chrominance-offset-parameter generated function to generate a predicted chrominance offset parameter.
Some embodiments of the present invention may be described in relation to Figure 12. In these embodiments, a scale parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted scale parameter which may then be corrected according to enhancement data. LDR image data may be received 120, and a scale mapping parameter may be derived 122 from the LDR image data. The scale mapping parameter may be mapped 124 to a scale parameter prediction. Correction data may be received 132 as part of high dynamic range (HDR) image data. The predicted scale parameter value determined 124 from the scale mapping parameter may be corrected 134 , also
considered refined, in accordance with the HDR image data. In some embodiments wherein the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter, the correction data may comprise a single correction which may be applied to both the predicted luminance scale parameter and the predicted chrominance scale parameter. In alternative embodiments wherein the scale parameter may comprise a luminance scale parameter and a chrominance scale parameter, the correction data may comprise a luminance correction which may be applied to the predicted luminance scale parameter and a chrominance correction which may be applied to the chrominance scale parameter.
Some embodiments of the present invention may be described in relation to Figure 13. In these embodiments, an offset parameter may be derived by analyzing decoded, low dynamic range (LDR) image data to determine a predicted offset parameter which may then be corrected according to enhancement data. LDR image data may be received 126 , and an offset mapping parameter may be derived 128 from the
LDR image data. The offset mapping parameter may be mapped 130 to an offset parameter prediction. Correction data may be received 136 as part of high dynamic range (HDR) image data. The predicted offset parameter value determined 130 from the offset mapping parameter may be
corrected 138, also considered refined, in accordance with the HDR image data. In some embodiments wherein the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter, the correction data may comprise a single correction which may be applied to both the predicted luminance offset parameter and the predicted chrominance offset parameter. In alternative embodiments wherein the offset parameter may comprise a luminance offset parameter and a chrominance offset parameter, the correction data may comprise a luminance correction which may be applied to the predicted luminance offset parameter and a chrominance correction which may be applied to the chrominance offset parameter.
In some embodiments of the present invention, both a scale parameter and a offset parameter may be determined.
The following description of methods and system for determining a mapping parameter may apply to either a scale mapping parameter or an offset mapping parameter. In the description, the term scale/ offset mapping parameter is to be understood to denote a scale mapping parameter, an offset mapping parameter and a parameter which may be used as both a scale and offset mapping parameter.
In some embodiments of the present invention, a scale/ offset mapping parameter may be determined from LDR image data. In some embodiments, a scale / offset mapping
parameter may be determined for a block of LDR image data. In some of these embodiments, the LDR image data within the block may be used to determine the scale / offset mapping parameter. In alternative embodiments, select image data within the block may be used to determine the scale / offset mapping parameter. In still alternative embodiments, image data outside of the block may be used to determine the scale/ offset mapping parameter. The LDR data in the region of LDR image data used to determine the scale / offset mapping parameter may be referred to as the LDR parameter data. In some embodiments, the LDR parameter data may comprise color components. In some of these embodiments, the color components may comprise a luminance component and a chrominance component. In some embodiments of the present invention, the region size or region shape of LDR image data used as LDR parameter data may vary based on spatial location.
In some embodiments of the present invention, the scale/ offset mapping parameter may be determined by computing a single value from the LDR parameter data.
Exemplary values which may computed from the LDR parameter data may be the average of the LDR parameter data, the mean of the LDR parameter data, the summation of the LDR parameter data, the weighted average of the LDR parameter data, the weighted mean of the LDR parameter data,
the weighted summation of the LDR parameter data, the median of the LDR parameter data, normalized summation of the LDR parameter data, block transform of the LDR parameter data, and other computations using the LDR parameter data. In alternative embodiments, the scale / offset mapping parameter may determined by transforming the LDR parameter data. In some of these alternative embodiments, the scale/ offset mapping parameter may be related to one of the transform coefficients. In other of these alternative embodiments, the scale/ offset mapping parameter may be related to a subset of the transform coefficients.
The following description of methods and system for mapping a mapping parameter to a predicted value may apply to either a scale mapping parameter and a predicted scale value or an offset mapping parameter and a predicted offset value. In the description, the terms mapping parameter and predicted value is to be understood to denote either a scale mapping parameter and a predicted scale value or an offset mapping parameter and a predicted offset value. In some embodiments of the present invention, a mapping parameter may be used as an index into a look-up- table of candidate values to determine a predicted value. In alternative embodiments, a mapping parameter may be used in a predicted value generating function to generate the predicted value . In yet alternative embodiments of the
present invention, a first mapping parameter may be used as an index into a first look-up-table wherein entries in the first look-up-table correspond to a candidate-value look-up-table, and a second mapping parameter may be used as an index into the candidate-value look-up-table identified by the first mapping parameter. In still alternative embodiments, a first mapping parameter may be used as an index into a look-up- table of generating functions, and a second mapping parameter may be used in the indexed generating function to generated the predicted value.
In some embodiments of the present invention, a scale value and/ or an offset value may be determined according to different mappings for different spatial regions of LDR image data. In some embodiments of the present invention, the method for mapping the mapping parameter to the predicted value may signaled from an encoder to a decoder. In some embodiments, a mapping, for example, a look-up-table or generating function, may be signaled from the encoder to the decoder. In some embodiments, the mapping may be signaled on a slice-by-slice, picture-by-picture, sequence-by-sequence or other basis. In alternative embodiments, a plurality of mappings may be pre-stored at the decoder, and an indicator selecting one of the pre-stored mappings may be explicitly signaled from the encoder to the decoder on a slice-by-slice ,
picture-by-picture, sequence-by-sequence or other basis. In yet alternative embodiments of the present invention, the mapping may be signaled according to the methods and systems described in U. S . Patent Publication No. 2007- 0201560 A l (Application No. 1 1 / 566,657) , filed on December
04, 2006, by Christopher Andrew Segall, Louis Joseph Kerofsky and Shawmin Lei, which is hereby incorporated herein by reference.
In some embodiments of the present invention, wherein the size of the region of LDR image data used for the LDR parameter data is not constant over an image frame, the mapping between the mapping parameter and the prediction value may be different depending on the size of the region of LDR image data used for the LDR parameter data. In alternative embodiments, wherein the size of the region of
LDR image data used for the LDR parameter data is not constant over an image frame, the mapping between the mapping parameter and the prediction value may be the same for different region sizes, and a normalization factor may be applied to the predicted value based on the region size . In some embodiments, the region size may be signaled explicitly in the enhancement-layer bitstream. In alternative embodiments, the region size may be derived from other information in either the base layer or the enhancement layer. Exemplary derivations may be based on the transform size or
the prediction mode determined from the bitstream.
In some embodiments of the present invention, a scale parameter value in a current image frame (first image frame) of an image sequence may be predicted from the scale parameter values corresponding to previously transmitted plurality of image frames. Some of these embodiments of the present invention may be described in relation to Figure 14. In these embodiments, LDR image data for a current image frame may be received 140. A scale correspondence indicator (first correspondence indicator) may be received 142 , and the scale correspondence indicator may be used to determine 144 a prediction of the current scale parameter value. The scale correspondence indicator may relate the current scale parameter value to a corresponding scale parameter value from a previously transmitted image frame.
In some embodiments of the present invention, an offset parameter value in a current image frame of an image sequence may be predicted from the offset parameter values corresponding to previously transmitted image frames. Some of these embodiments of the present invention may be described in relation to Figure 15. In these embodiments, LDR image data for a current image frame may be received 150. An offset correspondence indicator may be received 152 , and the offset correspondence indicator may be used to determine 154 a prediction of the current offset parameter
value. The offset correspondence indicator may relate the current offset parameter value to a corresponding offset parameter value from a previously transmitted image frame.
In some embodiments of the present invention described in relation to Figure 16, HDR prediction parameter values may be stored 160 for each previously decoded image frame . Exemplary HDR prediction parameters include offset parameters, scale parameters and other parameters which may be used in predicting HDR image data from LDR image data. In alternative embodiments, the HDR prediction parameter values may be stored for a subset of previously decoded image frames. An HDR prediction parameter correspondence indicator (second correspondence indicator) may be received 162. In some embodiments, the HDR prediction parameter correspondence indicator may be decoded from an enhancement-layer bitstream. The HDR prediction parameter correspondence indicator may determine 164 the location in the stored parameter values of the value which may be used as the current predicted HDR prediction parameter value.
In an exemplary embodiment, a parameter buffer may contain the parameter values for each 4-by-4 block in the previously reconstructed image frame . In this exemplary embodiment, the enhancement-layer bitstream may signal two indices which define the location in the parameter buffer of
previous parameter values to use for the parameter prediction. The two indices may be signaled explicitly in some embodiments. In alternative embodiments, the indices may be signaled relative to the location of the current block in the current frame.
Some embodiments of the present invention may be described in relation to Figure 17. In these embodiments, a second predicted HDR prediction parameter value may be refined according to residual information which may be received in and HDR enhancement-layer bitstream. HDR prediction parameter values may be stored 160 for each previously decoded image frame. Exemplary HDR prediction parameters include offset parameters, scale parameters and other parameters which may be used in predicting HDR image data from LDR image data. In alternative embodiments, the
HDR prediction parameter values may be stored for a subset of previously decoded image frames. An HDR prediction parameter correspondence indicator may be received 162. In some embodiments, the HDR prediction parameter correspondence indicator may be decoded from an enhancement-layer bitstream. The HDR prediction parameter correspondence indicator may determine 164 the location in the stored parameter values of the value which may be used as the current predicted HDR prediction parameter value then extracting 166 the parameter value from storage. An HDR
prediction parameter residual may be received 168. In some embodiments the HDR prediction parameter residual may be decoded from an enhancement-layer bitstream. The predicted HDR prediction parameter value may be corrected 170 according to the HDR prediction parameter residual.
In alternative embodiments of the present invention wherein an HDR prediction parameter value in a current image frame of an image sequence may be predicted from the HDR prediction parameter values corresponding to previously transmitted image frames, knowledge of the correspondence between pixel values in the LDR representation of the current, decoded frame and previous, decoded image data may be used to determine the predicted HDR prediction parameter value. In some embodiments of the present invention, the correspondence may be determined from parameters transmitted as part of the LDR bitstream, for example, motion vectors. In alternative embodiments, the correspondence may be determined by comparison of the LDR pixel data for the current frame and a previous LDR image frame. In some of these alternative embodiments, a motion estimation technique may be used to determine the correspondence .
In an exemplary embodiment, the correspondence between pixels in a current block and a block in a previously decoded image frame may be described with a motion vector comprising two indices. In some embodiments, the two
indices may correspond to a vertical displacement and a horizontal displacement. In some embodiments, the motion vector may be used explicitly for predicting the HDR prediction parameter value. In these embodiments, the predicted HDR prediction parameter value at a pixel may be the HDR prediction parameter value of the corresponding pixel in the previous frame. In these embodiments, a different predicted HDR prediction parameter value may be obtained at each pixel in a block being decoded. In alternative embodiments wherein a motion vector correspondence may be determined between pixels in the current frame and pixels in a previous frame, a single predicted HDR prediction parameter value may be determined based on the corresponding HDR prediction parameter values in the previous frame. Exemplary methods for combining the multiple parameter values may comprise a median computation, a mean computation, an average computation, a weighted mean computation, a weighted average computation, a voting calculation wherein the largest number of referenced parameter values in the previous locations may be used as the prediction and other combinations of the multiple parameter values.
In some embodiments of the present invention described in relation to Figure 18, motion vector correspondence may be refined based on residual information decoded from an
enhancement-layer bitstream. In these embodiments, LDR image data may be received 180, and HDR image data may be received 182. Motion vector correspondence may be determined 184. In some embodiments, motion vector correspondence may be determined 184 from parameters, for example motion vectors, transmitted with the LDR image data. In alternative embodiments, motion vector correspondence may be determined 184 through a motion vector correspondence estimation process wherein the pixel values of the LDR data may be examined. A motion vector correspondence residual value may be decoded 186 from the HDR image data. The motion vector correspondence may be corrected 188 in accordance with the motion vector correspondence residual value. The corrected motion vector correspondence may be used to determine 190 a predicted
HDR prediction parameter value.
In some embodiments of the present invention, a predicted HDR prediction parameter value may be corrected according to transmitted residual values. In these embodiments, an HDR prediction parameter value residual may be decoded from an enhancement-layer bitstream, and a predicted HDR prediction parameter value may be corrected, or refined, in accordance with the HDR prediction parameter value residual. In some embodiments of the present invention, a first
predicted HDR prediction parameter value and a second predicted HDR prediction parameter value may be determined. In some of these embodiments, the first predicted HDR prediction parameter value may be a predicted scale value, and the second predicted HDR prediction parameter value may be a predicted offset value. In some of these embodiments, a scale value residual may be determined, and an offset value may be determined. In these embodiments, the predicted scale value may be refined, and the predicted offset value may be refined. In alternative embodiments, a single residual value may be signaled, and a correction, or refinement, may be made to only one of these predicted values. The other predicted value may be adjusted based on the signaled residual value, characteristics of the decoded LDR image data and the other predicted value .
In some embodiments of the present invention, a flag may be send within the enhancement-layer bitstream from an encoder to a decoder. The flag may be sent on a block-by- block, macro-block-by-macro-block, slice-by-slice, frame-by- frame, sequence-by-sequence or other basis. The value, or state, of the flag may determine the prediction method to be used for predicting the HDR prediction parameter value .
Exemplary prediction methods include those described herein.
The above methods can also be incorporated into an apparatus for predicting a high dynamic range image element.
This apparatus comprises an LDR image data receiving section, for receiving a plurality of LDR image data; an LDR image data generating section, for generating a first predicted HDR prediction prarameter value based on the plurality of LDR image data; an LDR image data extracting section, for extracting an LDR image value from the plurality of LDR image data; and an HDR image element prediction forming section for forming an HDR image element prediction using the first predicted HDR prediction parameter value and the LDR image value.
The above apparatus can also include a computer program used for the prediction of images comprising multiple dynamic range layers, on a computer system. This computer program is stored on a storage medium such as an optical disc or a magnetic disk.
The storage medium containing the content data and the computer programs realizing the functions of the content processing device is by no means limited to the optical disc, which may be a CD-ROM (compact disc read-only memory) , MO (magneto-optical disc) , MD (MiniDisc) , or DVD (digital versatile disc) or the magnetic disk which may be a FD (flexible disk) or hard disk. Examples of such a storage medium include tapes, such as magnetic tapes and cassette tapes; card storage media, such as IC (integrated circuit) cards and optical cards; and semiconductor memories, such
as mask ROMs, EPROMs (erasable programmable ROMs) , EEPROMs (electrically erasable programmable ROMs) , and flash ROMs. Nevertheless, the computer system needs to have a readout device for retrieval from these storage media. The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding equivalence of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
Claims
1. A method for predicting a high dynamic range image element, said method comprising: a) receiving a plurality of low dynamic range (LDR) image data; b) generating a first predicted high dynamic range (HDR) prediction parameter value based on said plurality of LDR image data; c) extracting an LDR image value from said plurality of LDR image data; and d) forming an HDR image element prediction using said first predicted HDR prediction parameter value and said
LDR image value.
2. A method for predicting a high dynamic range image element, said method comprising: a) receiving a low dynamic range (LDR) image data corresponding to a first image frame; b) receiving a first correspondence indicator relating said LDR image data to a first previous-image-frame first high dynamic range (HDR) prediction parameter value; c) generating a first predicted HDR prediction parameter value based on said first correspondence indicator; d) extracting an LDR image value from said LDR image data; and e) forming an HDR image element prediction using said first predicted HDR prediction parameter value and said LDR image value.
3. A method as described in claims 1 or 2, wherein said first predicted HDR prediction parameter value is an HDR prediction parameter value selected from the group consisting of a predicted scale value and a predicted offset value.
4. A method as described in claim 1 further comprising: a) generating a second predicted HDR prediction parameter value based on said plurality of LDR image data; and b) forming said HDR image element prediction using said first predicted HDR prediction parameter value, said second predicted HDR prediction parameter value and said LDR image value.
5. A method as described in claim 4, wherein said first predicted HDR prediction parameter value is a predicted scale value and said second predicted HDR prediction parameter value is a predicted offset value.
6. A method as described in claims 1 or 2, wherein said first predicted HDR prediction parameter value comprises a luma HDR prediction parameter value and a chroma HDR prediction parameter value .
7. A method as described in claims 1 or 2 , wherein said forming an HDR image element prediction using said first predicted HDR prediction parameter value and said LDR image value comprises: a) receiving a first HDR prediction parameter value residual; and b) adjusting said first predicted HDR prediction parameter value in accordance with said received first HDR prediction parameter value residual.
8. A method as described in claims 1 or 2 further comprising: a) receiving an HDR residual image element; and b) combining said HDR image element prediction with said HDR residual image element to form a HDR image element.
9. A method as described in claim 1 , wherein said generating a first predicted HDR prediction parameter value based on said plurality of LDR image data comprises: a) determining a mapping parameter value based on said plurality of LDR image data; and b) selecting said first predicted HDR prediction parameter value from a plurality of candidate first HDR prediction parameter values based on said mapping parameter value.
10. A method as described in claim 9 , wherein said determining a mapping parameter value based on said plurality of LDR image data comprises combining said plurality of LDR image data using a method selected from the group consisting of summation, average, mean, weighted summation, weighted average, weighted mean, normalized summation, block transform and median.
1 1 . A method as described in claim 9 , wherein said selecting said first predicted HDR prediction parameter value from a plurality of candidate first HDR prediction parameter values based on said mapping parameter value comprises using said mapping parameter value as an index into a lookup-table comprising said plurality of candidate first HDR prediction parameter values.
12. A method as described in claim 2 further comprising: a) receiving a second correspondence indicator relating said LDR image data to a second-previous-image frame second HDR prediction parameter value; b) generating a second predicted HDR prediction parameter value based on said second correspondence indicator; and c) forming said HDR image element prediction using said first predicted HDR prediction parameter value, said second predicted HDR prediction parameter value and said
LDR image value.
13. A method as described in claim 12 , wherein said first predicted HDR prediction parameter value is a predicted scale value and said second predicted HDR prediction parameter value is a predicted offset value .
14. A method as described in claim 2 , wherein said first correspondence indicator indicates a location in a first HDR prediction parameter value buffer, wherein said first
HDR prediction parameter value buffer comprises a plurality of first HDR prediction parameter values associated with said first previous frame.
15. A method as described in claim 2 , wherein said first correspondence indicator comprises motion vector information relating said first frame to said first previous frame.
16. A method as described in claim 15 further comprising: a) receiving motion vector residual information; and b) correcting said motion vector information with said motion vector residual information.
17. A method as described in claim 15, wherein said motion vector information is calculated from said LDR image data corresponding to said first frame and LDR image data corresponding to said first previous frame.
18. An apparatus for predicting a high dynamic range image element, said apparatus comprising: a) an LDR image data receiving section, for receiving a plurality of LDR image data; b) an LDR image data generating section, for generating a first predicted HDR prediction prarameter value based on the plurality of LDR image data; c) an LDR image data extracting section, for extracting an LDR image value from the plurality of LDR image data; and d) an HDR image element prediction forming section, for forming an HDR image element prediction using the first predicted HDR prediction parameter value and the LDR image value.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010526094A JP2011509536A (en) | 2008-01-04 | 2008-12-26 | Method and apparatus for determining inter-layer image prediction parameters |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/969,719 | 2008-01-04 | ||
US11/969,719 US8175158B2 (en) | 2008-01-04 | 2008-01-04 | Methods and systems for inter-layer image prediction parameter determination |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2009087952A1 true WO2009087952A1 (en) | 2009-07-16 |
Family
ID=40844524
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2008/073966 WO2009087952A1 (en) | 2008-01-04 | 2008-12-26 | Methods and apparatus for inter-layer image prediction parameter determination |
Country Status (3)
Country | Link |
---|---|
US (1) | US8175158B2 (en) |
JP (1) | JP2011509536A (en) |
WO (1) | WO2009087952A1 (en) |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012086203A1 (en) * | 2010-12-22 | 2012-06-28 | パナソニック株式会社 | Image encoding apparatus, image decoding apparatus, image encoding method, and image decoding method |
JP2013543290A (en) * | 2010-08-25 | 2013-11-28 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Increase the dynamic range of images |
KR20130141676A (en) * | 2011-04-15 | 2013-12-26 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Encoding, decoding, and representing high dynamic range images |
US9036042B2 (en) | 2011-04-15 | 2015-05-19 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
CN104956679A (en) * | 2013-12-27 | 2015-09-30 | 索尼公司 | Image processing device and image processing method |
WO2016040255A1 (en) * | 2014-09-09 | 2016-03-17 | Dolby Laboratories Licensing Corporation | Self-adaptive prediction method for multi-layer codec |
JP2017099007A (en) * | 2011-03-24 | 2017-06-01 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Device and method for analyzing image grading |
CN107147942A (en) * | 2013-10-02 | 2017-09-08 | 杜比实验室特许公司 | Display management metadata is transmitted by HDMI |
JP2018101867A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing device, image processing method and program |
JP2018101863A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing system, image processing method, and program |
JP2018101866A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing device, image processing method and program |
US10110921B2 (en) | 2012-01-06 | 2018-10-23 | Thomson Licensing | Method of and device for encoding an HDR video together with an LDR video, method of and device for reconstructing one of an HDR video and an LDR video coded together and PF storage medium |
US10511837B2 (en) | 2011-04-15 | 2019-12-17 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
Families Citing this family (54)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8194997B2 (en) * | 2006-03-24 | 2012-06-05 | Sharp Laboratories Of America, Inc. | Methods and systems for tone mapping messaging |
US8503524B2 (en) * | 2007-01-23 | 2013-08-06 | Sharp Laboratories Of America, Inc. | Methods and systems for inter-layer image prediction |
US8665942B2 (en) * | 2007-01-23 | 2014-03-04 | Sharp Laboratories Of America, Inc. | Methods and systems for inter-layer image prediction signaling |
US7826673B2 (en) * | 2007-01-23 | 2010-11-02 | Sharp Laboratories Of America, Inc. | Methods and systems for inter-layer image prediction with color-conversion |
US8233536B2 (en) * | 2007-01-23 | 2012-07-31 | Sharp Laboratories Of America, Inc. | Methods and systems for multiplication-free inter-layer image prediction |
US20130107956A1 (en) | 2010-07-06 | 2013-05-02 | Koninklijke Philips Electronics N.V. | Generation of high dynamic range images from low dynamic range images |
US9066104B2 (en) | 2011-01-14 | 2015-06-23 | Google Inc. | Spatial block merge mode |
ES2750234T3 (en) | 2011-04-14 | 2020-03-25 | Dolby Laboratories Licensing Corp | Multiple color channel multiple regression predictor |
EP2702764B1 (en) | 2011-04-25 | 2016-08-31 | Dolby Laboratories Licensing Corporation | Non-linear vdr residual quantizer |
US9292940B2 (en) | 2011-04-28 | 2016-03-22 | Koninklijke Philips N.V. | Method and apparatus for generating an image coding signal |
WO2013068132A1 (en) | 2011-11-09 | 2013-05-16 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Inter-layer prediction between layers of different dynamic sample value range |
TWI556629B (en) * | 2012-01-03 | 2016-11-01 | 杜比實驗室特許公司 | Specifying visual dynamic range coding operations and parameters |
US9531990B1 (en) | 2012-01-21 | 2016-12-27 | Google Inc. | Compound prediction using multiple sources or prediction modes |
US8737824B1 (en) * | 2012-03-09 | 2014-05-27 | Google Inc. | Adaptively encoding a media stream with compound prediction |
US9392274B2 (en) | 2012-03-22 | 2016-07-12 | Qualcomm Incorporated | Inter layer texture prediction for video coding |
US9357197B2 (en) * | 2012-05-24 | 2016-05-31 | Dolby Laboratories Licensing Corporation | Multi-layer backwards-compatible video delivery for enhanced dynamic range and enhanced resolution formats |
US9185414B1 (en) | 2012-06-29 | 2015-11-10 | Google Inc. | Video encoding using variance |
JP2015526013A (en) | 2012-07-02 | 2015-09-07 | エレクトロニクス アンド テレコミュニケーションズ リサーチ インスチチュートElectronics And Telecommunications Research Institute | Video encoding method and apparatus, and video decoding method and apparatus |
US20140044162A1 (en) * | 2012-08-08 | 2014-02-13 | Qualcomm Incorporated | Adaptive inference mode information derivation in scalable video coding |
US10194158B2 (en) | 2012-09-04 | 2019-01-29 | Qualcomm Incorporated | Transform basis adjustment in scalable video coding |
CN108401157B (en) * | 2012-10-01 | 2022-06-24 | Ge视频压缩有限责任公司 | Scalable video decoder, scalable video encoder, and scalable video decoding and encoding methods |
KR102149115B1 (en) | 2012-11-16 | 2020-08-27 | 인터디지털 브이씨 홀딩스 인코포레이티드 | Processing high dynamic range images |
EP2741487A1 (en) * | 2012-12-06 | 2014-06-11 | Thomson Licensing | Video coding and decoding methods and corresponding devices |
US9532057B2 (en) * | 2012-12-28 | 2016-12-27 | Qualcomm Incorporated | Inter-layer prediction using sample-adaptive adjustments for bit depth scalable video coding |
PL2941872T3 (en) | 2013-01-02 | 2019-03-29 | Dolby Laboratories Licensing Corporation | Backward-compatible coding for ultra high definition video signals with enhanced dynamic range |
US9628790B1 (en) | 2013-01-03 | 2017-04-18 | Google Inc. | Adaptive composite intra prediction for image and video compression |
BR112015024172B1 (en) * | 2013-03-26 | 2023-01-31 | Dolby Laboratories Licensing Corporation | METHOD, SYSTEM AND COMPUTER READABLE STORAGE MEDIA |
EP2979447B1 (en) | 2013-03-28 | 2018-01-03 | Huawei Technologies Co., Ltd. | Method for determining predictor blocks for a spatially scalable video codec |
US20140301463A1 (en) * | 2013-04-05 | 2014-10-09 | Nokia Corporation | Method and apparatus for video coding and decoding |
JP6345650B2 (en) * | 2013-04-05 | 2018-06-20 | サターン ライセンシング エルエルシーSaturn Licensing LLC | Image processing apparatus and image processing method |
US9374578B1 (en) | 2013-05-23 | 2016-06-21 | Google Inc. | Video coding using combined inter and intra predictors |
KR20160019060A (en) * | 2013-06-10 | 2016-02-18 | 톰슨 라이센싱 | Encoding and decoding methods for adapting the average luminance of high dynamic range pictures and corresponding encoder and decoder |
JP2015005899A (en) * | 2013-06-21 | 2015-01-08 | ソニー株式会社 | Decoder and decoding method, encoder and encoding method |
US20150229932A1 (en) * | 2013-07-12 | 2015-08-13 | Sony Corporation | Image processing device and image processing method |
KR101797505B1 (en) * | 2013-11-13 | 2017-12-12 | 엘지전자 주식회사 | Broadcast signal transmission method and apparatus for providing hdr broadcast service |
EP2874397A1 (en) | 2013-11-15 | 2015-05-20 | Thomson Licensing | Method of embedding of an image with a color transform |
EP2887673A1 (en) * | 2013-12-17 | 2015-06-24 | Thomson Licensing | Method for coding a sequence of pictures and method for decoding a bitstream and corresponding devices |
EP2887672A1 (en) * | 2013-12-17 | 2015-06-24 | Thomson Licensing | Method for coding a sequence of pictures and method for decoding a bitstream and corresponding devices |
US9609343B1 (en) | 2013-12-20 | 2017-03-28 | Google Inc. | Video coding using compound prediction |
US10536731B2 (en) | 2014-05-12 | 2020-01-14 | Apple Inc. | Techniques for HDR/WCR video coding |
CN106464967B (en) | 2014-05-15 | 2019-11-22 | 索尼公司 | Communication device, communication means and computer readable storage medium |
EP2958328A1 (en) * | 2014-06-20 | 2015-12-23 | Thomson Licensing | Method and device for signaling in a bitstream a picture/video format of an LDR picture and a picture/video format of a decoded HDR picture obtained from said LDR picture and an illumination picture |
EP2958075A1 (en) * | 2014-06-20 | 2015-12-23 | Thomson Licensing | Method and apparatus for dynamic range expansion of LDR video sequence |
EP3205093B1 (en) * | 2014-10-07 | 2022-05-18 | TRELLIS EUROPE S.r.l. | Improved video and image encoding process |
EP3051818A1 (en) | 2015-01-30 | 2016-08-03 | Thomson Licensing | Method and device for decoding a color picture |
EP3051821A1 (en) * | 2015-01-30 | 2016-08-03 | Thomson Licensing | Method and apparatus for encoding and decoding high dynamic range (HDR) videos |
EP3076669A1 (en) * | 2015-04-03 | 2016-10-05 | Thomson Licensing | Method and apparatus for generating color mapping parameters for video encoding |
US20160286224A1 (en) * | 2015-03-26 | 2016-09-29 | Thomson Licensing | Method and apparatus for generating color mapping parameters for video encoding |
EP3350774A1 (en) * | 2015-09-18 | 2018-07-25 | Thomson Licensing | Determination of a co-located luminance sample of a color component sample, for hdr coding/decoding |
EP3457694A4 (en) * | 2016-05-24 | 2019-05-22 | Huawei Technologies Co., Ltd. | Image prediction method and related device |
JP6867774B2 (en) * | 2016-10-17 | 2021-05-12 | キヤノン株式会社 | Image coding device and its control method |
US20190171927A1 (en) * | 2017-12-06 | 2019-06-06 | Facebook, Inc. | Layer-level quantization in neural networks |
CN110209884B (en) * | 2018-01-10 | 2022-08-05 | 杭州海康威视数字技术股份有限公司 | Index checking method and device |
JP6584538B2 (en) * | 2018-01-16 | 2019-10-02 | トムソン ライセンシングThomson Licensing | High dynamic range image processing |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004007377A (en) * | 2002-04-18 | 2004-01-08 | Toshiba Corp | Motion picture coding / decoding method and apparatus |
US20060104533A1 (en) * | 2004-11-16 | 2006-05-18 | Sharp Laboratories Of America, Inc. | High dynamic range images from low dynamic range images |
JP2006279383A (en) * | 2005-03-29 | 2006-10-12 | Nippon Telegr & Teleph Corp <Ntt> | Interhierarchy prediction coding method, apparatus thereof, interhierarchy prediction decoding method, apparatus thereof, interhierarchy prediction coding program, interhierarchy prediction decoding program and program recording medium thereof |
WO2007082562A2 (en) * | 2006-01-23 | 2007-07-26 | MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. | High dynamic range codecs |
JP2007243942A (en) * | 2006-02-24 | 2007-09-20 | Sharp Corp | Encoding method and decoding method of image data, and codec system |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6400996B1 (en) | 1999-02-01 | 2002-06-04 | Steven M. Hoffberg | Adaptive pattern recognition based control system and method |
US6850252B1 (en) | 1999-10-05 | 2005-02-01 | Steven M. Hoffberg | Intelligent electronic appliance system and method |
US7006881B1 (en) | 1991-12-23 | 2006-02-28 | Steven Hoffberg | Media recording device with remote graphic user interface |
US5832120A (en) | 1995-12-22 | 1998-11-03 | Cirrus Logic, Inc. | Universal MPEG decoder with scalable picture size |
US6829301B1 (en) | 1998-01-16 | 2004-12-07 | Sarnoff Corporation | Enhanced MPEG information distribution apparatus and method |
US7966078B2 (en) | 1999-02-01 | 2011-06-21 | Steven Hoffberg | Network media appliance system and method |
US6396422B1 (en) | 1999-06-09 | 2002-05-28 | Creoscitex Corporation Ltd. | Methods for quantizing and compressing digital image data |
US7929610B2 (en) | 2001-03-26 | 2011-04-19 | Sharp Kabushiki Kaisha | Methods and systems for reducing blocking artifacts with reduced complexity for spatially-scalable video coding |
US20030112863A1 (en) | 2001-07-12 | 2003-06-19 | Demos Gary A. | Method and system for improving compressed image chroma information |
US6867717B1 (en) | 2002-04-04 | 2005-03-15 | Dalsa, Inc. | Digital encoder and method of encoding high dynamic range video images |
HUP0301368A3 (en) | 2003-05-20 | 2005-09-28 | Amt Advanced Multimedia Techno | Method and equipment for compressing motion picture data |
EP1745436B1 (en) * | 2004-04-15 | 2012-05-30 | Dolby Laboratories Licensing Corporation | Methods and systems for converting images from low dynamic range to high dynamic range |
US8218625B2 (en) | 2004-04-23 | 2012-07-10 | Dolby Laboratories Licensing Corporation | Encoding, decoding and representing high dynamic range images |
US7483486B2 (en) | 2004-07-02 | 2009-01-27 | Max-Planck-Gesellschaft Zur Forderung Der Wissenschaften E.V. | Method and apparatus for encoding high dynamic range video |
US8175168B2 (en) | 2005-03-18 | 2012-05-08 | Sharp Laboratories Of America, Inc. | Methods and systems for picture up-sampling |
US7961963B2 (en) | 2005-03-18 | 2011-06-14 | Sharp Laboratories Of America, Inc. | Methods and systems for extended spatial scalability with picture-level adaptation |
US8023569B2 (en) | 2005-12-15 | 2011-09-20 | Sharp Laboratories Of America, Inc. | Methods and systems for block-based residual upsampling |
US8194997B2 (en) * | 2006-03-24 | 2012-06-05 | Sharp Laboratories Of America, Inc. | Methods and systems for tone mapping messaging |
EP1871113A1 (en) * | 2006-06-20 | 2007-12-26 | THOMSON Licensing | Method and apparatus for encoding video enhancement layer with multiresolution color scalability |
US8085852B2 (en) * | 2007-06-26 | 2011-12-27 | Mitsubishi Electric Research Laboratories, Inc. | Inverse tone mapping for bit-depth scalable image coding |
US8432968B2 (en) * | 2007-10-15 | 2013-04-30 | Qualcomm Incorporated | Scalable video coding techniques for scalable bitdepths |
-
2008
- 2008-01-04 US US11/969,719 patent/US8175158B2/en not_active Expired - Fee Related
- 2008-12-26 JP JP2010526094A patent/JP2011509536A/en active Pending
- 2008-12-26 WO PCT/JP2008/073966 patent/WO2009087952A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004007377A (en) * | 2002-04-18 | 2004-01-08 | Toshiba Corp | Motion picture coding / decoding method and apparatus |
US20060104533A1 (en) * | 2004-11-16 | 2006-05-18 | Sharp Laboratories Of America, Inc. | High dynamic range images from low dynamic range images |
JP2006279383A (en) * | 2005-03-29 | 2006-10-12 | Nippon Telegr & Teleph Corp <Ntt> | Interhierarchy prediction coding method, apparatus thereof, interhierarchy prediction decoding method, apparatus thereof, interhierarchy prediction coding program, interhierarchy prediction decoding program and program recording medium thereof |
WO2007082562A2 (en) * | 2006-01-23 | 2007-07-26 | MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. | High dynamic range codecs |
JP2007243942A (en) * | 2006-02-24 | 2007-09-20 | Sharp Corp | Encoding method and decoding method of image data, and codec system |
Non-Patent Citations (2)
Cited By (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2013543290A (en) * | 2010-08-25 | 2013-11-28 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Increase the dynamic range of images |
US9137539B2 (en) | 2010-12-22 | 2015-09-15 | Panasonic Corporation | Image coding apparatus, image decoding apparatus, image coding method, and image decoding method |
WO2012086203A1 (en) * | 2010-12-22 | 2012-06-28 | パナソニック株式会社 | Image encoding apparatus, image decoding apparatus, image encoding method, and image decoding method |
JP2017099007A (en) * | 2011-03-24 | 2017-06-01 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Device and method for analyzing image grading |
US10264259B2 (en) | 2011-04-15 | 2019-04-16 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US9819938B2 (en) | 2011-04-15 | 2017-11-14 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US9036042B2 (en) | 2011-04-15 | 2015-05-19 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
CN103888743A (en) * | 2011-04-15 | 2014-06-25 | 杜比实验室特许公司 | Encoding, decoding, and representing high dynamic range images |
US10992936B2 (en) | 2011-04-15 | 2021-04-27 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US9271011B2 (en) | 2011-04-15 | 2016-02-23 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
TWI690211B (en) * | 2011-04-15 | 2020-04-01 | 美商杜比實驗室特許公司 | Decoding method for high dynamic range images, processor non-transistory readable medium and computer program product thereof |
CN103888743B (en) * | 2011-04-15 | 2016-05-25 | 杜比实验室特许公司 | For the treatment of the method for high dynamic range images |
KR101632596B1 (en) | 2011-04-15 | 2016-06-22 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Encoding, decoding, and representing high dynamic range images |
RU2589857C2 (en) * | 2011-04-15 | 2016-07-10 | Долби Лабораторис Лайсэнзин Корпорейшн | Encoding, decoding and representing high dynamic range images |
EP3166298A1 (en) * | 2011-04-15 | 2017-05-10 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US9654781B2 (en) | 2011-04-15 | 2017-05-16 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
EP2697962A2 (en) * | 2011-04-15 | 2014-02-19 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
KR101751226B1 (en) | 2011-04-15 | 2017-06-27 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Encoding, decoding, and representing high dynamic range images |
US10511837B2 (en) | 2011-04-15 | 2019-12-17 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
EP2697962A4 (en) * | 2011-04-15 | 2014-11-19 | Dolby Lab Licensing Corp | Encoding, decoding, and representing high dynamic range images |
RU2640717C1 (en) * | 2011-04-15 | 2018-01-11 | Долби Лабораторис Лайсэнзин Корпорейшн | Coding, decoding and presenting high dynamic range images |
KR20130141676A (en) * | 2011-04-15 | 2013-12-26 | 돌비 레버러토리즈 라이쎈싱 코오포레이션 | Encoding, decoding, and representing high dynamic range images |
EP3376749A1 (en) * | 2011-04-15 | 2018-09-19 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US10027961B2 (en) | 2011-04-15 | 2018-07-17 | Dolby Laboratories Licensing Corporation | Encoding, decoding, and representing high dynamic range images |
US10110921B2 (en) | 2012-01-06 | 2018-10-23 | Thomson Licensing | Method of and device for encoding an HDR video together with an LDR video, method of and device for reconstructing one of an HDR video and an LDR video coded together and PF storage medium |
US10708620B2 (en) | 2012-01-06 | 2020-07-07 | Interdigital Vc Holdings, Inc. | Method of and device for encoding an HDR video together with an LDR video, method of and device for reconstructing one of an HDR video and an LDR video coded together and non-transitory storage medium |
CN107147942A (en) * | 2013-10-02 | 2017-09-08 | 杜比实验室特许公司 | Display management metadata is transmitted by HDMI |
CN107147942B (en) * | 2013-10-02 | 2021-01-26 | 杜比实验室特许公司 | Video signal transmission method, device, apparatus and storage medium |
CN104956679A (en) * | 2013-12-27 | 2015-09-30 | 索尼公司 | Image processing device and image processing method |
US10542265B2 (en) | 2014-09-09 | 2020-01-21 | Dolby Laboratories Licensing Corporation | Self-adaptive prediction method for multi-layer codec |
WO2016040255A1 (en) * | 2014-09-09 | 2016-03-17 | Dolby Laboratories Licensing Corporation | Self-adaptive prediction method for multi-layer codec |
JP2018101866A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing device, image processing method and program |
JP2018101863A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing system, image processing method, and program |
JP2018101867A (en) * | 2016-12-19 | 2018-06-28 | ソニー株式会社 | Image processing device, image processing method and program |
US11006113B2 (en) | 2016-12-19 | 2021-05-11 | Sony Corporation | Image processing device, method, and program deciding a processing parameter |
US11190744B2 (en) | 2016-12-19 | 2021-11-30 | Sony Corporation | Image processing device, image processing method, and program for determining a cost function for mode selection |
Also Published As
Publication number | Publication date |
---|---|
JP2011509536A (en) | 2011-03-24 |
US8175158B2 (en) | 2012-05-08 |
US20090175338A1 (en) | 2009-07-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8175158B2 (en) | Methods and systems for inter-layer image prediction parameter determination | |
US8953677B2 (en) | Methods and systems for inter-layer image prediction signaling | |
US7826673B2 (en) | Methods and systems for inter-layer image prediction with color-conversion | |
CN107211128B (en) | Adaptive chroma downsampling and color space conversion techniques | |
CN111819852B (en) | Method and apparatus for residual symbol prediction in the transform domain | |
CN107707914B (en) | Method and apparatus for providing compensation offset for a set of reconstructed samples of an image | |
US10708588B2 (en) | Sample adaptive offset control | |
US8503524B2 (en) | Methods and systems for inter-layer image prediction | |
US8014613B2 (en) | Methods and systems for inter-layer image parameter prediction | |
US8233536B2 (en) | Methods and systems for multiplication-free inter-layer image prediction | |
CN109644273B (en) | Apparatus and method for video encoding | |
WO2015140402A2 (en) | Method and technical equipment for video encoding and decoding | |
EP3868098A1 (en) | Method and apparatus of local illumination compensation for predictive coding | |
EP3963884A1 (en) | Method and apparatus of local illumination compensation for inter prediction | |
WO2020242350A9 (en) | Usage of dct based interpolation filter | |
WO2017093604A1 (en) | A method, an apparatus and a computer program product for encoding and decoding video | |
CN112889280A (en) | Method and apparatus for encoding and decoding digital image/video material | |
WO2020149769A1 (en) | An encoder, a decoder and corresponding methods for local illumination compensation | |
CN114598873B (en) | Decoding method and device for quantization parameter | |
US20240064311A1 (en) | A method, an apparatus and a computer program product for encoding and decoding | |
US20220159281A1 (en) | No Reference Image Quality Assessment Based Decoder Side Intra Prediction | |
WO2023242466A1 (en) | A method, an apparatus and a computer program product for video coding | |
WO2023237808A1 (en) | A method, an apparatus and a computer program product for encoding and decoding of digital media content | |
CN117956169A (en) | Image coding and decoding method and device | |
WO2023111384A1 (en) | A method, an apparatus and a computer program product for video encoding and video decoding |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08870242 Country of ref document: EP Kind code of ref document: A1 |
|
DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
WWE | Wipo information: entry into national phase |
Ref document number: 2010526094 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 08870242 Country of ref document: EP Kind code of ref document: A1 |