WO2016137240A1 - Hybrid image correction for dual-aperture camera - Google Patents

Hybrid image correction for dual-aperture camera Download PDF

Info

Publication number
WO2016137240A1
WO2016137240A1 PCT/KR2016/001837 KR2016001837W WO2016137240A1 WO 2016137240 A1 WO2016137240 A1 WO 2016137240A1 KR 2016001837 W KR2016001837 W KR 2016001837W WO 2016137240 A1 WO2016137240 A1 WO 2016137240A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
infrared
edges
visible light
aperture
Prior art date
Application number
PCT/KR2016/001837
Other languages
French (fr)
Inventor
Andrew Augustine Wajs
Hyunsang Park
Original Assignee
Dual Aperture International Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dual Aperture International Co., Ltd. filed Critical Dual Aperture International Co., Ltd.
Publication of WO2016137240A1 publication Critical patent/WO2016137240A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/142—Edging; Contouring
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/30—Transforming light or analogous information into electric information
    • H04N5/33—Transforming infrared radiation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/73—Colour balance circuits, e.g. white balance circuits or colour temperature control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/77—Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase

Definitions

  • the present invention relates to a hybrid image correction for a dual-aperture camera.
  • a dual-aperture camera may have two apertures, a narrow aperture for a first wavelength range (e.g. infrared (IR) light) and a wider aperture for a second wavelength range such as visible light (e.g. from red (R), green (G), and blue (B) pixels or an RGB signal).
  • the first wavelength range may produce a relatively sharp IR image from a relatively small aperture.
  • the second wavelength range may produce a less sharp RGB image from a relatively wide aperture.
  • An IR image may be relatively noisy.For example, without color correction, pseudo color imaging defects may cause false colors at depicted edges of the image due to the IR noise combined with RGB components.When producing an enhanced image (e.g.
  • Embodiments relate to cameras.
  • Embodiments relate to cameras, imaging devices, image processing devices, and associated methods.
  • Embodiments relate to a dual-aperture camera that uses two apertures.
  • a first aperture of the dual-aperture camera is configured for a first wavelength range to obtain a relatively sharp image.
  • a second aperture of the dual aperture camera is a wider aperture than the first aperture.
  • the second aperture is configured for a second wavelength range to obtain a relatively blurry image compared to the relatively sharp image.
  • Depth information e.g. 3D image information
  • 3D image information for the image subject may be measured by comparing the blurriness of the relatively sharp image and the relatively blurry image.
  • Embodiments relate to a method for reducing the IR noise of a dual-aperture camera while preserving edge information.
  • a method may subtract a gain of the IR noise from a pixel of the RGB image by the amount proportional to the distance of the pixel from the edge. For example, on the depicted edges of an image, substantially no IR noise may subtracted from the RGB image, while further away from the edges (such as on a depicted flat surface),substantially all of the IR noise may be subtracted from the RGB image.
  • Embodiments relate to a method to reduce false colors along the edges when the IR noise is not reduced at all.
  • false colors are suppressed by selectively adjusting the chrominance ofdepicted edges and/or near the depicted edges.
  • Example Figure1 Illustrates a color correction process in accordance with the related art.
  • Example Figure 2 illustrates a color correction process, in accordance with embodiments.
  • Example Figure 3 illustratesa false color suppression process on RGB edges, in accordance with embodiments.
  • Example Figure 4 illustrates a false color suppression process on IR edges, in accordance with embodiments.
  • Example Figure 1 is a method to measure depth in a dual-aperture camera.
  • two images may be produced from the same pixel array using two different sized apertures.
  • Infrared (IR) noise of an IR image 12 may be subtracted 14 from an RGB image 10 that may create image defects along depicted edges in the corresponding IR image 12 and RGB image 10.
  • the subtracted IR noise of the IR image 12 is subjected to color correction and/or white balance 16 to produce a color corrected and balanced RGB image 18.
  • color correction and/or white balance 16 to produce a color corrected and balanced RGB image 18.
  • edge information e.g. portions of an image that show a clear straight line
  • An RGB image 10 of light in the visible wavelength range includes pixel data from a relatively wide aperture of a dual aperture camera.
  • the infrared (e.g. IR) image 12 of light in the infrared wavelength range has a relatively small aperture.
  • the relatively narrow aperture for the IR image 12 highly collimated rays are admitted, resulting in a sharp focus across the entire image plane.
  • uncollimated rays are admitted, resulting in a sharp focus only for rays with a certain focal length range on a portion of the image plane.
  • Example Figure 2 is a method according to embodiments.
  • An infrared (IR) image 12 is subjected to noise reduction 13.
  • the RGB image 10 is subjected to edge detection 20.
  • the IR gain 22 of the noise reduced IR image 12 is selectively controlled to compensate for anticipated defects at the edges due to IR noise.
  • the selectively IR gain 22 of the noise reduced IR image 12 is subtracted from the RGB image 10.
  • the subtracted image is subjected to color correction and white balance 16 to produce a color corrected and balanced RGB image 18.
  • the method subtracts 24 a gain of the IR channel 12 from the RGB image 10, in accordance with embodiments.
  • the selective IR gain 22 is proportional to the distance of a given pixel from a corresponding edge as determined by edge detection 20. For example, substantially no IR is subtracted from an RGB image on the edges, while substantially all of the IR is subtracted from an RGB image in the areas relatively far from the depicted edges, such as on a depictedflatsurface.
  • a selective averaging filter such as the Sigma filter function may be used, in accordance with embodiments.
  • the IR gain 22 100%, in accordance with embodiments.
  • the gain of IR 0%, in accordance with embodiments.For example, the IR gain varies between 100% and 0% based on the distance of a pixel from the corresponding edge.
  • noise reduction 13 may be omitted. However, when there is no IR noise reduction at all, the colors on the edges may be inconsistent, exhibiting false colors. Thispseudo color problem or false color problem may be corrected by suppressing the false colors. Noise reduction 13 may remove noise and edges from the IR image 12, in accordance with embodiments.
  • Example Figure 3 shows a process of suppressing the effect of false colors, in accordance with embodiments.
  • an RGB image 10 may be subject to color correction and white balance 26, in accordance with embodiments.
  • a RGB image 10 may be converted to an image in the YCbCr color space 28, where Y is the luminance component, Cb is the blue chrominance component, and Cr is the red chrominance component. Edges are detected 30 in the luminance Y component of the YCbCr converted RGB image.
  • the Cb and Cr components on the Y edges are suppressed 32 to generate suppressed Cb' and Cr' chrominance components, in accordance with embodiments.
  • the effect of false colors may be minimized by converting the inconsistent color edges to more consistent monochrome-like edges.
  • the amount of suppression may be proportional to the magnitude of the Y edges, as determined by image processing, in accordance with embodiments.
  • an YCb'Cr' color space is converted 34 back to RGB to generate a color corrected and balanced RGB image 36, in accordance with embodiments.
  • Example Figure 4 illustrates a process of suppressing the effect of false colors, in accordance with embodiments.
  • An IR image by be subjected to noise reduction 38 and subtracted 39 from an RGB image.
  • the subtracted image may be color corrected and white balanced 26.
  • the colorcorrected and/or while balanced image may be converted from RGB to YCbCr color space 28.
  • the luminance components Y of the YCbCr image may be subjected to edge detection 30.
  • the corresponding IR image may be subjected edge detection 40.
  • the edge detection 40 on the IR imageand the edge detection 30 on the luminance components Y of the YCbCr are both used to generate an YCb'Cr' image.
  • the Cb and Cr components on both the detected Y edges and the detected IR edges are suppressed 42 to generate suppressed Cb' and Cr' chrominance components, in accordance with embodiments.
  • the effect of false colors may be minimized by converting the inconsistent color edges to more consistent monochrome-like edges.
  • the chrominance (Cb and Cr component) of the original image may be reduced to reduce the effect of false colors by converting the inconsistent color edges to more consistent monochrome-like edges. Since edges may be detected from both the IR image and the RGB image, suppression of false colors at edges may be compensated.
  • the amount of suppression may be proportional to the magnitude of the IR edges.
  • the amount of suppression may be proportional to the magnitude of both the Y edges and IR edges, as determined by image processing, in accordance with embodiments.
  • one of the tradeoffs in generating a good quality image using color correction is the level of noise versus color quality.
  • relatively large coefficients in the color correction matrix may be appliedto get the best possible color result.
  • using relatively large coefficientin the color correction matrix mayamplify noise levels in the image. Accordingly, in embodiments,for the best possible noise results the coefficientsin the color correction matrixshould be as close as possible to 1, while for the best possible color the coefficientsin the color correction matrixmay be significantlylarger than 1.
  • a method for noise reduction in images detects proximity of a pixel to an edge and adjusts the level of averaging based on the proximity to the edge, in accordance with embodiments. For example, if a pixel is relatively far from an edge, arelativelylarge averaging window is used, while if a pixel isrelativelyclose to an edge a relatively small averaging or no averaging window is used.
  • Embodiments relate to a combined approach tocolor correction.
  • a color correction matrix may be used that producesrelatively low noise but with relatively poor color reproduction. For example, color is best observed by a user in flat areasof an image, whilecolor perception near edges of an image are less pronounced or noticeable.
  • a lower level of color accuracy of the color correction matrix may be less noticeable to a user near the edges depicted in an image compared relatively flat surfaces of an image.
  • a color correction matrix with relatively poor noise characteristics but relatively good color characteristics may be used for the pixels that are relatively far from the edges depicted in an image.
  • such relatively poor noise characteristics at relatively flat surfaces of an image may bereducedor minimizedby averaging with a relatively large window to reduce the noise that is generated by the color correction matrix.
  • Embodiments may enable relatively low noise color correction to be performed on edges where it is relatively difficultto perform averaging without causing the image to become blurred.
  • relatively desirable noise reduction color correction (which may have relatively poor color correction) may be performed in regionsof a depicted image wherecolor is less noticeable by the human eye (e.g. edges depicted in the image).
  • regions of a depicted image with limited edge details, a color correction matrix with relatively good color clarity, while sacrificing relatively high noise characteristics may be applied together with noise reduction through averaging to remove the effect of the noise that is amplified by therelatively good color claritycolor correction matrix.
  • Embodiments relate to multiple color correction matrices which range from having high noise but good color correction to low noise but poor color correction to achieve optimum color correction balanced with minimum noise.
  • an optimum color correction matrix may be chosen from multiple different color correction matrixes, depending on the distance of a pixel from the edge to balance noise and color correction.
  • Embodiments may employnoise reduction kernels, wherethe selection of the noise reduction kernel is based on the distance of a pixel from a depicted edge in an image.
  • Embodiments mayuse interpolation to generate a specific color correction matrix and/or noise reduction kernel based on the distance of a pixel from a depictededge in an image.

Abstract

A method to correct an image in a dual-aperture camera. Infrared noise may be subtracted from a pixel of an RGB image by the amount proportional to the distance from the corresponding edge to minimize imaging defects in a dual aperture camera. Such defect minimization may prevent edge information from being lost. A method reduces the effect of false color along the edges by converting inconsistent color edges to more consistent monochrome edges using the YCbCr color space.

Description

HYBRID IMAGE CORRECTION FOR DUAL-APERTURE CAMERA
The present invention relates to a hybrid image correction for a dual-aperture camera.
A dual-aperture camera may have two apertures, a narrow aperture for a first wavelength range (e.g. infrared (IR) light) and a wider aperture for a second wavelength range such as visible light (e.g. from red (R), green (G), and blue (B) pixels or an RGB signal). The first wavelength range may produce a relatively sharp IR image from a relatively small aperture. The second wavelength range may produce a less sharp RGB image from a relatively wide aperture. An IR imagemay be relatively noisy.For example, without color correction, pseudo color imaging defects may cause false colors at depicted edges of the image due to the IR noise combined with RGB components.When producing an enhanced image (e.g. a 3D image) from the relatively sharp IR image and the less sharp RGB image, color correction of RGB components due to IR noise may be performed. However, during color correction,overcompensating for IR noise reduction maycause edge information (e.g. portions of an image that show a clear straight line) to be lost, which may create an undesirable defect.Accordingly, there is a need to reduce the IR noise as well as preserve the edge information to produce a relatively high quality image.
Embodiments relate to cameras.
Embodiments relate to cameras, imaging devices, image processing devices, and associated methods. Embodiments relate to a dual-aperture camera that uses two apertures. A first aperture of the dual-aperture camera is configured for a first wavelength range to obtain a relatively sharp image. A second aperture of the dual aperture camera is a wider aperture than the first aperture. The second aperture is configured for a second wavelength range to obtain a relatively blurry image compared to the relatively sharp image. Depth information (e.g. 3D image information) for the image subject may be measured by comparing the blurriness of the relatively sharp image and the relatively blurry image.
Embodiments relate to a method for reducing the IR noise of a dual-aperture camera while preserving edge information. In accordance with embodiments, a method may subtract a gain of the IR noise from a pixel of the RGB image by the amount proportional to the distance of the pixel from the edge. For example, on the depicted edges of an image, substantially no IR noise may subtracted from the RGB image, while further away from the edges (such as on a depicted flat surface),substantially all of the IR noise may be subtracted from the RGB image. Embodiments relate to a method to reduce false colors along the edges when the IR noise is not reduced at all.
In accordance with embodiments, false colorsare suppressed by selectively adjusting the chrominance ofdepicted edges and/or near the depicted edges.
Example Figure1illustrates a color correction process in accordance with the related art.
Example Figure 2 illustrates a color correction process, in accordance with embodiments.
Example Figure 3 illustratesa false color suppression process on RGB edges, in accordance with embodiments.
Example Figure 4 illustrates a false color suppression process on IR edges, in accordance with embodiments.
Example Figure 1 is a method to measure depth in a dual-aperture camera. For example, in a dual aperture camera, two images may be produced from the same pixel array using two different sized apertures. Infrared (IR) noise of an IR image 12 may be subtracted 14 from an RGB image 10 that may create image defects along depicted edges in the corresponding IR image 12 and RGB image 10. The subtracted IR noise of the IR image 12 is subjected to color correction and/or white balance 16 to produce a color corrected and balanced RGB image 18. However, during color correction 16,overcompensation due to the IR noise subtractionmaycause edge information (e.g. portions of an image that show a clear straight line) to be lost, which may create an undesirable defect.
An RGB image 10 of light in the visible wavelength range includes pixel data from a relatively wide aperture of a dual aperture camera. Meanwhile, the infrared (e.g. IR) image 12 of light in the infrared wavelength range has a relatively small aperture. In the relatively narrow aperture for the IR image 12, highly collimated rays are admitted, resulting in a sharp focus across the entire image plane. In the relatively wide aperture for RGB image 10, uncollimated rays are admitted, resulting in a sharp focus only for rays with a certain focal length range on a portion of the image plane. This means that the relatively wider aperture for the RGB image 10 results in a visible light image that is only sharparound a focal point based on the relative width of the aperture. Since the relative width of the aperture also determines how many of the incoming rays are actually admitted, a relatively narrow aperture allows less light to reach the image plane than a relatively wide aperture.
Example Figure 2 is a method according to embodiments. An infrared (IR) image 12 is subjected to noise reduction 13. The RGB image 10 is subjected to edge detection 20. Based on the result of the edge detection 20, the IR gain 22 of the noise reduced IR image 12 is selectively controlled to compensate for anticipated defects at the edges due to IR noise. The selectively IR gain 22 of the noise reduced IR image 12 is subtracted from the RGB image 10. The subtracted image is subjected to color correction and white balance 16 to produce a color corrected and balanced RGB image 18. The method subtracts 24 a gain of the IR channel 12 from the RGB image 10, in accordance with embodiments. The selective IR gain 22 is proportional to the distance of a given pixel from a corresponding edge as determined by edge detection 20. For example, substantially no IR is subtracted from an RGB image on the edges, while substantially all of the IR is subtracted from an RGB image in the areas relatively far from the depicted edges, such as on a depictedflatsurface.
For edge detection 20, a selective averaging filter such as the Sigma filter function may be used, in accordance with embodiments. In embodiments, it is not necessary to control the magnitude of a low pass function in the noise reduction mode, since the edge detection 20 information governs the magnitude of the IR gain 22 to be subtracted 24 from the RGB components 10. For example, when a pixel is in the middle of a flat surface, the IR gain 22 = 100%, in accordance with embodiments. For example, when a pixel is on an edge, the gain of IR = 0%, in accordance with embodiments.For example, the IR gain varies between 100% and 0% based on the distance of a pixel from the corresponding edge.
In embodiments, noise reduction 13 may be omitted. However, when there is no IR noise reduction at all, the colors on the edges may be inconsistent, exhibiting false colors. Thispseudo color problem or false color problem may be corrected by suppressing the false colors. Noise reduction 13 may remove noise and edges from the IR image 12, in accordance with embodiments.
Example Figure 3 shows a process of suppressing the effect of false colors, in accordance with embodiments. Optionally, an RGB image 10 may be subject to color correction and white balance 26, in accordance with embodiments. A RGB image 10 may be converted to an image in the YCbCr color space 28, where Y is the luminance component, Cb is the blue chrominance component, and Cr is the red chrominance component. Edges are detected 30 in the luminance Y component of the YCbCr converted RGB image. The Cb and Cr components on the Y edges are suppressed 32 to generate suppressed Cb' and Cr' chrominance components, in accordance with embodiments. By generating suppressed Cb' and Cr' chrominance components, the effect of false colors may be minimized by converting the inconsistent color edges to more consistent monochrome-like edges. The amount of suppression may be proportional to the magnitude of the Y edges, as determined by image processing, in accordance with embodiments. Using the generated suppressed Cb' and Cr' chrominance components and the original luminance Y component, an YCb'Cr' color space is converted 34 back to RGB to generate a color corrected and balanced RGB image 36, in accordance with embodiments.
Example Figure 4 illustrates a process of suppressing the effect of false colors, in accordance with embodiments. An IR image by be subjected to noise reduction 38 and subtracted 39 from an RGB image. The subtracted image may be color corrected and white balanced 26. The colorcorrected and/or while balanced image may be converted from RGB to YCbCr color space 28. The luminance components Y of the YCbCr image may be subjected to edge detection 30. In parallel, the corresponding IR image may be subjected edge detection 40. In embodiments, the edge detection 40 on the IR imageand the edge detection 30 on the luminance components Y of the YCbCr are both used to generate an YCb'Cr' image. The Cb and Cr components on both the detected Y edges and the detected IR edges are suppressed 42 to generate suppressed Cb' and Cr' chrominance components, in accordance with embodiments.
By generating suppressed Cb' and Cr' chrominance components, the effect of false colors may be minimized by converting the inconsistent color edges to more consistent monochrome-like edges. The chrominance (Cb and Cr component) of the original image may be reduced to reduce the effect of false colors by converting the inconsistent color edges to more consistent monochrome-like edges. Since edges may be detected from both the IR image and the RGB image, suppression of false colors at edges may be compensated. In embodiments, the amount of suppression may be proportional to the magnitude of the IR edges. The amount of suppression may be proportional to the magnitude of both the Y edges and IR edges, as determined by image processing, in accordance with embodiments. Using the generated suppressed Cb' and Cr' chrominance components and the original luminance Y component, an YCb'Cr' color space is converted 34 back to RGB to generate a color corrected and balanced RGB image 364, in accordance with embodiments.
In some applications, one of the tradeoffs in generating a good quality image using color correction is the level of noise versus color quality. In embodiments, toget high quality color, relatively large coefficients in the color correction matrixmay be appliedto get the best possible color result. However,in embodiments,using relatively large coefficientin the color correction matrix mayamplify noise levels in the image. Accordingly, in embodiments,for the best possible noise results the coefficientsin the color correction matrixshould be as close as possible to 1, while for the best possible color the coefficientsin the color correction matrixmay be significantlylarger than 1.
A method for noise reduction in images detects proximity of a pixel to an edge and adjusts the level of averaging based on the proximity to the edge, in accordance with embodiments. For example, if a pixel is relatively far from an edge, arelativelylarge averaging window is used, while if a pixel isrelativelyclose to an edge a relatively small averaging or no averaging window is used.
Embodiments relate to a combined approach tocolor correction. In embodiments, when a pixel is relatively near to an edge a color correction matrix may be used that producesrelatively low noise but with relatively poor color reproduction. For example, color is best observed by a user in flat areasof an image, whilecolor perception near edges of an image are less pronounced or noticeable.In embodiments, a lower level of color accuracy of the color correction matrix may be less noticeable to a user near the edges depicted in an image compared relatively flat surfaces of an image. Accordingly,in embodiments, for the pixels that are relatively far from the edges depicted in an image, a color correction matrix with relatively poor noise characteristics but relatively good color characteristics may be used. In embodiments, such relatively poor noise characteristics at relatively flat surfaces of an image may bereducedor minimizedby averaging with a relatively large window to reduce the noise that is generated by the color correction matrix.
Embodiments may enable relatively low noise color correction to be performed on edges where it is relatively difficultto perform averaging without causing the image to become blurred. For example, relatively desirable noise reduction color correction (which may have relatively poor color correction) may be performed in regionsof a depicted image wherecolor is less noticeable by the human eye (e.g. edges depicted in the image). In embodiments,regions of a depicted image with limited edge details, a color correction matrix with relatively good color clarity, while sacrificing relatively high noise characteristics may be applied together with noise reduction through averaging to remove the effect of the noise that is amplified by therelatively good color claritycolor correction matrix.
Embodiments relate to multiple color correction matrices which range from having high noise but good color correction to low noise but poor color correction to achieve optimum color correction balanced with minimum noise. In embodiments, an optimum color correction matrix may be chosen from multiple different color correction matrixes, depending on the distance of a pixel from the edge to balance noise and color correction.Embodiments may employnoise reduction kernels, wherethe selection of the noise reduction kernel is based on the distance of a pixel from a depicted edge in an image. Embodiments mayuse interpolation to generate a specific color correction matrix and/or noise reduction kernel based on the distance of a pixel from a depictededge in an image.
It is to be understood that the above descriptions are only illustrative only, and numerous other embodiments can be devised without departing the sprit and scope of the disclosed embodiments.It will be obvious and apparent to those skilled in the art that various modifications and variations can be made in the embodiments disclosed, with the claim scope claimed in plain language by the accompanying claims.

Claims (9)

  1. A method comprising:
    generating a visible light image from visible light pixels comprised in an image sensor pixel array using a first aperture for visible light having a first width;
    generating an infrared image from infrared pixels comprised in the image sensor array using a second aperture for infrared light having a second width smaller than the first with of the first aperture, wherein the infrared image comprises infrared noise;
    identifying depicted edges from at least one of the visible light image or the infrared light image; and
    selectively subtracting the infrared noise from the visible light image based ona distance of an image pixel of the image sensor pixel array fromthe identified depicted edges.
  2. The method of claim 1, whereinthe method reduces infrared noise in an image using a dual-aperture camera.
  3. The method of claim 1, wherein the visible light image comprises red (R), green (G), and blue (B) pixel data or RGB pixel data.
  4. The method of claim 1, wherein the identifying depicted edges is from both the visible light image or the infrared light image.
  5. The method of claim 1, wherein the selectively subtracting the infrared noise from the visible light image is proportional to distance of a pixel from the identified depicted edges.
  6. The method of Claim 5, where the gain of infrared noise suppression is greater at a flat surface of the depicted image than at the identified depicted edges of the depicted image.
  7. The method of Claim 5, where the gain of the infrared noise suppression is approximately 0% at the identified depicted edges.
  8. A method comprising:
    generating a visible light image from visible light pixels comprised in an image sensor pixel array using a first aperture for visible light having a first width;
    generating an infrared image from infrared pixels comprised in the image sensor array using a second aperture for infrared light having a second width smaller than the first with of the first aperture;
    converting the visible light image into a YCbCr color space image;
    identifying depicted edges in Y luminance components of the YCbCr color space image;
    suppressing blue chrominance components Cb and red chrominance components Cr of the YCbCr color space image is based upon the identified depicted edges in the Y luminance components of the YCbCr color space image.
  9. The method of claim 8, comprising:
    identifying depicted edges in the infrared image,
    wherein said suppressing blue chrominance components Cb and red chrominance components Cr of the YCbCr color space image is based upon both the identified depicted edges in the Y luminance components of the YCbCr color space image and the identified depicted edges of the infrared image.
PCT/KR2016/001837 2015-02-26 2016-02-25 Hybrid image correction for dual-aperture camera WO2016137240A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201562121216P 2015-02-26 2015-02-26
US62/121,216 2015-02-26
US14/956,379 US9674466B2 (en) 2015-02-26 2015-12-01 Hybrid image correction for dual-aperture camera
US14/956,379 2015-12-01

Publications (1)

Publication Number Publication Date
WO2016137240A1 true WO2016137240A1 (en) 2016-09-01

Family

ID=56788917

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2016/001837 WO2016137240A1 (en) 2015-02-26 2016-02-25 Hybrid image correction for dual-aperture camera

Country Status (2)

Country Link
US (1) US9674466B2 (en)
WO (1) WO2016137240A1 (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106470309A (en) * 2015-08-19 2017-03-01 中兴通讯股份有限公司 The method and apparatus obtaining the focal length of double shootings
CN106462957B (en) * 2016-05-19 2019-03-08 深圳大学 The minimizing technology and system of fringes noise in a kind of infrared image
US10638060B2 (en) * 2016-06-28 2020-04-28 Intel Corporation Color correction of RGBIR sensor stream based on resolution recovery of RGB and IR channels
CN109756713B (en) 2017-11-08 2021-12-21 超威半导体公司 Image capturing apparatus, method of performing processing, and computer readable medium
WO2019196109A1 (en) * 2018-04-13 2019-10-17 深圳市锐明技术股份有限公司 Method and apparatus for suppressing image pseudo-colour
US20210185285A1 (en) * 2018-09-18 2021-06-17 Zhejiang Uniview Technologies Co., Ltd. Image processing method and apparatus, electronic device, and readable storage medium
CN109302566A (en) * 2018-11-22 2019-02-01 深圳市景阳科技股份有限公司 A kind of method, apparatus and terminal device of determining screening-mode
US11212498B2 (en) * 2018-12-11 2021-12-28 Intel Corporation Infrared crosstalk correction for hybrid RGB-IR sensors
TWI715142B (en) * 2019-08-07 2021-01-01 瑞昱半導體股份有限公司 Image sensing device and method for auto white balance therefor
CN112991468A (en) * 2019-12-17 2021-06-18 深圳君正时代集成电路有限公司 Method for identifying infrared picture and visible light picture by using wavelength information
CN111047592B (en) * 2020-03-13 2020-06-05 武汉理工大学 Monitoring system and image communication processing method thereof
KR20230072702A (en) * 2021-11-18 2023-05-25 삼성전자주식회사 Electronic device and method for performing color conversion
CN114926395B (en) * 2022-04-12 2023-06-16 尚特杰电力科技有限公司 Method and system for detecting infrared image drop strings of photovoltaic panel

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007129361A (en) * 2005-11-01 2007-05-24 Eastman Kodak Co Chromatic aberration suppression circuit and imaging apparatus having the same, and chromatic aberration suppression program
US20100104180A1 (en) * 2008-10-28 2010-04-29 Novatek Microelectronics Corp. Image noise reduction method and image processing apparatus using the same
US20100231759A1 (en) * 2009-03-16 2010-09-16 Fujitsu Microelectronics Limited Image processing apparatus
KR20110101880A (en) * 2010-03-10 2011-09-16 삼성전자주식회사 Method and device for reduction image color noise
US20130033578A1 (en) * 2010-02-19 2013-02-07 Andrew Augustine Wajs Processing multi-aperture image data

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9900478B2 (en) * 2003-09-04 2018-02-20 Flir Systems, Inc. Device attachment with infrared imaging sensor
US10244190B2 (en) * 2009-03-02 2019-03-26 Flir Systems, Inc. Compact multi-spectrum imaging with fusion
US9635285B2 (en) * 2009-03-02 2017-04-25 Flir Systems, Inc. Infrared imaging enhancement with fusion
US9451183B2 (en) * 2009-03-02 2016-09-20 Flir Systems, Inc. Time spaced infrared image enhancement
US9986175B2 (en) * 2009-03-02 2018-05-29 Flir Systems, Inc. Device attachment with infrared imaging sensor
EP2471258B1 (en) * 2009-08-25 2017-05-24 Dual Aperture International Co. Ltd. Reducing noise in a color image
US8824797B2 (en) * 2011-10-03 2014-09-02 Xerox Corporation Graph-based segmentation integrating visible and NIR information
US20160254300A1 (en) * 2015-02-26 2016-09-01 Dual Aperture International Co., Ltd. Sensor for dual-aperture camera

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007129361A (en) * 2005-11-01 2007-05-24 Eastman Kodak Co Chromatic aberration suppression circuit and imaging apparatus having the same, and chromatic aberration suppression program
US20100104180A1 (en) * 2008-10-28 2010-04-29 Novatek Microelectronics Corp. Image noise reduction method and image processing apparatus using the same
US20100231759A1 (en) * 2009-03-16 2010-09-16 Fujitsu Microelectronics Limited Image processing apparatus
US20130033578A1 (en) * 2010-02-19 2013-02-07 Andrew Augustine Wajs Processing multi-aperture image data
KR20110101880A (en) * 2010-03-10 2011-09-16 삼성전자주식회사 Method and device for reduction image color noise

Also Published As

Publication number Publication date
US20160255290A1 (en) 2016-09-01
US9674466B2 (en) 2017-06-06

Similar Documents

Publication Publication Date Title
WO2016137240A1 (en) Hybrid image correction for dual-aperture camera
US8081239B2 (en) Image processing apparatus and image processing method
KR100230391B1 (en) Method for correcting contour components of luminance signal and circuit thereof
US7969480B2 (en) Method of controlling auto white balance
US9830690B2 (en) Wide dynamic range imaging method
WO2006064913A1 (en) Image processing method
US8427560B2 (en) Image processing device
US8648937B2 (en) Image processing apparatus, image processing method, and camera module
JP2007318280A (en) Image processor
JP5439210B2 (en) Image processing device
US20120237124A1 (en) Image chroma noise reduction in the bayer domain
US9071803B2 (en) Image processing apparatus, image pickup apparatus, image processing method and non-transitory computer-readable storage medium storing image processing program
JP2009130841A (en) Video signal processing apparatus and video signal processing method
US9843703B2 (en) Image processing apparatus and method of controlling the same
JP2010068465A (en) Image processing apparatus, image capturing apparatus, and image processing method
CN111770321B (en) Image processing apparatus, image capturing apparatus, image processing method, and storage medium
JP2005354585A (en) Device, method and program of image processing
Tomaselli et al. Automatic detection and correction of purple fringing artifacts through a window based approach
JP2006332732A (en) Apparatus for suppressing color shift noise
TW201514925A (en) Color degradation compensation method
KR20130000537A (en) Apparatus and method of correcting purple fringing
JP2014158165A (en) Image processing device, image processing method, and program
CN111064944B (en) Colorless constant white balance method
KR102315200B1 (en) Image processing apparatus for auto white balance and processing method therefor
JP2009239396A (en) Image processor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16755890

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 02.01.2018)

122 Ep: pct application non-entry in european phase

Ref document number: 16755890

Country of ref document: EP

Kind code of ref document: A1