US20050243178A1 - Method for reducing motion blur in a digital image - Google Patents

Method for reducing motion blur in a digital image Download PDF

Info

Publication number
US20050243178A1
US20050243178A1 US11/122,906 US12290605A US2005243178A1 US 20050243178 A1 US20050243178 A1 US 20050243178A1 US 12290605 A US12290605 A US 12290605A US 2005243178 A1 US2005243178 A1 US 2005243178A1
Authority
US
United States
Prior art keywords
merit
motion blur
image data
image
analyzing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/122,906
Inventor
Charles McConica
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US11/122,906 priority Critical patent/US20050243178A1/en
Publication of US20050243178A1 publication Critical patent/US20050243178A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/68Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681Motion detection
    • H04N23/6811Motion detection based on the image signal
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/68Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/68Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681Motion detection
    • H04N23/6812Motion detection based on additional sensors, e.g. acceleration sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/68Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682Vibration or motion blur correction

Definitions

  • the present invention relates to reducing motion blur in a digital image and, more particularly, to analyzing image data representative of a digital image to determine the amount and direction of motion blur and processing the image data to reduce the motion blur.
  • Digital still cameras generate image data representative of an image of an object.
  • the process of generating image data representative of an image of an object is often referred to simply as “imaging” or “capturing” the object.
  • the image data is processed and output to a device that displays a replicated image of the object.
  • the replicated image of the object may be displayed on a video monitor or printed by a printer.
  • the digital camera focuses the image of the object onto a two-dimensional array of photodetecting elements.
  • the photodetecting elements are relatively small and each one generates image data representative of a very small portion of the image of the object.
  • the two-dimensional array may have several million photodetecting elements that each generate image data representative of a small portion of the image of the object.
  • the image data generated by the individual photodetecting elements is processed to recreate the image of the object.
  • One type of photodetecting element is a charge-coupled device that outputs a voltage that is proportional to the amount of light it receives over a preselected period.
  • the density of photodetecting elements on the two-dimensional array is increased.
  • the increased density of photodetecting elements increases the number of photodetecting elements that image an object, which in turn improves the quality of the image by reducing spaces between imaged portions of the object.
  • Another method of improving the image generated by a digital camera, especially in low light conditions is by using a long period to generate the image data. This long period of image generation is achieved by having the photodetecting elements detect light for an extended period.
  • the motion blur problem is exacerbated as the above-described imaging period is extended.
  • the extended imaging period results in a higher probability that the camera will be moved during the imaging period, which results in a higher probability that motion blur will occur in the replicated image. Accordingly, the benefit of an extended imaging period may be offset by a higher probability of generating a blurred image.
  • the problem of motion blur is further exacerbated by the use of more photodetecting elements to generate an image.
  • the photodetecting elements will be smaller and receive less light. Accordingly, the exposure time of the camera must be extended in order for the smaller photodetecting elements to receive enough light to generate accurate image data.
  • FIG. 1 is a schematic illustration of a digital camera imaging an object.
  • FIG. 2 is a front, enlarged view of a two-dimensional photosensor array used within the digital camera of FIG. 1 .
  • FIGS. 4A and 4B are a flowchart depicting a method of detecting and minimizing motion blur.
  • FIG. 5A is a graph representative of photodetecting element outputs detecting a single point of light without the presence of a blur filter in the camera.
  • FIG. 5B is a graph representative of photodetecting element outputs of the point of light of FIG. 5A with the addition of a blur filter.
  • FIG. 5C is a graph representative of photodetecting element outputs of the point of light of FIG. 5A with the addition of a blur filter and motion blur.
  • FIG. 6A is a graph showing the discrete spatial frequency representation of the graph of FIG. 5A .
  • FIG. 6B is a graph showing the discrete spatial frequency representation of the graph of FIG. 5B .
  • FIG. 6B is a graph showing the discrete spatial frequency representation of the graph of FIG. 5C .
  • FIG. 7 is an enlarged view of photodetecting elements located on the surface of the two-dimensional photosensor array of FIG. 2 .
  • FIG. 8 is a graph showing the representation of FIG. 5C with motion blur minimized.
  • FIGS. 1 through 8 in general, illustrate a method for detecting motion blur in a digital image.
  • the method may comprise providing image data representative of the digital image.
  • the image data is analyzed to calculate a first figure of merit of the digital image in a first direction (X) and a second figure of merit of the digital image in a second direction (Y), wherein the first and the second directions are substantially orthogonal.
  • a first ratio of the first figure of merit to the second figure of merit is calculated, wherein the ratio is the greater of the first or second figure of merit is divided by the lesser of the first or second figure of merit.
  • the first ratio is compared to a preselected value, wherein motion blur exists in the digital image if the first ratio is greater than the preselected value.
  • FIGS. 1 through 8 also, in general, illustrate a method for reducing motion blur in an image.
  • the method may comprise providing image data representative of the image.
  • the image data is analyzed to detect the presence of motion blur in the image.
  • the image data is further analyzed to detect the direction of motion blur in the digital image.
  • the image data is then processed to increase edge acuity the image in the direction of the motion blur.
  • FIGS. 1 through 8 also, in general, illustrate an apparatus for detecting motion blur in an image.
  • the apparatus may comprise a computer and a computer-readable medium operatively associated with the computer, wherein the computer-readable medium contains instructions for controlling the computer to detecting motion blur in an image by the method described below.
  • the apparatus receives image data representative of the image.
  • a first figure of merit of the image data is calculated in a first direction (X).
  • a second figure of merit of the image data is calculated in a second direction (Y), wherein the first and the second directions (X, Y) are substantially orthogonal.
  • a first ratio of the first figure of merit to the second figure of merit is calculated.
  • the first ratio is the greater of the first or the second figure of merit divided by the lesser of the first or the second figure of merit.
  • the first ratio is compared to a preselected value, wherein motion blur exists in the digital image if the first ratio is greater than the preselected value.
  • the direction of motion blur corresponds to the lesser of the figures
  • the camera 100 may have a housing 120 with an aperture 122 formed therein.
  • a lens 126 or a plurality of lenses may be located within or adjacent the aperture 122 and may serve to focus an image of the object 110 onto components located within the camera as is described below.
  • the lens 126 may, as a non-limiting example, have a focal length of approximately seven millimeters.
  • a two-dimensional photosensor array 130 and a processor 132 may also be located within the housing 120 . As illustrated in FIG. 1 , the image of the object 110 is focused onto the two-dimensional photosensor array 130 by the lens 126 . It should be noted that the view of the two-dimensional photosensor array 130 illustrated in FIG. 1 is a side view.
  • the processor 132 may be electrically connected to the two-dimensional photosensor array 130 and may serve to process data generated by the two-dimensional photosensor array 130 as is described in greater detail below.
  • FIG. 2 A front, enlarged view of the two-dimensional photosensor array 130 is illustrated in FIG. 2 .
  • the two-dimensional photosensor array 130 may have a surface 136 to which a plurality of photodetecting elements 138 are mounted. It should be noted that the photodetecting elements illustrated in FIG. 2 have been greatly enlarged for illustration purposes.
  • the photodetecting elements 138 may be arranged so as to form a plurality of rows 140 and columns 142 . Each of the photdetecting elements 138 may be adapted to generate image data representative of the intensity of a specific frequency band of light.
  • a color filter array 143 may be attached to the surface 136 of the two-dimensional photosensor array 130 to filter the light before the light reaches the photodetecting elements 138 .
  • each photodetecting element 138 may receive light having a specific and narrow band of wavelengths.
  • the color filter array 143 shown in FIG. 3 illustrates the Bayer patter, wherein R, G, and B represent color filters that allow specific bands of red, green and blue light to pass to the photodetecting elements 138 . Accordingly, each of the red, green, and blue filters shown in FIG. 3 are associated with single photodetecting elements 138 . It should be noted that the Bayer pattern has twice as many green filters as red and blue. The increased number of green filters compensates for human vision that is more sensitive to green light that red or blue light. It should also be noted that the color filter array 143 may have filters arranged in patterns other than the Bayer patter shown in FIG. 3 .
  • the lens 126 causes a blur of the image of the object 110 onto the photodetecting elements 138 .
  • the blur in the embodiment described herein is equal to approximately two photodetecting elements 138 and serves to prevent aliasing of the image during processing.
  • the processor 132 may be electrically connected to the two-dimensional photosensor array 130 .
  • the processor 132 receives and processes image data generated by the two-dimensional photosensor array 130 .
  • the processor 132 processes information generated by the individual photodetecting elements 138 to “demosaic” the image and to properly combine the colors as described in greater detail below.
  • Some examples of the functions performed by the processor 132 are described in the U.S. patent application, Ser. No. 09/177,729 for IMAGE DEMOSAICING AND ENHANCEMENT SYSTEM of Taubman, filed on Oct. 23, 1998, which is hereby incorporated by reference for all that is disclosed therein. It should be noted that the processor 132 described herein has the additional ability to detect and minimize motion blur in an image.
  • the processor 132 may also be electrically connected to a peripheral viewing device, not shown, such as a computer monitor or a printer that serves to display and/or process the image data generated by the camera 100 .
  • the processor 132 may also be electrically connected to a peripheral computer, not shown, that stores and/or processes the image data generated by the two-dimensional photosensor array 130 and, thus, the camera 100 . It should be noted that the processing techniques and methods described herein with reference to the processor 132 may be performed by the peripheral viewing device or computer. For example, image data generated by the two-dimensional photosensor array 130 may be transmitted directly to a peripheral processor for processing.
  • the camera 100 is typically held by a user and used to generate image data representative of an object, such as the object 110 .
  • the image data typically represents a still picture of the object 110 , however, the image data may represent a moving picture, e.g., a motion picture, of the object 110 .
  • Light 150 reflects from the object 110 and enters the housing 120 of the camera 100 via the aperture 122 .
  • the lens 126 then focuses an image of the object 110 onto the two-dimensional photosensor array 130 .
  • the two-dimensional photosensor array 130 generates image data representative of the image of the object 110 , which is output to the processor 132 for processing.
  • the lens 126 blurs the image focused onto the two-dimensional photosensor array 130 by two photodetecting elements 138 , FIG. 2 .
  • FIG. 2 illustrates an example of an image 156 of the object 110 , FIG. 1 , focused onto the two-dimensional photosensor array 130 .
  • an instruction is transmitted to the two-dimensional photosensor array 130 , which causes the photodetecting elements 138 to generate image data. More specifically, the photodetecting elements 138 develop electrical charges that are dependent on the amount of light they receive over a preselected period that is reflected from the object 110 .
  • the preselected period is relatively long and when the camera 100 is used in bright environments, the preselected period is relatively short.
  • the image data may be processed as a plurality of individual color planes.
  • the image data may have a green plane, a red plane, and a blue plane.
  • the green plane may be processed as two green planes because there are twice as many green photodetecting elements 138 as red or blue photodetecting elements 138 .
  • the green plane consists of image data generated by photodetecting elements 138 having a green color filter of the color filter array 143 associated with them.
  • the red and blue planes consist of image data generated by photodetecting elements 138 having red and blue color filters of the color filter array 143 respectively associated with them.
  • These color planes allow for each color component of the composite image data to be processed individually. For example, individual colors may be scaled to increase or decrease their relative intensity. Likewise, image data from individual photodetecting elements 138 may be scaled accordingly.
  • the color filter array 143 provides a reference for directions used herein, including the directions of motion blur.
  • a y-direction Y extends perpendicular to the x-direction X that was described above.
  • An a-direction A extends diagonally to the x-direction X and the y-direction Y.
  • a b-direction B extends perpendicular to the a-direction A.
  • both the camera 100 and the object 110 remain stationary relative to each other during the above-described preselected period in which the two-dimensional photosensor array 130 generates image data.
  • Ideal circumstances are rarely achieved and the camera 100 generally moves somewhat relative to the object 110 while image data is being generated. For example, a user typically is unable to maintain the camera 100 in a fixed position relative to the object 110 while image data is being generated.
  • Motion blur is typically attributed to either rotational or translational movement of the camera 100 relative to the object 110 occurring as the object 110 is being imaged.
  • the motion blur correction methods described herein are directed toward correcting motion blur caused by translational motion between the object 110 and the camera 100 as the object 100 is being imaged. It should be noted, however, that motion blur caused by rotational motion may also be corrected by the methods described herein. For example, if the center of relative motion between the camera 100 and the object 110 is a significant distance from the image area, the motion blur associated with the image may be substantially similar to motion blur caused by translational motion.
  • One test to determine if motion blur attributed to rotational motion can be corrected as though it is attributed to translational motion is my measuring the blur on the image closest to the center of rotation and furthest from the center of rotation. If the two blurs are substantially equivalent, the motion blur may be corrected as though it is the result of translational motion. In one embodiment of the methods described herein, the blurs at the edges of the image are measured to determine if they are substantially equivalent, meaning that the blur is the result of translational motion.
  • the operation of the camera 100 will now be described generating image data that is blurred.
  • the camera 100 has moved in the x-direction X as the image data was being generated.
  • the movement in the x-direction X is an amount that causes the image 156 to move a distance of two and one-half photodetecting elements 138 .
  • the movement of the camera 100 relative to the object 110 as the two-dimensional photosensor array 130 generates image data results in the image of the object 110 being blurred. More specifically, when the image data is processed using conventional processing methods, the replicated image of the object 110 will be blurred. An example of this blurring is illustrated in FIG. 2 .
  • the image 156 of the object 110 FIG. 1 , is shown focused onto the two-dimensional photosensor array 130 at the commencement of the generation of the image data. As the generation of the image data by the photodetecting elements 138 proceeds, the image 156 focused onto the two-dimensional photosensor array 130 moves in the negative x-direction XN.
  • the image 156 has moved to a different location on the two-dimensional photosensor array 130 .
  • the location of the image of the object 110 , FIG. 1 , at the termination of the generation of image data is referenced as the second image 158 and is illustrated by dashed lines of FIG. 2 .
  • the image 156 has moved a distance that is represented by two and one-half columns 142 in the negative x-direction XN during the generation of the image data.
  • the distance that the image 156 moved is proportional to the amplitude of the motion blur.
  • the direction that the image 156 moved is referred to as the direction of the motion blur.
  • image data is analyzed to determine the direction and amplitude of the blur.
  • the amplitude is the amount of movement that occurred during the generation of the image data.
  • the direction of motion blur as described herein is, for illustration purposes only, limited to the x-direction X, the y-direction Y, the a-direction A and the b-direction B. It should be noted that the motion blur detection and correction methods described herein are applicable to a plurality of amplitudes and directions.
  • the image data generated by the two-dimensional photosensor array 130 is transmitted to the processor 132 for processing.
  • the processor 132 correlates the image data generated by each of the photodetecting elements 138 , FIG. 2 , with its associated color filter per the color filter array 143 of FIG. 3 .
  • the image data may be sorted into the three color groups or planes described above, which are the red plane, the green plane, and the blue plane. Alternatively, the image data may be sorted into four groups if two green color planes are considered.
  • the processor 132 may store a numeric value corresponding to the image data values, or simply image data, generated by each of the photodetecting elements 138 .
  • the image data generated by each of the photodetecting elements 138 is proportional to the intensity of light received by each of the photodetecting elements 138 .
  • the processor 132 may also store the location from where each value of the image data was generated on the two-dimensional photosensor array 130 .
  • Image data generated by the two-dimensional photosensor array 130 is representative of the amplitude of light at a plurality of spatial locations within the area of the image.
  • the image data is representative of the amplitude of light at a plurality of spatial locations for a plurality of color planes.
  • the image data may be represented in three color planes, a red plane, a green plane, and a blue plane.
  • the image data of each color plane may be transformed to a frequency domain by application of a Fourier transform and is sometimes referred to as the “transformed” image data.
  • the transformed image data represents an amplitude and a phase for each of a set of spatial frequencies.
  • the transformed image data offers a perspective on the image data relating to the amplitude and direction of blur within the image.
  • the transformed image data may be manipulated as described below to reduce motion blur.
  • the image data may then be retransformed by way of an inverse Fourier transform to its original format for presentation to a user.
  • the motion blur in the retransformed image data has been reduced by the manipulation.
  • Motion blur reduces the amplitude of the transformed image data, especially in the higher spatial frequencies and always in the direction of the blur motion.
  • the image data is transformed into the frequency domain using a Fourier transform or the like, the magnitude of the amplitude of sinusoidal signals throughout the set of spatial frequencies in a direction perpendicular to the direction of the motion remains virtually unaffected. The same occurs with regard to directions that are close to perpendicular to the direction of the motion.
  • FIGS. 5A through 5C An example of the spatial affects of motion blur are shown in the graphs of FIGS. 5A through 5C .
  • the graphs 5 A through 5 C are, for illustration purposes, based on a linear view of the two-dimensional photosensor array 130 rather than a planar view of the two-dimensional photosensor array 130 .
  • the horizontal axes P in the graphs 5 A through 5 C represent spatial locations, and more particularly photodetecting element locations or pixel locations, from where image data was generated. Accordingly, the horizontal axes correspond to the locations of the photodetecting elements 138 of FIG. 2 .
  • the vertical axes I(P) represent the intensity of light received at the photodetecting element locations, which are values or magnitudes of the image data generated by the photodetecting elements.
  • the graphs of FIGS. 6A through 6C represent the magnitudes of the amplitudes of the spatial frequencies of the graphs of FIGS. 5A through 5C respectively.
  • the graph of FIG. 5A shows the ideal representation of image data I(P) of a single point of light that would be generated by a two-dimensional photosensor array without a blur filter associated therewith.
  • a single photodetecting element, eight or spatial location eight generates image data having a magnitude of eight.
  • the graph of FIG. 5B is a representation of image data I(P) of the single point of light of FIG. 5A that would be generated by a two-dimensional photosensor array having a blur filter of two pixels associated therewith.
  • the blur filter causes the image data I(P) to have a reduced high spatial frequency content, which is evident by the discrete pulse of the graph of FIG.
  • the magnitude of the spatial location eight drops to an amplitude of four and the magnitudes of spatial locations seven and nine increase from zero to two.
  • the graph of FIG. 5B is considered the “perfect” point of light as imaged by the camera with the two pixels of blur. As the number of photodetecting elements in a digital camera increases, this “perfect” image represented by the graph of FIG. 5B more closely resembles the point of light of FIG. 5A . Accordingly, the negative affects of the blur filter become insignificant.
  • blur filters such as a blurry lens, a birefringent filter or lens, or other blur filter device.
  • the blur filter is to prevent objectionable aliased signal artifacts in the image. Aliased signal artifacts are typically associated with large areas of moderately high spatial frequency repetitive patters, such as the weave of a shirt.
  • the graph of FIG. 5C shows the amplitude of image data received by a camera having a blur filter associated therewith and undergoing motion equivalent to two and one-half pixels or photodetecting elements as image data is being generated.
  • the energy from a single point of light FIG. 5A
  • the amplitude of image data generated at the sixth and tenth spatial locations has increased to 0.6.
  • the amplitude of image data generated at the seventh and ninth spatial locations have decreased of two.
  • the amplitude of image data generated at the eighth spatial location has decreased to 2.8.
  • the image data generated with respect to the spatial locations represents a very wide triangular function. This is indicative of a very low magnitude of high spatial frequency content of the replicated image, meaning that the replicated image is not sharp and is likely blurred.
  • motion blur in the replicated image is reduced by increasing the magnitude of the amplitude of the sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in the direction of the motion blur.
  • sinusoidal signals are used herein for illustration purposes and that they are derived by way of a Fourier transform. It should also be noted that the inventive concepts described herein are applicable to other signals derived from other transform functions.
  • Different sharpening kernels may be applied to the image data in the direction of the motion blur to increase the magnitude of the amplitude of the high spatial frequency content.
  • a Weiner correction algorithm is applied to the frequency domain transformed image data in the direction of the motion blur to increase the magnitude of the amplitude of a specific set of spatial frequency content. It should be noted that other correction algorithms may be applied to the image data to reduce motion blur.
  • motion blur due to translational motion unlike other blurs, only occurs in a single direction.
  • the motion blur only occurs in the x-direction X.
  • Other blurs, such as optical blur typically occur in all directions.
  • the magnitude of the amplitude of the high spatial frequency content of an image affected by motion blur will be low in the direction corresponding to the direction of the motion.
  • the magnitude of the amplitude of the high spatial frequency content corresponding to directions perpendicular or substantially perpendicular to the direction of motion will be less.
  • the magnitude of the amplitude of the high spatial frequency content in the direction of the motion blur will be significantly lower than the magnitude of the amplitude of the same specific high spatial frequency content in a perpendicular direction.
  • the detection of motion blur may be accomplished by analyzing image data generated by photodetecting elements 138 located in a portion of the two-dimensional photosensor array 130 .
  • a user of a camera typically assures that the subject of the photograph is focused and substantially centered in the photograph. Accordingly, the center of the photograph is where the sharpest focused image is located. For this reason, in the examples described herein, image data generated by a central portion of the two-dimensional photosensor array 130 will be analyzed for motion blur. For example, image data generated by the middle one-ninth of the two-dimensional photosensor array 130 may be analyzed for motion blur. It should be understood, however, that any portion of the two-dimensional photosensor array 130 may be analyzed for motion blur.
  • FIG. 7 is an exploded view of the photodetecting elements 138 located in the corner 160 of the two-dimensional photosensor array 130 .
  • Image data generated by a row 164 of photodetecting elements 138 extending in the x-direction X will be analyzed in the following example.
  • image data generated by a column 166 of photodetecting elements 138 extending in the y-direction Y will also be analyzed in the following example.
  • the photodetecting elements 138 in the row 164 are referenced as X(1), Y(1) through X(10), Y(1) and the photodetecting elements 138 in the column 166 are referenced as X(1), Y(1) through X(1), Y(10). It should be noted that for illustration purposes, only ten photodetecting elements 138 extending in the x-direction X and the y-direction Y are being analyzed. It is to be understood, however, that any number of photodetecting elements 138 in the x-direction X and the y-direction Y may be analyzed. It is also to be further understood that any number of photodetecting elements 138 extending in the a-direction A and the b-direction B may also be analyzed to detect motion blur in other directions.
  • the first step in determining whether motion blur exists in the image data is to analyze the magnitude of the amplitude of the high spatial frequency content in orthogonal directions over an area of an image.
  • the magnitude of the amplitude of the high spatial frequency content in the x-direction X and the y-direction Y is proportional to the figure of merit in the x-direction X and the y-direction Y respectively.
  • the figure of merit provides a basis for determining the degree to which light/dark transitions occur in a specific direction in an image.
  • Fx A non-limiting example of a figure of merit in the x-direction X is referred to herein as Fx and is calculated as follows:
  • the figures of merits described herein are non-limiting examples of figures of merit.
  • Other variations of the figure of merit can use weighted multiple color planes.
  • the figures of merit may be calculated in all the color planes. The values from the different color planes can be weighted and summed togther. For example, each green plane may be weighted thirty percent, the red plane may be weighted thirty percent and the blue plane may be weighted ten percent.
  • the spatial frequencies may be weighted. It is also possible to combine weighted color planes with weighted spatial frequencies.
  • higher order numerical methods may be used to generate the slope estimates. These slope estimate may be used rather than the simple difference calculations described above.
  • the figures of merit provide indications of the magnitudes of the amplitudes of spatial frequencies of the image in the specified directions.
  • the graph shows a steep transition between adjacent photodetecting elements for image data generated without motion blur and without a blur filter. More precisely, the single point of light has caused image data to be generated solely at the eighth spatial position. Accordingly, the figure of merit associated with the graph of FIG. 5A is high.
  • the graph of FIG. 5B shows transitions between image data values that are not as steep as the graph of FIG. 5A . Accordingly, the figure of merit associated with the graph of FIG. 5B is less that the figure of merit associated with the graph of FIG. 5A .
  • the graph of FIG. 5C shows image data generated in the presence of motion blur.
  • the motion blur decreases the magnitudes of the transitions between image data values of adjacent photodetecting elements 138 , FIG. 2 . Accordingly, the figure of merit of the image data associated with the graph of FIG. 5C is less than the figures of merit associated with the graphs of FIG. 5A and FIG. 5B . It should also be noted that the figure of merit as described herein is proportional to the magnitude of the slope of the graphs. By using the figure of merit equations described above, the figures of merit for the graphs of FIGS. 5A is sixteen, the figure of merit of the graph of FIG. 5B is eight, and the figure of merit of the graph of FIG. 5C is 5.6.
  • motion blur due to translational motion occurs only in one direction. Accordingly, if the figure of merit in the x-direction X, F X , differs from the figure of merit in the y-direction Y, F Y , motion blur likely occurred during the generation of the image data.
  • This difference in the figures of merit may be calculated by taking the ratio of F X to F Y or F Y to F X and comparing the ratio to a preselected value. As a non-limiting example, if the ratio is greater than 1.4, it can be assumed that motion blur occurred. It follows that the direction of the motion blur will be in the direction having the lower figure of merit because the magnitudes of the transitions in this direction are lower.
  • the value of the ratio determines the amount of motion blur. For example, a greater value of the ratio means that the image was blurred over a greater number of photodetecting elements 138 , FIG. 2 .
  • the value of the ratio is used to determine the amount to which motion blur is to be minimized.
  • the non-limiting methods described herein for determining the direction of motion blur determine the direction of motion blur closest to a preselected direction. For example, the methods described herein may determine the direction of motion blur that is closest to the x-direction X, the y-direction Y, the a-direction A or the b-direction B. Similar methods can be used to expand the possible number of directions in which motion blur may be analyzed.
  • the image data is processed to minimize the blur of the replicated image.
  • Minimizing motion blur involves increasing the magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image data in the direction of the blur.
  • a non-limiting example may be achieved by amplifying image data values corresponding to higher spatial frequencies.
  • the image data generated by individual rows 140 , FIG. 2 , of photodetecting elements 138 are processed to increase the magnitude of the amplitude of spatial frequency.
  • Other scaling and processing techniques as described below may also be applied to the image data.
  • image data from one color plane may be analyzed to determine the direction and amount of motion blur.
  • the magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image of the color planes may be increased equally in order to minimize the affects of motion blur on the whole composite image.
  • processing the image data to minimize the motion blur may be performed during the process of demosaicing the image data as disclosed in the United States patent application of Taubman, previously referenced. This may, as an example, involve scaling demosaicing coefficients in the direction of the motion blur to increase the magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in the direction of the motion blur.
  • Other techniques can be used to reduce motion blur, such as directional sharpening kernels, windowed directional sharpening kernels, and directional deconvolution correction.
  • the procedure described above determines the amount and direction of motion blur. It should be noted that the procedure described above is applicable for detecting and minimizing motion blur along any orthogonal directions. For illustration purposes, an example of detecting and minimizing motion blur in additional directions diagonal to the x-direction X and the y-direction Y is provided. The diagonal directions are referenced in FIG. 3 as the a-direction A and the b-direction B, which are perpendicular to each other. In this situation, the figures of merit are calculated in the a-direction A and the b-direction B along with the ratio of the figures of merit as described above with reference to the x-direction X and the y-direction Y.
  • ratios exist, one for figures of merit in the x and y-directions X and Y and one for figures of merit in the a and b-directions A and B.
  • the ratio having the greatest value indicates the orthogonal directions associated with the motion blur.
  • the actual direction of the motion blur is determined by the lowest figure of merit associated with the ratio having the greatest value.
  • Motion blur in this direction is minimized in the manner described herein by increasing the magnitude of the amplitude of the magnitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in this direction.
  • the following example is based on the imaged point of light of FIGS. 5A through 5C .
  • the point of light without the use of a blur filter appears solely at the eighth spatial position and generates image data having a magnitude of eight.
  • the point of light imaged by a camera having a two-pixel blur filter and undergoing motion blur of two and one half pixels generates image data as described above with reference to the graph of FIG. 5C .
  • the values of spatial frequency of the graph of FIG. 5C are significantly less that the values of spatial frequency of the graph of FIG. 5A .
  • the values of the spatial frequencies of the graphs of FIGS. 5A through 5C are shown in the bar graphs of FIGS. 6A through 6C .
  • the graphs of FIGS. 6A through 6C do not include phase information.
  • the single point of light of FIG. 5A has a normalized amplitude of one across the frequency spectrum, similar to an impulse function.
  • the graph of FIG. 7B shows the decreased spatial frequency values due to the blur filter added to the camera.
  • FIG. 7C shows that the values of spatial frequency are even more decreased by motion blur.
  • the values of the spatial frequencies due to the blur filter and motion blur are detailed in Table 1. It should be noted that in the spatial frequency eight is the highest and spatial frequencies zero and fifteen are the lowest.
  • motion blur attenuated the high spatial frequency content of the transformed image. More specifically, the values of the high spatial frequencies of five, six, seven, and eight are attenuated. Motion blur did not attenuate the values of the lower spatial frequencies.
  • Various image sharpening kernels can be applied to the image data in the direction of the motion blur in order to increase the magnitude of the amplitude of the high spatial frequency content.
  • the result of the increased amplitude of the high spatial frequency content is a sharper image, which is a reduction of the blur caused by motion. More specifically, the transitions between light and dark areas are accentuated.
  • Table 1 shows the Weiner correction factor that is applied to the signal at the specific spatial frequencies in the direction of the motion blur so as to reduce motion blur. It should be noted that the Weiner correction factor used herein has a minimal impact on the high spatial frequency content. As shown in Table 1, the frequencies seven and eight are minimally impacted by the Weiner correction factor applied herein. This is because the very high spatial frequency content of the image data tends to include a lot of noise. Thus, if the very high spatial frequency content is amplified, the noise will likely be amplified more than the signal.
  • FIG. 8 is a graph of the point of light of FIG. 5C imaged under a motion blur condition, but with the Weiner correction factor applied.
  • the original motion blurred image of FIG. 5C is shown in dashed lines in FIG. 8 for reference purposes.
  • the values of the blurred image of FIG. 5C are compared to the values of the corrected image of FIG. 8 in Table 2 under the portion titled Weiner Corrected Image.
  • the Wiener corrected image factor corrected the image for two pixels or photodetecting elements 138 , FIG. 2 , of motion blur.
  • the Weiner correction algorithm may be modified to correct for any number of pixels of motion blur.
  • the Weiner corrected image restores the image to a close approximation to the image represented by the blur filter column. It should be noted that the value of the highest amplitude at the spatial location nine ideally would be restored to a value of four, however, the Weiner correction factor as described herein restored it to a value of 3.37 from its motion blurred value of 2.8. It should also be noted that the Weiner correction factor illustrated herein has very few negative numbers, which have very small magnitudes. High magnitude negative numbers tend to increase the Gibbs effect, which is detrimental to image quality. The ⁇ 1, 3, ⁇ 1 sharpening kernel did increase the value of the highest spatial frequency. However, the value of 4.4 is much greater than the ideal value of 4.0, which increases the noise and is actually detrimental to the image quality.
  • the method described herein corrects for two pixels of motion blur. It is to be understood, however, that the method may be modified to correct for different amounts of motion blur.
  • a determination is made as to the amount of motion blur in the image.
  • the image is then ‘deblurred’ an amount based on the amount of blur in the image.
  • the method may be able to correct for two or four pixels of motion blur. If the image is determined to have greater than one and less than three pixels of motion blur, the image may be corrected for two pixels of motion blur. If the image is determined to have three to five pixels of motion blur, the image is corrected for four pixels of motion blur.
  • Motion blur has been described herein as being accomplished by analyzing image data representative of an object.
  • Motion blur may also be detected by using a sensor or plurality of sensors mounted to the camera 100 , FIG. 1 , that detect the amount and direction of movement of the camera during the period in which image data is being generated. The direction and amount of motion blur then establish the coefficients for the image sharpening algorithms to reduce the motion blur.
  • the detection and minimization of motion blur has been described herein as being performed by the processor 132 located within the camera 100 .
  • the processor 132 may facilitate the transfer of image data to a peripheral processor that may detect and minimize motion blur. For example, a blurred photograph may be captured or otherwise scanned so as to generate image data representative of the object.
  • the figures of merit have been calculated based on image data that is generated by photodetecting elements 138 , FIG. 2 , that are located next to each other or next to each other in a given color plane.
  • One embodiment of the above-described method bases the calculation of the figures of merit on image data that is generated by photodetecting elements 138 that are spaced a distance from each other. This will measure the figure of merit at a different spatial frequency.
  • the figures of merit may be calculated based on image data generated by every other or every third photodetecting element 138 . This embodiment may be used in processing information from digital cameras having a great number of photodetecting elements 138 . By not processing the image data generated by all the photodetecting elements 138 , the time required to determine motion blur is decreased.
  • small portions of the image are analyzed to determine if motion blur exists in a small portion of the image.
  • the motion blur is then minimized in that small section.
  • the camera may be held stationary relative to a background as image data is being generated.
  • An object being imaged may be moving relative to the camera and the background as image data is being generated. The method described above, will reduce the motion blur of the image of the object without creating blur in the background.
  • the above described method of minimizing motion blur becomes more useful as the density of photodetecting elements located on a two-dimensional photosensor array increases.
  • the purpose of having an increased density of photodetecting elements is to provide a more defined or sharper image.
  • the susceptibility to motion blur increases as the density of photodetecting elements increases.
  • the image of the object is more likely to shift between photodetecting elements when the density of photodetecting elements is high. This shifting will be detected as motion blur and will be minimized accordingly.
  • the above described method of minimizing motion blur also becomes more useful in generating image data in relatively dark environments.
  • the period required by the photodetecting elements to generate image data is proportional to the amount of light reflecting from the object being imaged. Accordingly, when imaging is performed in a relatively dark environment, the period required to generate the image data increases. This increased period increases the likelihood that the user of the camera will move the camera while image data is being generated, which will cause motion blur.
  • the motion blur can be minimized.

Abstract

A method for reducing motion blur in a digital image is disclosed. An embodiment of the method comprises increasing the magnitude of the amplitude of the signals in a preselected set of spatial frequencies of the image in the direction of the motion blur.

Description

  • This application is a continuation application of Ser. No. 09/867,352 of Charles H. McConica for METHOD FOR REDUCING MOTION BLUR IN A DIGITAL IMAGE filed on May 29, 2001, which is hereby incorporated for all that is disclosed therein.
  • METHOD FOR REDUCING MOTION BLUR IN A DIGITAL IMAGE TECHNICAL FIELD OF THE INVENTION
  • The present invention relates to reducing motion blur in a digital image and, more particularly, to analyzing image data representative of a digital image to determine the amount and direction of motion blur and processing the image data to reduce the motion blur.
  • BACKGROUND OF THE INVENTION
  • Digital still cameras generate image data representative of an image of an object. The process of generating image data representative of an image of an object is often referred to simply as “imaging” or “capturing” the object. The image data is processed and output to a device that displays a replicated image of the object. For example, the replicated image of the object may be displayed on a video monitor or printed by a printer.
  • The digital camera focuses the image of the object onto a two-dimensional array of photodetecting elements. The photodetecting elements are relatively small and each one generates image data representative of a very small portion of the image of the object. For example, the two-dimensional array may have several million photodetecting elements that each generate image data representative of a small portion of the image of the object. The image data generated by the individual photodetecting elements is processed to recreate the image of the object. One type of photodetecting element is a charge-coupled device that outputs a voltage that is proportional to the amount of light it receives over a preselected period.
  • In order to improve the image generated by a digital camera, the density of photodetecting elements on the two-dimensional array is increased. The increased density of photodetecting elements increases the number of photodetecting elements that image an object, which in turn improves the quality of the image by reducing spaces between imaged portions of the object. Another method of improving the image generated by a digital camera, especially in low light conditions, is by using a long period to generate the image data. This long period of image generation is achieved by having the photodetecting elements detect light for an extended period.
  • One problem with photography, including digital photography, is that the image generated by the camera will be blurred if the camera moves as the photodetecting elements generate image data. For example, under ideal conditions of no movement in the digital camera relative to the object being imaged, each photodetecting element generates image data representative of a particular portion of the image of the object. If, however, the camera is moved as the image data is generated, the individual portions of the image of the object will be imaged by several photodetecting elements. Accordingly, each photodetecting element images several different portions of the image of the object, which causes the replicated image of the object to be blurred. This type of blur is referred to as motion blur.
  • The motion blur problem is exacerbated as the above-described imaging period is extended. The extended imaging period results in a higher probability that the camera will be moved during the imaging period, which results in a higher probability that motion blur will occur in the replicated image. Accordingly, the benefit of an extended imaging period may be offset by a higher probability of generating a blurred image. The problem of motion blur is further exacerbated by the use of more photodetecting elements to generate an image. The photodetecting elements will be smaller and receive less light. Accordingly, the exposure time of the camera must be extended in order for the smaller photodetecting elements to receive enough light to generate accurate image data.
  • Therefore, a need exists for a method and device for detecting and reducing motion blur in a digital image.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a schematic illustration of a digital camera imaging an object.
  • FIG. 2 is a front, enlarged view of a two-dimensional photosensor array used within the digital camera of FIG. 1.
  • FIG. 3 is a color filter array using the Bayer pattern.
  • FIGS. 4A and 4B are a flowchart depicting a method of detecting and minimizing motion blur.
  • FIG. 5A is a graph representative of photodetecting element outputs detecting a single point of light without the presence of a blur filter in the camera.
  • FIG. 5B is a graph representative of photodetecting element outputs of the point of light of FIG. 5A with the addition of a blur filter.
  • FIG. 5C is a graph representative of photodetecting element outputs of the point of light of FIG. 5A with the addition of a blur filter and motion blur.
  • FIG. 6A is a graph showing the discrete spatial frequency representation of the graph of FIG. 5A.
  • FIG. 6B is a graph showing the discrete spatial frequency representation of the graph of FIG. 5B.
  • FIG. 6B is a graph showing the discrete spatial frequency representation of the graph of FIG. 5C.
  • FIG. 7 is an enlarged view of photodetecting elements located on the surface of the two-dimensional photosensor array of FIG. 2.
  • FIG. 8 is a graph showing the representation of FIG. 5C with motion blur minimized.
  • DETAILED DESCRIPTION OF THE INVENTION
  • A method and apparatus for detecting and reducing motion blur in an image is generally described below followed by a more detailed description.
  • FIGS. 1 through 8, in general, illustrate a method for detecting motion blur in a digital image. The method may comprise providing image data representative of the digital image. The image data is analyzed to calculate a first figure of merit of the digital image in a first direction (X) and a second figure of merit of the digital image in a second direction (Y), wherein the first and the second directions are substantially orthogonal. A first ratio of the first figure of merit to the second figure of merit is calculated, wherein the ratio is the greater of the first or second figure of merit is divided by the lesser of the first or second figure of merit. The first ratio is compared to a preselected value, wherein motion blur exists in the digital image if the first ratio is greater than the preselected value.
  • FIGS. 1 through 8 also, in general, illustrate a method for reducing motion blur in an image. The method may comprise providing image data representative of the image. The image data is analyzed to detect the presence of motion blur in the image. The image data is further analyzed to detect the direction of motion blur in the digital image. The image data is then processed to increase edge acuity the image in the direction of the motion blur.
  • FIGS. 1 through 8 also, in general, illustrate an apparatus for detecting motion blur in an image. The apparatus may comprise a computer and a computer-readable medium operatively associated with the computer, wherein the computer-readable medium contains instructions for controlling the computer to detecting motion blur in an image by the method described below. The apparatus receives image data representative of the image. A first figure of merit of the image data is calculated in a first direction (X). A second figure of merit of the image data is calculated in a second direction (Y), wherein the first and the second directions (X, Y) are substantially orthogonal. A first ratio of the first figure of merit to the second figure of merit is calculated. The first ratio is the greater of the first or the second figure of merit divided by the lesser of the first or the second figure of merit. The first ratio is compared to a preselected value, wherein motion blur exists in the digital image if the first ratio is greater than the preselected value. The direction of motion blur corresponds to the lesser of the figures of merit.
  • Having generally described the digital camera 100 and a method for reducing blur in a digital image, they will now be described in greater detail. The following description focuses on the operation of the camera 100 followed by detection and reduction of motion blur in images generated by the camera 100.
  • A schematic illustration of a digital camera 100 generating image data representative of an object 110 is illustrated in FIG. 1. The process of generating image data representative of an object is sometimes referred to simply as imaging or capturing the object. An x-direction X and a z-direction Z are used for reference purposes with regard to the camera 100 and the object 110 shown in FIG. 1. The x-direction X is described in greater detail with reference to a positive x-direction XP and a negative x-direction XN. Likewise, the z-direction Z is described in greater detail with reference to a positive z-direction ZP and a negative z-direction ZN A y-direction, not shown in FIG. 1, extends perpendicular to both the x-direction X and the z-direction Z and is described below.
  • The camera 100 may have a housing 120 with an aperture 122 formed therein. A lens 126 or a plurality of lenses may be located within or adjacent the aperture 122 and may serve to focus an image of the object 110 onto components located within the camera as is described below. The lens 126 may, as a non-limiting example, have a focal length of approximately seven millimeters. A two-dimensional photosensor array 130 and a processor 132 may also be located within the housing 120. As illustrated in FIG. 1, the image of the object 110 is focused onto the two-dimensional photosensor array 130 by the lens 126. It should be noted that the view of the two-dimensional photosensor array 130 illustrated in FIG. 1 is a side view. The processor 132 may be electrically connected to the two-dimensional photosensor array 130 and may serve to process data generated by the two-dimensional photosensor array 130 as is described in greater detail below.
  • A front, enlarged view of the two-dimensional photosensor array 130 is illustrated in FIG. 2. The two-dimensional photosensor array 130 may have a surface 136 to which a plurality of photodetecting elements 138 are mounted. It should be noted that the photodetecting elements illustrated in FIG. 2 have been greatly enlarged for illustration purposes. The photodetecting elements 138 may be arranged so as to form a plurality of rows 140 and columns 142. Each of the photdetecting elements 138 may be adapted to generate image data representative of the intensity of a specific frequency band of light. For example, with additional reference to FIG. 3, a color filter array 143 may be attached to the surface 136 of the two-dimensional photosensor array 130 to filter the light before the light reaches the photodetecting elements 138. Accordingly, each photodetecting element 138 may receive light having a specific and narrow band of wavelengths. The color filter array 143 shown in FIG. 3 illustrates the Bayer patter, wherein R, G, and B represent color filters that allow specific bands of red, green and blue light to pass to the photodetecting elements 138. Accordingly, each of the red, green, and blue filters shown in FIG. 3 are associated with single photodetecting elements 138. It should be noted that the Bayer pattern has twice as many green filters as red and blue. The increased number of green filters compensates for human vision that is more sensitive to green light that red or blue light. It should also be noted that the color filter array 143 may have filters arranged in patterns other than the Bayer patter shown in FIG. 3.
  • The photodetecting elements 138 may be charge-coupled devices that develop a charge that is proportional to the amount of light they receive during a preselected period. The preselected period is dependent on the intensity of light to which the object 110, FIG. 1, is exposed and, thus, the intensity of light 150 that reflects from the object 110. For example, when an image is generated in a relatively dark environment, the preselected period is increased to enhance the generated image. Likewise, when an image is generated in a relatively bright environment, the preselected period is decreased.
  • Referring to FIGS. 1 and 2, in the embodiment of the camera 100 described herein, the lens 126 causes a blur of the image of the object 110 onto the photodetecting elements 138. The blur in the embodiment described herein is equal to approximately two photodetecting elements 138 and serves to prevent aliasing of the image during processing.
  • Referring again to FIG. 1, as described above, the processor 132 may be electrically connected to the two-dimensional photosensor array 130. The processor 132 receives and processes image data generated by the two-dimensional photosensor array 130. For example, the processor 132 processes information generated by the individual photodetecting elements 138 to “demosaic” the image and to properly combine the colors as described in greater detail below. Some examples of the functions performed by the processor 132 are described in the U.S. patent application, Ser. No. 09/177,729 for IMAGE DEMOSAICING AND ENHANCEMENT SYSTEM of Taubman, filed on Oct. 23, 1998, which is hereby incorporated by reference for all that is disclosed therein. It should be noted that the processor 132 described herein has the additional ability to detect and minimize motion blur in an image.
  • The processor 132 may also be electrically connected to a peripheral viewing device, not shown, such as a computer monitor or a printer that serves to display and/or process the image data generated by the camera 100. The processor 132 may also be electrically connected to a peripheral computer, not shown, that stores and/or processes the image data generated by the two-dimensional photosensor array 130 and, thus, the camera 100. It should be noted that the processing techniques and methods described herein with reference to the processor 132 may be performed by the peripheral viewing device or computer. For example, image data generated by the two-dimensional photosensor array 130 may be transmitted directly to a peripheral processor for processing.
  • Having described the components of the camera 100 that are essential for the correction of motion blur, the operation of the camera 100 will now be described followed by a description of the camera 100 correcting for motion blur.
  • The camera 100 is typically held by a user and used to generate image data representative of an object, such as the object 110. The image data typically represents a still picture of the object 110, however, the image data may represent a moving picture, e.g., a motion picture, of the object 110. Light 150 reflects from the object 110 and enters the housing 120 of the camera 100 via the aperture 122. The lens 126 then focuses an image of the object 110 onto the two-dimensional photosensor array 130. The two-dimensional photosensor array 130 generates image data representative of the image of the object 110, which is output to the processor 132 for processing. As described above, in the embodiment of the camera 100 described herein, the lens 126 blurs the image focused onto the two-dimensional photosensor array 130 by two photodetecting elements 138, FIG. 2.
  • FIG. 2 illustrates an example of an image 156 of the object 110, FIG. 1, focused onto the two-dimensional photosensor array 130. When the image 156 is properly focused onto the two-dimensional photosensor array 130, an instruction is transmitted to the two-dimensional photosensor array 130, which causes the photodetecting elements 138 to generate image data. More specifically, the photodetecting elements 138 develop electrical charges that are dependent on the amount of light they receive over a preselected period that is reflected from the object 110. When the camera 100, FIG. 1, is used in dark environments, the preselected period is relatively long and when the camera 100 is used in bright environments, the preselected period is relatively short.
  • Referring to FIGS. 2 and 3, the image data may be processed as a plurality of individual color planes. For example, the image data may have a green plane, a red plane, and a blue plane. Alternatively, the green plane may be processed as two green planes because there are twice as many green photodetecting elements 138 as red or blue photodetecting elements 138. The green plane consists of image data generated by photodetecting elements 138 having a green color filter of the color filter array 143 associated with them. Likewise, the red and blue planes consist of image data generated by photodetecting elements 138 having red and blue color filters of the color filter array 143 respectively associated with them. These color planes allow for each color component of the composite image data to be processed individually. For example, individual colors may be scaled to increase or decrease their relative intensity. Likewise, image data from individual photodetecting elements 138 may be scaled accordingly.
  • The color filter array 143 provides a reference for directions used herein, including the directions of motion blur. A y-direction Y extends perpendicular to the x-direction X that was described above. An a-direction A extends diagonally to the x-direction X and the y-direction Y. A b-direction B extends perpendicular to the a-direction A. The aforementioned directions provide non-limiting examples of blur motion and correction as will be described in greater detail below.
  • Referring again to FIG. 1, under ideal circumstances, both the camera 100 and the object 110 remain stationary relative to each other during the above-described preselected period in which the two-dimensional photosensor array 130 generates image data. Ideal circumstances, however, are rarely achieved and the camera 100 generally moves somewhat relative to the object 110 while image data is being generated. For example, a user typically is unable to maintain the camera 100 in a fixed position relative to the object 110 while image data is being generated.
  • Motion blur is typically attributed to either rotational or translational movement of the camera 100 relative to the object 110 occurring as the object 110 is being imaged. The motion blur correction methods described herein are directed toward correcting motion blur caused by translational motion between the object 110 and the camera 100 as the object 100 is being imaged. It should be noted, however, that motion blur caused by rotational motion may also be corrected by the methods described herein. For example, if the center of relative motion between the camera 100 and the object 110 is a significant distance from the image area, the motion blur associated with the image may be substantially similar to motion blur caused by translational motion. One test to determine if motion blur attributed to rotational motion can be corrected as though it is attributed to translational motion is my measuring the blur on the image closest to the center of rotation and furthest from the center of rotation. If the two blurs are substantially equivalent, the motion blur may be corrected as though it is the result of translational motion. In one embodiment of the methods described herein, the blurs at the edges of the image are measured to determine if they are substantially equivalent, meaning that the blur is the result of translational motion.
  • Having summarily described the operation of the camera 100, the operation of the camera 100 will now be described generating image data that is blurred. In the example illustrated herein, the camera 100 has moved in the x-direction X as the image data was being generated. The movement in the x-direction X is an amount that causes the image 156 to move a distance of two and one-half photodetecting elements 138.
  • The movement of the camera 100 relative to the object 110 as the two-dimensional photosensor array 130 generates image data results in the image of the object 110 being blurred. More specifically, when the image data is processed using conventional processing methods, the replicated image of the object 110 will be blurred. An example of this blurring is illustrated in FIG. 2. The image 156 of the object 110, FIG. 1, is shown focused onto the two-dimensional photosensor array 130 at the commencement of the generation of the image data. As the generation of the image data by the photodetecting elements 138 proceeds, the image 156 focused onto the two-dimensional photosensor array 130 moves in the negative x-direction XN. By the time the photodetecting elements 138 have ceased generating image data, the image 156 has moved to a different location on the two-dimensional photosensor array 130. The location of the image of the object 110, FIG. 1, at the termination of the generation of image data is referenced as the second image 158 and is illustrated by dashed lines of FIG. 2. In the example cited herein, the image 156 has moved a distance that is represented by two and one-half columns 142 in the negative x-direction XN during the generation of the image data. The distance that the image 156 moved is proportional to the amplitude of the motion blur. The direction that the image 156 moved is referred to as the direction of the motion blur.
  • As described in greater detail below, image data is analyzed to determine the direction and amplitude of the blur. The amplitude is the amount of movement that occurred during the generation of the image data. The direction of motion blur as described herein is, for illustration purposes only, limited to the x-direction X, the y-direction Y, the a-direction A and the b-direction B. It should be noted that the motion blur detection and correction methods described herein are applicable to a plurality of amplitudes and directions.
  • Referring again to FIG. 1, the image data generated by the two-dimensional photosensor array 130 is transmitted to the processor 132 for processing. The processor 132 correlates the image data generated by each of the photodetecting elements 138, FIG. 2, with its associated color filter per the color filter array 143 of FIG. 3. For processing purposes, the image data may be sorted into the three color groups or planes described above, which are the red plane, the green plane, and the blue plane. Alternatively, the image data may be sorted into four groups if two green color planes are considered.
  • The processor 132 may store a numeric value corresponding to the image data values, or simply image data, generated by each of the photodetecting elements 138. The image data generated by each of the photodetecting elements 138 is proportional to the intensity of light received by each of the photodetecting elements 138. The processor 132 may also store the location from where each value of the image data was generated on the two-dimensional photosensor array 130.
  • Having described the generation of image data, the detection and reduction of motion blur will now be described. The following procedure is outlined in the flowchart of FIGS. 4A and 4B.
  • Image data generated by the two-dimensional photosensor array 130 is representative of the amplitude of light at a plurality of spatial locations within the area of the image. In the situation where the image data is representative of a color image, the image data is representative of the amplitude of light at a plurality of spatial locations for a plurality of color planes. For example, the image data may be represented in three color planes, a red plane, a green plane, and a blue plane. The image data of each color plane may be transformed to a frequency domain by application of a Fourier transform and is sometimes referred to as the “transformed” image data. The transformed image data represents an amplitude and a phase for each of a set of spatial frequencies.
  • The transformed image data offers a perspective on the image data relating to the amplitude and direction of blur within the image. The transformed image data may be manipulated as described below to reduce motion blur. The image data may then be retransformed by way of an inverse Fourier transform to its original format for presentation to a user. The motion blur in the retransformed image data has been reduced by the manipulation.
  • Motion blur reduces the amplitude of the transformed image data, especially in the higher spatial frequencies and always in the direction of the blur motion. When the image data is transformed into the frequency domain using a Fourier transform or the like, the magnitude of the amplitude of sinusoidal signals throughout the set of spatial frequencies in a direction perpendicular to the direction of the motion remains virtually unaffected. The same occurs with regard to directions that are close to perpendicular to the direction of the motion.
  • Having summarized motion blur, its detection and minimization will now be described in greater detail.
  • An example of the spatial affects of motion blur are shown in the graphs of FIGS. 5A through 5C. The graphs 5A through 5C are, for illustration purposes, based on a linear view of the two-dimensional photosensor array 130 rather than a planar view of the two-dimensional photosensor array 130. The horizontal axes P in the graphs 5A through 5C represent spatial locations, and more particularly photodetecting element locations or pixel locations, from where image data was generated. Accordingly, the horizontal axes correspond to the locations of the photodetecting elements 138 of FIG. 2. The vertical axes I(P) represent the intensity of light received at the photodetecting element locations, which are values or magnitudes of the image data generated by the photodetecting elements. The graphs of FIGS. 6A through 6C represent the magnitudes of the amplitudes of the spatial frequencies of the graphs of FIGS. 5A through 5C respectively.
  • The graph of FIG. 5A shows the ideal representation of image data I(P) of a single point of light that would be generated by a two-dimensional photosensor array without a blur filter associated therewith. As shown in FIG. 5A, a single photodetecting element, eight or spatial location eight, generates image data having a magnitude of eight. The graph of FIG. 5B is a representation of image data I(P) of the single point of light of FIG. 5A that would be generated by a two-dimensional photosensor array having a blur filter of two pixels associated therewith. As shown in FIGS. 5A and 5B, the blur filter causes the image data I(P) to have a reduced high spatial frequency content, which is evident by the discrete pulse of the graph of FIG. 5A versus the discrete triangular function of the graph of FIG. 5B. In the graph of FIG. 5B, the magnitude of the spatial location eight drops to an amplitude of four and the magnitudes of spatial locations seven and nine increase from zero to two. It should be noted that the graph of FIG. 5B is considered the “perfect” point of light as imaged by the camera with the two pixels of blur. As the number of photodetecting elements in a digital camera increases, this “perfect” image represented by the graph of FIG. 5B more closely resembles the point of light of FIG. 5A. Accordingly, the negative affects of the blur filter become insignificant.
  • It should be noted that many digital cameras have various types of blur filters, such as a blurry lens, a birefringent filter or lens, or other blur filter device. The blur filter is to prevent objectionable aliased signal artifacts in the image. Aliased signal artifacts are typically associated with large areas of moderately high spatial frequency repetitive patters, such as the weave of a shirt.
  • The graph of FIG. 5C shows the amplitude of image data received by a camera having a blur filter associated therewith and undergoing motion equivalent to two and one-half pixels or photodetecting elements as image data is being generated. As shown in FIG. 5C, the energy from a single point of light (FIG. 5A) is spread over a total of five photodetecting elements due to the blur filter and the motion blur. In the non-limiting example of FIG. 5C, the amplitude of image data generated at the sixth and tenth spatial locations has increased to 0.6. The amplitude of image data generated at the seventh and ninth spatial locations have decreased of two. The amplitude of image data generated at the eighth spatial location has decreased to 2.8. As shown by the graph of FIG. 5C, the image data generated with respect to the spatial locations represents a very wide triangular function. This is indicative of a very low magnitude of high spatial frequency content of the replicated image, meaning that the replicated image is not sharp and is likely blurred.
  • As will be described in greater detail below, motion blur in the replicated image is reduced by increasing the magnitude of the amplitude of the sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in the direction of the motion blur. It should be noted that the sinusoidal signals are used herein for illustration purposes and that they are derived by way of a Fourier transform. It should also be noted that the inventive concepts described herein are applicable to other signals derived from other transform functions.
  • Different sharpening kernels may be applied to the image data in the direction of the motion blur to increase the magnitude of the amplitude of the high spatial frequency content. In one non-limiting example, a Weiner correction algorithm is applied to the frequency domain transformed image data in the direction of the motion blur to increase the magnitude of the amplitude of a specific set of spatial frequency content. It should be noted that other correction algorithms may be applied to the image data to reduce motion blur.
  • Having summarily described the effects of motion blur and a method of minimizing the effects of motion blur, the detection of motion blur will now be described.
  • As summarily described above, motion blur due to translational motion, unlike other blurs, only occurs in a single direction. For example, with reference to the motion blur illustrated by the images 156 and 158 of FIG. 2, the motion blur only occurs in the x-direction X. Other blurs, such as optical blur, typically occur in all directions. Accordingly, the magnitude of the amplitude of the high spatial frequency content of an image affected by motion blur will be low in the direction corresponding to the direction of the motion. The magnitude of the amplitude of the high spatial frequency content corresponding to directions perpendicular or substantially perpendicular to the direction of motion will be less. When applied in practice, the magnitude of the amplitude of the high spatial frequency content in the direction of the motion blur will be significantly lower than the magnitude of the amplitude of the same specific high spatial frequency content in a perpendicular direction.
  • The detection of motion blur may be accomplished by analyzing image data generated by photodetecting elements 138 located in a portion of the two-dimensional photosensor array 130. A user of a camera typically assures that the subject of the photograph is focused and substantially centered in the photograph. Accordingly, the center of the photograph is where the sharpest focused image is located. For this reason, in the examples described herein, image data generated by a central portion of the two-dimensional photosensor array 130 will be analyzed for motion blur. For example, image data generated by the middle one-ninth of the two-dimensional photosensor array 130 may be analyzed for motion blur. It should be understood, however, that any portion of the two-dimensional photosensor array 130 may be analyzed for motion blur.
  • For illustration purposes, an example of analyzing image data to determine the direction and magnitude of motion blur is provided. The analysis commences analyzing image data generated by the photodetecting elements 138 located in the vicinity of a corner 160 of the image 156. FIG. 7 is an exploded view of the photodetecting elements 138 located in the corner 160 of the two-dimensional photosensor array 130. Image data generated by a row 164 of photodetecting elements 138 extending in the x-direction X will be analyzed in the following example. Likewise, image data generated by a column 166 of photodetecting elements 138 extending in the y-direction Y will also be analyzed in the following example. The photodetecting elements 138 in the row 164 are referenced as X(1), Y(1) through X(10), Y(1) and the photodetecting elements 138 in the column 166 are referenced as X(1), Y(1) through X(1), Y(10). It should be noted that for illustration purposes, only ten photodetecting elements 138 extending in the x-direction X and the y-direction Y are being analyzed. It is to be understood, however, that any number of photodetecting elements 138 in the x-direction X and the y-direction Y may be analyzed. It is also to be further understood that any number of photodetecting elements 138 extending in the a-direction A and the b-direction B may also be analyzed to detect motion blur in other directions.
  • The first step in determining whether motion blur exists in the image data is to analyze the magnitude of the amplitude of the high spatial frequency content in orthogonal directions over an area of an image. The magnitude of the amplitude of the high spatial frequency content in the x-direction X and the y-direction Y is proportional to the figure of merit in the x-direction X and the y-direction Y respectively. The figure of merit provides a basis for determining the degree to which light/dark transitions occur in a specific direction in an image. A non-limiting example of a figure of merit in the x-direction X is referred to herein as Fx and is calculated as follows: F X = m = 1 m = q n = 1 n = p X ( n , m ) - X ( n - 1 , m )
      • wherein p is the number of photodetecting elements 138 in the x-direction X that are to be analyzed, q is the number of rows that are analyzed in the x-direction X, and X(n,m) is the value of the image data generated by the photodetecting element 138 at that location. Accordingly, n designates the column number of the photodetector array and m designates the row number. It should be noted that in the non-limiting example described herein, the figure of merit FX may calculated in the center of the image where the sharpest focus occurs. It should also be noted that the figures of merit are calculated in a single color plane and not by adjacent photodetecting elements for the example of the digital camera described herein. This assures that the figure of merit measures motion blur and not color transitions in the image. Furthermore, by not analyzing adjacent photodetecting elements, the anti-aliasing blur filter will have little, if any, influence on the figure of merit calculation. It should be noted that virtually any increment of pixel values may be used to calculate the figure of merit. A preferred embodiment will use photodetecting elements that are close together within one color plane, because this will emphasize the higher spatial frequencies. The higher spatial frequencies are the most sensitive measures of motion blur. It should also be noted that the figures of merit may be calculated in any direction to measure motion blur in any direction.
  • As with the figure of merit in the x-direction X, the figure of merit in the y-direction Y is referred to as FY and is calculated as follows: F Y = n = 1 n = p m = 1 m = q Y ( n , m ) - Y ( n , m - 1 )
      • wherein q is the number of photodetecting elements 138 in the y-direction Y that are to be analyzed, p is the number of columns in the y-direction Y, and Y(n,m) is the value of the image data generated by the photodetecting element 138 at that location. It should be noted that in the non-limiting example described herein, the figure of merit FY is calculated along the column 166. In one preferred embodiment, p is equal to q so that the figures of merit are calculated from a square section of the image.
  • As described above, the figures of merits described herein are non-limiting examples of figures of merit. Other variations of the figure of merit can use weighted multiple color planes. For example, the figures of merit may be calculated in all the color planes. The values from the different color planes can be weighted and summed togther. For example, each green plane may be weighted thirty percent, the red plane may be weighted thirty percent and the blue plane may be weighted ten percent. Likewise, the spatial frequencies may be weighted. It is also possible to combine weighted color planes with weighted spatial frequencies. In another variation, higher order numerical methods may be used to generate the slope estimates. These slope estimate may be used rather than the simple difference calculations described above.
  • The figures of merit provide indications of the magnitudes of the amplitudes of spatial frequencies of the image in the specified directions. Referring briefly to FIG. 5A, the graph shows a steep transition between adjacent photodetecting elements for image data generated without motion blur and without a blur filter. More precisely, the single point of light has caused image data to be generated solely at the eighth spatial position. Accordingly, the figure of merit associated with the graph of FIG. 5A is high. The graph of FIG. 5B shows transitions between image data values that are not as steep as the graph of FIG. 5A. Accordingly, the figure of merit associated with the graph of FIG. 5B is less that the figure of merit associated with the graph of FIG. 5A. The graph of FIG. 5C shows image data generated in the presence of motion blur. The motion blur decreases the magnitudes of the transitions between image data values of adjacent photodetecting elements 138, FIG. 2. Accordingly, the figure of merit of the image data associated with the graph of FIG. 5C is less than the figures of merit associated with the graphs of FIG. 5A and FIG. 5B. It should also be noted that the figure of merit as described herein is proportional to the magnitude of the slope of the graphs. By using the figure of merit equations described above, the figures of merit for the graphs of FIGS. 5A is sixteen, the figure of merit of the graph of FIG. 5B is eight, and the figure of merit of the graph of FIG. 5C is 5.6.
  • As described above, motion blur due to translational motion, unlike other blurs, occurs only in one direction. Accordingly, if the figure of merit in the x-direction X, FX, differs from the figure of merit in the y-direction Y, FY, motion blur likely occurred during the generation of the image data. This difference in the figures of merit may be calculated by taking the ratio of FX to FY or FY to FX and comparing the ratio to a preselected value. As a non-limiting example, if the ratio is greater than 1.4, it can be assumed that motion blur occurred. It follows that the direction of the motion blur will be in the direction having the lower figure of merit because the magnitudes of the transitions in this direction are lower. It also follows that the value of the ratio determines the amount of motion blur. For example, a greater value of the ratio means that the image was blurred over a greater number of photodetecting elements 138, FIG. 2. The value of the ratio is used to determine the amount to which motion blur is to be minimized. It should be noted that the non-limiting methods described herein for determining the direction of motion blur determine the direction of motion blur closest to a preselected direction. For example, the methods described herein may determine the direction of motion blur that is closest to the x-direction X, the y-direction Y, the a-direction A or the b-direction B. Similar methods can be used to expand the possible number of directions in which motion blur may be analyzed.
  • When a determination has been made that the image has been blurred or, more specifically, that the replicated image is blurred, the image data is processed to minimize the blur of the replicated image. Minimizing motion blur involves increasing the magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image data in the direction of the blur. A non-limiting example may be achieved by amplifying image data values corresponding to higher spatial frequencies. For example, in the case where the motion blur occurs in the x-direction X, the image data generated by individual rows 140, FIG. 2, of photodetecting elements 138 are processed to increase the magnitude of the amplitude of spatial frequency. Other scaling and processing techniques as described below may also be applied to the image data.
  • It should be noted that the effects of motion blur occur to all the color planes equally. Therefore, image data from one color plane may be analyzed to determine the direction and amount of motion blur. The magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image of the color planes may be increased equally in order to minimize the affects of motion blur on the whole composite image. It should be further noted that processing the image data to minimize the motion blur may be performed during the process of demosaicing the image data as disclosed in the United States patent application of Taubman, previously referenced. This may, as an example, involve scaling demosaicing coefficients in the direction of the motion blur to increase the magnitude of the amplitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in the direction of the motion blur. Other techniques can be used to reduce motion blur, such as directional sharpening kernels, windowed directional sharpening kernels, and directional deconvolution correction.
  • The procedure described above determines the amount and direction of motion blur. It should be noted that the procedure described above is applicable for detecting and minimizing motion blur along any orthogonal directions. For illustration purposes, an example of detecting and minimizing motion blur in additional directions diagonal to the x-direction X and the y-direction Y is provided. The diagonal directions are referenced in FIG. 3 as the a-direction A and the b-direction B, which are perpendicular to each other. In this situation, the figures of merit are calculated in the a-direction A and the b-direction B along with the ratio of the figures of merit as described above with reference to the x-direction X and the y-direction Y. At this point, two ratios exist, one for figures of merit in the x and y-directions X and Y and one for figures of merit in the a and b-directions A and B. The ratio having the greatest value indicates the orthogonal directions associated with the motion blur. The actual direction of the motion blur is determined by the lowest figure of merit associated with the ratio having the greatest value. Motion blur in this direction is minimized in the manner described herein by increasing the magnitude of the amplitude of the magnitude of sinusoidal signals at specific spatial frequencies of the frequency domain transformed image in this direction.
  • Having described the method for determining whether motion blur is present in an image and a method of minimizing the motion blur, a detailed example of reducing motion blur is now provided.
  • The following example is based on the imaged point of light of FIGS. 5A through 5C. As described above, the point of light without the use of a blur filter appears solely at the eighth spatial position and generates image data having a magnitude of eight. The point of light imaged by a camera having a two-pixel blur filter and undergoing motion blur of two and one half pixels generates image data as described above with reference to the graph of FIG. 5C. The values of spatial frequency of the graph of FIG. 5C are significantly less that the values of spatial frequency of the graph of FIG. 5A. For reference purposes, the values of the spatial frequencies of the graphs of FIGS. 5A through 5C are shown in the bar graphs of FIGS. 6A through 6C. It should be noted that the graphs of FIGS. 6A through 6C do not include phase information. As shown by the graph of FIG. 6A, the single point of light of FIG. 5A has a normalized amplitude of one across the frequency spectrum, similar to an impulse function. The graph of FIG. 7B shows the decreased spatial frequency values due to the blur filter added to the camera. FIG. 7C shows that the values of spatial frequency are even more decreased by motion blur. The values of the spatial frequencies due to the blur filter and motion blur are detailed in Table 1. It should be noted that in the spatial frequency eight is the highest and spatial frequencies zero and fifteen are the lowest.
    TABLE 1
    Blur Filter
    Spatial Ideal Amplitude Motion Blur Weiner Weiner
    Fre- Point of (perfect Amplitude Correction Corrected
    quency light image) w/Blur Filter Factor Amplitude
    0 1 1 1 1 1
    1 1 .96 .92 1.04 .957
    2 1 .85 .7 1.172 .82
    3 1 .69 .44 1.418 .623
    4 1 .5 .2 1.9 .38
    5 1 .31 .05 1.944 .097
    6 1 .15 .004 1.366 .005
    7 1 .04 .006 .578 .003
    8 1 0 0 0 0
    9 1 .04 .006 .578 .003
    10 1 .15 .004 1.366 .005
    11 1 .31 .05 1.944 .097
    12 1 .5 .2 1.9 .38
    13 1 .69 .44 1.418 .623
    14 1 .85 .7 1.172 .82
    15 1 .96 .92 1.04 .957
  • As shown by the data of Table 1, motion blur attenuated the high spatial frequency content of the transformed image. More specifically, the values of the high spatial frequencies of five, six, seven, and eight are attenuated. Motion blur did not attenuate the values of the lower spatial frequencies.
  • Various image sharpening kernels can be applied to the image data in the direction of the motion blur in order to increase the magnitude of the amplitude of the high spatial frequency content. The result of the increased amplitude of the high spatial frequency content is a sharper image, which is a reduction of the blur caused by motion. More specifically, the transitions between light and dark areas are accentuated.
  • Table 1 shows the Weiner correction factor that is applied to the signal at the specific spatial frequencies in the direction of the motion blur so as to reduce motion blur. It should be noted that the Weiner correction factor used herein has a minimal impact on the high spatial frequency content. As shown in Table 1, the frequencies seven and eight are minimally impacted by the Weiner correction factor applied herein. This is because the very high spatial frequency content of the image data tends to include a lot of noise. Thus, if the very high spatial frequency content is amplified, the noise will likely be amplified more than the signal.
  • The resulting image after the application of the Weiner correction factor is shown in FIG. 8, which is a graph of the point of light of FIG. 5C imaged under a motion blur condition, but with the Weiner correction factor applied. The original motion blurred image of FIG. 5C is shown in dashed lines in FIG. 8 for reference purposes. The values of the blurred image of FIG. 5C are compared to the values of the corrected image of FIG. 8 in Table 2 under the portion titled Weiner Corrected Image. It should be noted that the Wiener corrected image factor corrected the image for two pixels or photodetecting elements 138, FIG. 2, of motion blur. The Weiner correction algorithm may be modified to correct for any number of pixels of motion blur.
    TABLE 2
    Spa-
    tial Point Motion Weiner −1, 3, −1 −0.5, 2, −0.5
    Loca- of Blur Blurred Corrected Sharpening Sharpening
    tion Light Filter Image Image Kernel Kernel
    1 0 0 0 0.03 0 0
    2 0 0 0 0 0 0
    3 0 0 0 −0.04 0 0
    4 0 0 0 0.02 0 0
    5 0 0 0 0.06 0 0
    6 0 0 0 −0.20 −0.60 −0.30
    7 0 0 0.60 0.28 −0.20 0.20
    8 0 2 2.00 2.17 2.60 2.30
    9 8 4 2.80 3.37 4.40 3.60
    10 0 2 2.00 2.17 2.60 2.30
    11 0 0 0.60 0.28 −.020 0.20
    12 0 0 0 −0.20 −0.60 −0.30
    13 0 0 0 0.60 0 0
    14 0 0 0 0.02 0 0
    15 0 0 0 −0.04 0 0
  • As illustrated by Table 2, the Weiner corrected image restores the image to a close approximation to the image represented by the blur filter column. It should be noted that the value of the highest amplitude at the spatial location nine ideally would be restored to a value of four, however, the Weiner correction factor as described herein restored it to a value of 3.37 from its motion blurred value of 2.8. It should also be noted that the Weiner correction factor illustrated herein has very few negative numbers, which have very small magnitudes. High magnitude negative numbers tend to increase the Gibbs effect, which is detrimental to image quality. The −1, 3, −1 sharpening kernel did increase the value of the highest spatial frequency. However, the value of 4.4 is much greater than the ideal value of 4.0, which increases the noise and is actually detrimental to the image quality.
  • As described above, different sharpening algorithms can be applied to the image data in the direction of the motion blur in order to reduce the motion blur. In Table 2, two alternative examples of sharpening algorithms or kernels are provided. More specifically, the results of a −1, 3, −1 and a −0.5, 2, −0.5 sharpening kernels are shown in Table 2.
  • Having described an embodiment of the motion blur detection and minimization methods, other embodiments will now be described.
  • The method described herein corrects for two pixels of motion blur. It is to be understood, however, that the method may be modified to correct for different amounts of motion blur. In one embodiment, a determination is made as to the amount of motion blur in the image. The image is then ‘deblurred’ an amount based on the amount of blur in the image. For example, the method may be able to correct for two or four pixels of motion blur. If the image is determined to have greater than one and less than three pixels of motion blur, the image may be corrected for two pixels of motion blur. If the image is determined to have three to five pixels of motion blur, the image is corrected for four pixels of motion blur.
  • The detection of motion blur has been described herein as being accomplished by analyzing image data representative of an object. Motion blur may also be detected by using a sensor or plurality of sensors mounted to the camera 100, FIG. 1, that detect the amount and direction of movement of the camera during the period in which image data is being generated. The direction and amount of motion blur then establish the coefficients for the image sharpening algorithms to reduce the motion blur.
  • Referring to FIG. 1, the detection and minimization of motion blur has been described herein as being performed by the processor 132 located within the camera 100. In another embodiment, the processor 132 may facilitate the transfer of image data to a peripheral processor that may detect and minimize motion blur. For example, a blurred photograph may be captured or otherwise scanned so as to generate image data representative of the object.
  • Referring again to FIG. 6, the figures of merit have been calculated based on image data that is generated by photodetecting elements 138, FIG. 2, that are located next to each other or next to each other in a given color plane. One embodiment of the above-described method bases the calculation of the figures of merit on image data that is generated by photodetecting elements 138 that are spaced a distance from each other. This will measure the figure of merit at a different spatial frequency. For example, the figures of merit may be calculated based on image data generated by every other or every third photodetecting element 138. This embodiment may be used in processing information from digital cameras having a great number of photodetecting elements 138. By not processing the image data generated by all the photodetecting elements 138, the time required to determine motion blur is decreased. Several figures of merit at several spatial frequencies can be combined using weighting factors to make the overall figure of merit calculation more immune to image or scene content. Additionally, several color planes may be weighted in a similar manner. Likewise, a combination of color planes and spatial frequencies may be weighted.
  • In another embodiment, small portions of the image are analyzed to determine if motion blur exists in a small portion of the image. The motion blur is then minimized in that small section. For example, the camera may be held stationary relative to a background as image data is being generated. An object being imaged, however, may be moving relative to the camera and the background as image data is being generated. The method described above, will reduce the motion blur of the image of the object without creating blur in the background.
  • The above described method of minimizing motion blur becomes more useful as the density of photodetecting elements located on a two-dimensional photosensor array increases. The purpose of having an increased density of photodetecting elements is to provide a more defined or sharper image. The susceptibility to motion blur, however, increases as the density of photodetecting elements increases. During the period that the photodetecting elements are generating image data, the image of the object is more likely to shift between photodetecting elements when the density of photodetecting elements is high. This shifting will be detected as motion blur and will be minimized accordingly.
  • The above described method of minimizing motion blur also becomes more useful in generating image data in relatively dark environments. The period required by the photodetecting elements to generate image data is proportional to the amount of light reflecting from the object being imaged. Accordingly, when imaging is performed in a relatively dark environment, the period required to generate the image data increases. This increased period increases the likelihood that the user of the camera will move the camera while image data is being generated, which will cause motion blur. By implementing the above-described procedure, the motion blur can be minimized.
  • Other embodiments for determining and minimizing motion blur may be employed using the methods described herein. In one embodiment, several color planes are analyzed to determine the motion blur. These include the green, red, and blue color planes. In another embodiment, more than four directions are analyzed for the presence of motion blur. For example, eight directions may be analyzed, which offers a more precise determination of the direction of motion blur. In yet another embodiment, the image data is analyzed to determine if different amounts of motion blur are present. For example, the image data may be analyzed to determine if no motion blur exists, two pixel motion blur exists, four pixel motion blur exists and so on.
  • While an illustrative and presently preferred embodiment of the invention has been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed and that the appended claims are intended to be construed to include such variations except insofar as limited by the prior art.

Claims (20)

1. A method for reducing motion blur in a digital image, said method comprising increasing the magnitude of the amplitude of the signals in a preselected set of spatial frequencies of the image in the direction of the motion blur.
2. The method of claim 1, wherein said signals are sinusoidal signals and wherein said spatial frequencies are derived via a Fourier transform.
3. The method of claim 1 and further comprising determining the direction of motion blur.
4. The method of claim 3, wherein said determining the direction of motion blur comprises:
providing image data representative of at least a portion of said digital image;
analyzing said image data to calculate a first figure of merit of said digital image in a first direction;
analyzing said image data to calculate a second figure of merit of said digital image in a second direction, said first and said second directions being substantially orthogonal;
calculating a first ratio of said first figure of merit to said second figure of merit, said ratio being the greater of said first or second figure of merit divided by the lesser of said first or second figure of merit; and
comparing said first ratio to a preselected value, wherein motion blur exists in said digital image if said first ratio is greater than said preselected value.
5. The method of claim 3, wherein said determining the direction of motion blur comprises:
providing image data representative of at least a portion of said digital image;
analyzing said image data to calculate a plurality of first figures of merit of said digital image in a plurality of directions;
analyzing said image data to calculate a plurality of second figures of merit of said digital image, wherein each of said second figures of merit is in a direction substantially orthogonal to a corresponding first figure of merit;
calculating a plurality of ratios of said first figures of merit to their corresponding second figures of merit, each of said ratios being the greater of a first or second figure of merit divided by the lesser of its corresponding first or second figure of merit; and
comparing said ratios to a preselected value, wherein motion blur exists in said digital image if one of said ratios is greater than said preselected value.
6. An apparatus for reducing motion blur in an image, said apparatus comprising a computer and a computer-readable medium operatively associated with said computer, said computer-readable medium containing instructions for controlling said computer to reduce motion blur in an image by:
determining the direction of motion blur by analyzing image data representative of at least a portion of said image; and
increasing the magnitude of the amplitude of spatial frequency in the direction of said motion blur.
7. The apparatus of claim 6, wherein said increasing the magnitude of the amplitude of spatial frequency comprises increasing the magnitude of the amplitude of the signals in a preselected set of spatial frequencies of the image in the direction of the motion blur.
8. The apparatus of claim 6, wherein said increasing the magnitude of the amplitude of spatial frequency comprises increasing the magnitude of the amplitude of the sinusoidal signals in a preselected set of spatial frequencies of the image in the direction of the motion blur, wherein said spatial frequencies are derived by way of a Fourier transform.
9. The apparatus of claim 8, wherein said image data is transformed back to the spatial domain by an inverse Fourier transform.
10. The apparatus of claim 6, wherein said determining the direction of motion blur comprises:
analyzing said image data to calculate a first figure of merit of said digital image in a first direction;
analyzing said image data to calculate a second figure of merit of said digital image in a second direction, said first and said second directions being substantially orthogonal;
calculating a first ratio of said first figure of merit to said second figure of merit, said ratio being the greater of said first or second figure of merit divided by the lesser of said first or second figure of merit; and
comparing said first ratio to a preselected value, wherein motion blur exists in said digital image if said first ratio is greater than said preselected value.
11. The apparatus of claim 6, wherein said determining the direction of motion blur comprises:
analyzing said image data to calculate a plurality of first figures of merit of said digital image in a plurality of directions;
analyzing said image data to calculate a plurality of second figures of merit of said digital image, wherein each of said second figures of merit is in a direction substantially orthogonal to a corresponding first figure of merit;
calculating a plurality of ratios of said first figures of merit to their corresponding second figures of merit, each of said ratios being the greater of a first or second figure of merit divided by the lesser of its corresponding first or second figure of merit; and
comparing said ratios to a preselected value, wherein motion blur exists in said digital image if one of said ratios is greater than said preselected value.
12. A method for reducing motion blur in an image, said method comprising:
providing image data representative of at least a portion of said image;
analyzing said image data to detect the presence of motion blur in said image;
analyzing said image data to detect the direction of motion blur in said digital image;
processing said image data to increase edge acuity said image in said direction of said motion blur.
13. The method of claim 12, wherein said analyzing said image data to detect the presence of motion blur comprises:
analyzing said image data to calculate a first figure of merit of said digital image in a first direction;
analyzing said image data to calculate a second figure of merit of said digital image in a second direction, said first and said second directions being substantially orthogonal;
calculating a first ratio of said first figure of merit to said second figure of merit, said ratio being the greater of said first or said second figure of merit divided by the lesser of said first or said second figure of merit; and
comparing said first ratio to a preselected value, wherein motion blur exists in said digital image if said first ratio is greater than said preselected value.
14. The method of claim 13, wherein said analyzing said image data to detect the direction of motion blur comprises determining the lowest value of said first and said second figures of merit, said lowest value corresponding to said direction of motion blur.
15. The method of claim 12, wherein said analyzing said image data to detect the presence of motion blur comprises:
analyzing said image data to calculate a plurality of first figures of merit of said digital image in a plurality of directions;
analyzing said image data to calculate a plurality of second figures of merit of said digital image, wherein each of said second figures of merit is in a direction substantially orthogonal to a corresponding first figure of merit;
calculating a plurality of ratios of said first figures of merit to their corresponding second figures of merit, each of said ratios being the greater of a first or second figure of merit divided by the lesser of its corresponding first or second figure of merit; and
comparing said ratios to a preselected value, wherein motion blur exists in said digital image if one of said ratios is greater than said preselected value.
16. The method of claim 15, wherein said analyzing said image data to detect the direction of motion blur comprises determining which of said ratios has the highest value and determining the lowest figure of merit of said highest valued ratio, said lowest figure of merit corresponding to said direction of motion blur.
17. The method of claim 12, wherein said processing comprises increasing the magnitude of the amplitude of the signals in a preselected set of spatial frequencies of the transformed image data in the direction of the motion blur.
18. The method of claim 12, wherein said processing comprises increasing the magnitude of the amplitude of the sinusoidal signals in a preselected set of spatial frequencies of the transformed image data in the direction of the motion blur, wherein said image data is transformed by a Fourier transform.
19. The method of claim 18, and further comprising transforming said image data back to the spatial domain by an inverse Fourier transform.
20. The method of claim 12, wherein said processing comprises increasing the amplitude of signals of said image based on the detection and amplitude of motion blur.
US11/122,906 2001-05-29 2005-05-05 Method for reducing motion blur in a digital image Abandoned US20050243178A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/122,906 US20050243178A1 (en) 2001-05-29 2005-05-05 Method for reducing motion blur in a digital image

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/867,352 US6987530B2 (en) 2001-05-29 2001-05-29 Method for reducing motion blur in a digital image
US11/122,906 US20050243178A1 (en) 2001-05-29 2005-05-05 Method for reducing motion blur in a digital image

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/867,352 Continuation US6987530B2 (en) 2001-05-29 2001-05-29 Method for reducing motion blur in a digital image

Publications (1)

Publication Number Publication Date
US20050243178A1 true US20050243178A1 (en) 2005-11-03

Family

ID=25349623

Family Applications (2)

Application Number Title Priority Date Filing Date
US09/867,352 Expired - Fee Related US6987530B2 (en) 2001-05-29 2001-05-29 Method for reducing motion blur in a digital image
US11/122,906 Abandoned US20050243178A1 (en) 2001-05-29 2005-05-05 Method for reducing motion blur in a digital image

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09/867,352 Expired - Fee Related US6987530B2 (en) 2001-05-29 2001-05-29 Method for reducing motion blur in a digital image

Country Status (2)

Country Link
US (2) US6987530B2 (en)
JP (1) JP4121780B2 (en)

Cited By (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030164882A1 (en) * 2001-03-15 2003-09-04 Tetsujiro Kondo Image processing device and method, and imaging device
US20040001147A1 (en) * 2002-06-19 2004-01-01 Stmicroelectronics S.R.L. Method of stabilizing an image sequence
US20040027454A1 (en) * 2002-06-19 2004-02-12 Stmicroelectronics S.R.I. Motion estimation method and stabilization method for an image sequence
US20080238820A1 (en) * 2007-03-29 2008-10-02 Otsuka Electronics Co., Ltd Motion picture image processing system and motion picture image processing method
US20090015719A1 (en) * 2007-07-11 2009-01-15 Banner Ron Reducing motion blur from an image
US20090174782A1 (en) * 2008-01-07 2009-07-09 Philippe Kahn Method and Apparatus for Improving Photo Image Quality
WO2009110868A1 (en) * 2008-03-06 2009-09-11 Nikon Corporation Method for estimating of direction of motion blur in an image
US7623726B1 (en) 2005-11-30 2009-11-24 Adobe Systems, Incorporated Method and apparatus for using a virtual camera to dynamically refocus a digital image
US20100140461A1 (en) * 2006-12-13 2010-06-10 Georgia Tech Research Corporation Systems and methods for real time multispectral imaging
US20100231732A1 (en) * 2009-03-11 2010-09-16 Zoran Corporation Estimation of point spread functions from motion-blurred images
US20100231731A1 (en) * 2007-08-03 2010-09-16 Hideto Motomura Image-capturing apparatus, image-capturing method and program
US20110169977A1 (en) * 2010-01-12 2011-07-14 Nec Casio Mobile Communications, Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US7982770B1 (en) 2006-09-08 2011-07-19 Dp Technologies, Inc. Method and apparatus to provide improved image quality in a camera
US8064759B1 (en) 2009-04-15 2011-11-22 Dp Technologies, Inc. Method and apparatus for motion-state based image acquisition
US8285344B2 (en) 2008-05-21 2012-10-09 DP Technlogies, Inc. Method and apparatus for adjusting audio for a user environment
US8471920B2 (en) 2009-07-15 2013-06-25 Adobe Systems Incorporated Focused plenoptic camera employing different apertures or filtering at different microlenses
US8555282B1 (en) 2007-07-27 2013-10-08 Dp Technologies, Inc. Optimizing preemptive operating system with motion sensing
US8559756B2 (en) 2007-08-06 2013-10-15 Adobe Systems Incorporated Radiance processing by demultiplexing in the frequency domain
US8611693B2 (en) 2008-05-30 2013-12-17 Adobe Systems Incorporated Managing artifacts in frequency domain processing of light-field images
US8620353B1 (en) 2007-01-26 2013-12-31 Dp Technologies, Inc. Automatic sharing and publication of multimedia from a mobile device
US8665341B2 (en) 2010-08-27 2014-03-04 Adobe Systems Incorporated Methods and apparatus for rendering output images with simulated artistic effects from focused plenoptic camera data
US8724000B2 (en) * 2010-08-27 2014-05-13 Adobe Systems Incorporated Methods and apparatus for super-resolution in integral photography
US8749694B2 (en) 2010-08-27 2014-06-10 Adobe Systems Incorporated Methods and apparatus for rendering focused plenoptic camera data using super-resolved demosaicing
US8803918B2 (en) 2010-08-27 2014-08-12 Adobe Systems Incorporated Methods and apparatus for calibrating focused plenoptic camera data
US8817015B2 (en) 2010-03-03 2014-08-26 Adobe Systems Incorporated Methods, apparatus, and computer-readable storage media for depth-based rendering of focused plenoptic camera data
US8872646B2 (en) 2008-10-08 2014-10-28 Dp Technologies, Inc. Method and system for waking up a device due to motion
US8902154B1 (en) 2006-07-11 2014-12-02 Dp Technologies, Inc. Method and apparatus for utilizing motion user interface
US8949070B1 (en) 2007-02-08 2015-02-03 Dp Technologies, Inc. Human activity monitoring device with activity identification
US8996332B2 (en) 2008-06-24 2015-03-31 Dp Technologies, Inc. Program setting adjustments based on activity identification
US9030550B2 (en) 2011-03-25 2015-05-12 Adobe Systems Incorporated Thin plenoptic cameras using solid immersion lenses
US9092875B2 (en) 2011-04-12 2015-07-28 Panasonic Intellectual Property Management Co., Ltd. Motion estimation apparatus, depth estimation apparatus, and motion estimation method
US9232137B2 (en) 2013-04-24 2016-01-05 Microsoft Technology Licensing, Llc Motion blur avoidance
US9316840B2 (en) 2009-01-20 2016-04-19 Adobe Systems Incorporated Methods and apparatus for reducing plenoptic camera artifacts
US9390229B1 (en) 2006-04-26 2016-07-12 Dp Technologies, Inc. Method and apparatus for a health phone
US9529437B2 (en) 2009-05-26 2016-12-27 Dp Technologies, Inc. Method and apparatus for a motion state aware device

Families Citing this family (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9143706B2 (en) * 2001-06-06 2015-09-22 Andrew Zador Imaging system utilizing spatial image oscillation
US7813559B2 (en) * 2001-11-13 2010-10-12 Cyberoptics Corporation Image analysis for pick and place machines with in situ component placement inspection
US7239399B2 (en) * 2001-11-13 2007-07-03 Cyberoptics Corporation Pick and place machine with component placement inspection
JP4365315B2 (en) * 2002-05-23 2009-11-18 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Motion blur reduction based on edges
US7397500B2 (en) * 2003-04-30 2008-07-08 Hewlett-Packard Development Company, L.P. Camera shake warning and feedback system that teaches the photographer
US7379091B2 (en) 2003-04-30 2008-05-27 Hewlett-Packard Development Company, L.P. Method and apparatus for computing an image stability measure
JP4286068B2 (en) * 2003-06-03 2009-06-24 大塚電子株式会社 Screen quality evaluation method
KR101032233B1 (en) * 2003-06-23 2011-05-02 소니 주식회사 Image processing method and device, and program
US7266248B2 (en) * 2003-08-08 2007-09-04 Hewlett-Packard Development Company, L.P. Method and apparatus for generating data representative of an image
US7020579B1 (en) * 2003-09-18 2006-03-28 Sun Microsystems, Inc. Method and apparatus for detecting motion-induced artifacts in video displays
JP4515208B2 (en) * 2003-09-25 2010-07-28 富士フイルム株式会社 Image processing method, apparatus, and program
US7706595B2 (en) * 2003-11-07 2010-04-27 Cyberoptics Corporation Pick and place machine with workpiece motion inspection
DE50309706D1 (en) * 2003-12-19 2008-06-05 Imaging Solutions Ag Method of processing digital image data from blurred photographs
US7636489B2 (en) * 2004-04-16 2009-12-22 Apple Inc. Blur computation algorithm
US7561186B2 (en) * 2004-04-19 2009-07-14 Seiko Epson Corporation Motion blur correction
JP2006024193A (en) * 2004-06-07 2006-01-26 Fuji Photo Film Co Ltd Image correction device, image correction program, image correction method and image correction system
US8532338B2 (en) * 2004-07-06 2013-09-10 Hewlett-Packard Development Company, L.P. System and method for compensating for motion blur in optical navigation
US20060075631A1 (en) * 2004-10-05 2006-04-13 Case Steven K Pick and place machine with improved component pick up inspection
CN100504981C (en) * 2004-10-13 2009-06-24 皇家飞利浦电子股份有限公司 Method and apparatus for reducing blur of moving images
WO2006068292A1 (en) * 2004-12-21 2006-06-29 Sony Corporation Image processing device, image processing method and image processing program
EP1830562A4 (en) * 2004-12-21 2012-08-01 Sony Corp Learning device, learning method, and learning program
US20070003126A1 (en) * 2005-05-19 2007-01-04 Case Steven K Method and apparatus for evaluating a component pick action in an electronics assembly machine
US7755678B2 (en) * 2005-10-28 2010-07-13 Hewlett-Packard Development Company, L.P. Programmable anti-aliasing systems and methods for cameras
WO2007053557A1 (en) * 2005-10-31 2007-05-10 Cyberoptics Corporation Electronics assembly machine with embedded solder paste inspection
JP4484806B2 (en) * 2005-11-30 2010-06-16 キヤノン株式会社 RECORDING SYSTEM, RECORDING METHOD, HOST DEVICE, PROGRAM, AND COMPUTER-READABLE MEDIUM
US7680354B2 (en) * 2006-03-22 2010-03-16 Arcsoft, Inc. Image deblur based on two images
US8073280B2 (en) * 2006-05-16 2011-12-06 Panasonic Corporation Image processing apparatus and image processing program for correcting image degradation
JP4872508B2 (en) * 2006-07-28 2012-02-08 ソニー株式会社 Image processing apparatus, image processing method, and program
US7602418B2 (en) * 2006-10-11 2009-10-13 Eastman Kodak Company Digital image with reduced object motion blur
TW200840365A (en) * 2007-03-23 2008-10-01 Ind Tech Res Inst Motion-blur degraded image restoration method
KR100860967B1 (en) * 2007-04-16 2008-09-30 삼성전자주식회사 Apparatus and method for removing motion blur of image
US7899229B2 (en) * 2007-08-06 2011-03-01 Hui Luo Method for detecting anatomical motion blur in diagnostic images
US20090060387A1 (en) * 2007-09-04 2009-03-05 Microsoft Corporation Optimizations for radius optical blur
US8144780B2 (en) * 2007-09-24 2012-03-27 Microsoft Corporation Detecting visual gestural patterns
US8494306B2 (en) * 2007-12-13 2013-07-23 Samsung Electronics Co., Ltd. Method and an apparatus for creating a combined image
WO2009142783A1 (en) * 2008-05-21 2009-11-26 Nikon Corporation System and method for estimating a direction of motion blur in an image
JP4618355B2 (en) 2008-09-25 2011-01-26 ソニー株式会社 Image processing apparatus and image processing method
FR2947075A1 (en) * 2009-06-19 2010-12-24 St Microelectronics Rousset RESISTIVE EVALUATION OF THE COUPLING FACTOR OF AN ELECTROMAGNETIC TRANSPONDER
KR20100138603A (en) * 2009-06-25 2010-12-31 삼성전자주식회사 Digital photographing apparatus
WO2011121763A1 (en) * 2010-03-31 2011-10-06 キヤノン株式会社 Image processing apparatus and image capturing apparatus using same
WO2011122284A1 (en) * 2010-03-31 2011-10-06 キヤノン株式会社 Image processing device and image capturing apparatus using same
WO2011139288A1 (en) 2010-05-06 2011-11-10 Nikon Corporation Image sharpness classification system
WO2012060835A1 (en) 2010-11-03 2012-05-10 Nikon Corporation Blur detection system for night scene images
WO2013025220A1 (en) 2011-08-18 2013-02-21 Nikon Corporation Image sharpness classification system
US9652833B2 (en) * 2014-03-18 2017-05-16 Honeywell International, Inc. Point spread function estimation for motion invariant images
CN106651791B (en) * 2016-11-21 2023-07-07 云南电网有限责任公司电力科学研究院 Single motion blurred image recovery method
US20220012853A1 (en) * 2020-07-09 2022-01-13 Technoimaging, Llc Joint minimum entropy method for simultaneous processing and fusion of multi-physics data and images

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3627920A (en) * 1969-04-03 1971-12-14 Bell Telephone Labor Inc Restoration of degraded photographic images
US5109425A (en) * 1988-09-30 1992-04-28 The United States Of America As Represented By The United States National Aeronautics And Space Administration Method and apparatus for predicting the direction of movement in machine vision
US5198896A (en) * 1989-10-26 1993-03-30 Canon Kabushiki Kaisha Movement detection apparatus for detecting movement vectors from an image signal
US5712474A (en) * 1993-09-29 1998-01-27 Canon Kabushiki Kaisha Image processing apparatus for correcting blurring of an image photographed by a video camera
US5790709A (en) * 1995-02-14 1998-08-04 Ben-Gurion, University Of The Negev Method and apparatus for the restoration of images degraded by mechanical vibrations

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2843637B2 (en) * 1990-03-20 1999-01-06 オリンパス光学工業株式会社 Camera device
US5990942A (en) * 1991-05-01 1999-11-23 Canon Kabushiki Kaisha Photographing apparatus using image information for blurring correction
US6172707B1 (en) * 1992-06-22 2001-01-09 Canon Kabushiki Kaisha Image pickup device
US6507018B2 (en) * 1996-08-30 2003-01-14 Raytheon Company Ditherless non-uniformity compensation for infrared detector arrays with recursive spatial low pass filtering
JPH11284944A (en) * 1998-03-31 1999-10-15 Canon Inc Image processing method and system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3627920A (en) * 1969-04-03 1971-12-14 Bell Telephone Labor Inc Restoration of degraded photographic images
US5109425A (en) * 1988-09-30 1992-04-28 The United States Of America As Represented By The United States National Aeronautics And Space Administration Method and apparatus for predicting the direction of movement in machine vision
US5198896A (en) * 1989-10-26 1993-03-30 Canon Kabushiki Kaisha Movement detection apparatus for detecting movement vectors from an image signal
US5712474A (en) * 1993-09-29 1998-01-27 Canon Kabushiki Kaisha Image processing apparatus for correcting blurring of an image photographed by a video camera
US5790709A (en) * 1995-02-14 1998-08-04 Ben-Gurion, University Of The Negev Method and apparatus for the restoration of images degraded by mechanical vibrations

Cited By (61)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030164882A1 (en) * 2001-03-15 2003-09-04 Tetsujiro Kondo Image processing device and method, and imaging device
US7218791B2 (en) * 2001-03-15 2007-05-15 Sony Corporation Image processing device and method, and imaging device
US20040001147A1 (en) * 2002-06-19 2004-01-01 Stmicroelectronics S.R.L. Method of stabilizing an image sequence
US20040027454A1 (en) * 2002-06-19 2004-02-12 Stmicroelectronics S.R.I. Motion estimation method and stabilization method for an image sequence
US7852375B2 (en) 2002-06-19 2010-12-14 Stmicroelectronics S.R.L. Method of stabilizing an image sequence
US8325810B2 (en) * 2002-06-19 2012-12-04 Stmicroelectronics S.R.L. Motion estimation method and stabilization method for an image sequence
US7961970B1 (en) 2005-11-30 2011-06-14 Adobe Systems Incorporated Method and apparatus for using a virtual camera to dynamically refocus a digital image
US7623726B1 (en) 2005-11-30 2009-11-24 Adobe Systems, Incorporated Method and apparatus for using a virtual camera to dynamically refocus a digital image
US9390229B1 (en) 2006-04-26 2016-07-12 Dp Technologies, Inc. Method and apparatus for a health phone
US8902154B1 (en) 2006-07-11 2014-12-02 Dp Technologies, Inc. Method and apparatus for utilizing motion user interface
US9495015B1 (en) 2006-07-11 2016-11-15 Dp Technologies, Inc. Method and apparatus for utilizing motion user interface to determine command availability
US7982770B1 (en) 2006-09-08 2011-07-19 Dp Technologies, Inc. Method and apparatus to provide improved image quality in a camera
US20100140461A1 (en) * 2006-12-13 2010-06-10 Georgia Tech Research Corporation Systems and methods for real time multispectral imaging
US8620353B1 (en) 2007-01-26 2013-12-31 Dp Technologies, Inc. Automatic sharing and publication of multimedia from a mobile device
US10744390B1 (en) 2007-02-08 2020-08-18 Dp Technologies, Inc. Human activity monitoring device with activity identification
US8949070B1 (en) 2007-02-08 2015-02-03 Dp Technologies, Inc. Human activity monitoring device with activity identification
US20080238820A1 (en) * 2007-03-29 2008-10-02 Otsuka Electronics Co., Ltd Motion picture image processing system and motion picture image processing method
US7884854B2 (en) * 2007-07-11 2011-02-08 Hewlett-Packard Development Company, L.P. Reducing motion blur from an image
US20090015719A1 (en) * 2007-07-11 2009-01-15 Banner Ron Reducing motion blur from an image
US9183044B2 (en) 2007-07-27 2015-11-10 Dp Technologies, Inc. Optimizing preemptive operating system with motion sensing
US8555282B1 (en) 2007-07-27 2013-10-08 Dp Technologies, Inc. Optimizing preemptive operating system with motion sensing
US9940161B1 (en) 2007-07-27 2018-04-10 Dp Technologies, Inc. Optimizing preemptive operating system with motion sensing
US10754683B1 (en) 2007-07-27 2020-08-25 Dp Technologies, Inc. Optimizing preemptive operating system with motion sensing
US7916177B2 (en) 2007-08-03 2011-03-29 Panasonic Corporation Image-capturing apparatus, image-capturing method and program for detecting and correcting image blur
US20100231731A1 (en) * 2007-08-03 2010-09-16 Hideto Motomura Image-capturing apparatus, image-capturing method and program
US8559756B2 (en) 2007-08-06 2013-10-15 Adobe Systems Incorporated Radiance processing by demultiplexing in the frequency domain
US20090174782A1 (en) * 2008-01-07 2009-07-09 Philippe Kahn Method and Apparatus for Improving Photo Image Quality
US8040382B2 (en) 2008-01-07 2011-10-18 Dp Technologies, Inc. Method and apparatus for improving photo image quality
WO2009089220A1 (en) * 2008-01-07 2009-07-16 Dp Technologies, Inc. Method and apparatus for improving photo image quality
WO2009110868A1 (en) * 2008-03-06 2009-09-11 Nikon Corporation Method for estimating of direction of motion blur in an image
US8472743B2 (en) 2008-03-06 2013-06-25 Nikon Corporation Method for estimating of direction of motion blur in an image
US20100260431A1 (en) * 2008-03-06 2010-10-14 Nikon Corporation Method for estimating of direction of motion blur in an image
US8285344B2 (en) 2008-05-21 2012-10-09 DP Technlogies, Inc. Method and apparatus for adjusting audio for a user environment
US8611693B2 (en) 2008-05-30 2013-12-17 Adobe Systems Incorporated Managing artifacts in frequency domain processing of light-field images
US8996332B2 (en) 2008-06-24 2015-03-31 Dp Technologies, Inc. Program setting adjustments based on activity identification
US11249104B2 (en) 2008-06-24 2022-02-15 Huawei Technologies Co., Ltd. Program setting adjustments based on activity identification
US9797920B2 (en) 2008-06-24 2017-10-24 DPTechnologies, Inc. Program setting adjustments based on activity identification
US8872646B2 (en) 2008-10-08 2014-10-28 Dp Technologies, Inc. Method and system for waking up a device due to motion
US9316840B2 (en) 2009-01-20 2016-04-19 Adobe Systems Incorporated Methods and apparatus for reducing plenoptic camera artifacts
US8698905B2 (en) * 2009-03-11 2014-04-15 Csr Technology Inc. Estimation of point spread functions from motion-blurred images
US20100231732A1 (en) * 2009-03-11 2010-09-16 Zoran Corporation Estimation of point spread functions from motion-blurred images
US8064759B1 (en) 2009-04-15 2011-11-22 Dp Technologies, Inc. Method and apparatus for motion-state based image acquisition
US9529437B2 (en) 2009-05-26 2016-12-27 Dp Technologies, Inc. Method and apparatus for a motion state aware device
US8471920B2 (en) 2009-07-15 2013-06-25 Adobe Systems Incorporated Focused plenoptic camera employing different apertures or filtering at different microlenses
US8890963B2 (en) * 2010-01-12 2014-11-18 Nec Casio Mobile Communications Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US20130169823A1 (en) * 2010-01-12 2013-07-04 Nec Casio Mobile Communications, Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US8692896B2 (en) * 2010-01-12 2014-04-08 Nec Casio Mobile Communications Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US20110169977A1 (en) * 2010-01-12 2011-07-14 Nec Casio Mobile Communications, Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US8717446B2 (en) * 2010-01-12 2014-05-06 Nec Casio Mobile Communications Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US20130170745A1 (en) * 2010-01-12 2013-07-04 Nec Casio Mobile Communications Ltd. Image quality evaluation device, terminal device, image quality evaluation system, image quality evaluation method and computer-readable recording medium for storing programs
US8817015B2 (en) 2010-03-03 2014-08-26 Adobe Systems Incorporated Methods, apparatus, and computer-readable storage media for depth-based rendering of focused plenoptic camera data
US8860833B2 (en) 2010-03-03 2014-10-14 Adobe Systems Incorporated Blended rendering of focused plenoptic camera data
US8803918B2 (en) 2010-08-27 2014-08-12 Adobe Systems Incorporated Methods and apparatus for calibrating focused plenoptic camera data
US8749694B2 (en) 2010-08-27 2014-06-10 Adobe Systems Incorporated Methods and apparatus for rendering focused plenoptic camera data using super-resolved demosaicing
US8724000B2 (en) * 2010-08-27 2014-05-13 Adobe Systems Incorporated Methods and apparatus for super-resolution in integral photography
US8665341B2 (en) 2010-08-27 2014-03-04 Adobe Systems Incorporated Methods and apparatus for rendering output images with simulated artistic effects from focused plenoptic camera data
US9197798B2 (en) 2011-03-25 2015-11-24 Adobe Systems Incorporated Thin plenoptic cameras using microspheres
US9030550B2 (en) 2011-03-25 2015-05-12 Adobe Systems Incorporated Thin plenoptic cameras using solid immersion lenses
US9092875B2 (en) 2011-04-12 2015-07-28 Panasonic Intellectual Property Management Co., Ltd. Motion estimation apparatus, depth estimation apparatus, and motion estimation method
US9232137B2 (en) 2013-04-24 2016-01-05 Microsoft Technology Licensing, Llc Motion blur avoidance
US9538083B2 (en) 2013-04-24 2017-01-03 Microsoft Technology Licensing, Llc Motion blur avoidance

Also Published As

Publication number Publication date
US6987530B2 (en) 2006-01-17
JP2003078812A (en) 2003-03-14
JP4121780B2 (en) 2008-07-23
US20030011717A1 (en) 2003-01-16

Similar Documents

Publication Publication Date Title
US6987530B2 (en) Method for reducing motion blur in a digital image
US20020186309A1 (en) Bilateral filtering in a demosaicing process
KR100736356B1 (en) Image edge detection apparatus and method, image sharpness emphasizing apparatus and method, recorded medium recorded the program performing it
RU2431889C1 (en) Image super-resolution method and nonlinear digital filter for realising said method
EP1754191B1 (en) Characterizing a digital imaging system
KR101134208B1 (en) Imaging arrangements and methods therefor
US7486844B2 (en) Color interpolation apparatus and color interpolation method utilizing edge indicators adjusted by stochastic adjustment factors to reconstruct missing colors for image pixels
EP1529395B1 (en) Shading correction method for image reading means
EP1209903B1 (en) Method and system of noise removal for a sparsely sampled extended dynamic range image
US20120013760A1 (en) Characterization of image sensors
EP2323376B1 (en) Image sensor with shaking compensation
US8520081B2 (en) Imaging device and method, and image processing method for imaging device
US9143706B2 (en) Imaging system utilizing spatial image oscillation
US10341566B2 (en) Method and apparatus for implementing a digital graduated filter for an imaging apparatus
US11223787B2 (en) Information processing apparatus, information processing method, and imaging apparatus
CN112954308B (en) Focusing definition absolute evaluation method and system for imaging display screen
US8452115B2 (en) Method and apparatus for designing restoration filter, and method and apparatus for restoring image using the restoration filter
US20230308760A1 (en) Method and apparatus for implementing a digital graduated filter for an imaging apparatus
CN113228099A (en) Method and system for computing point spread function of digital image detector system based on quantum noise measurement of MTF modulation
EP1154633B1 (en) An apparatus for determining the best image from a dual resolution photo sensor
JP2000081368A (en) Lcd panel picture quality inspection method and device and image-reading method
Soulez et al. Joint deconvolution and demosaicing
US9258461B2 (en) Image processing device and method, and image processing program
US20230041139A1 (en) Image processing for oversampled infrared imaging
US11488286B2 (en) Method for determining Moire pattern, method for suppressing Moire pattern and circuit system thereof

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION