WO2015085956A1 - Processing method and device based on projection image - Google Patents

Abstract

The present invention is applicable to the field of image processing, and provided are a processing method and device based on a projection image. The method comprises: acquiring a secondary imaging image of a projection image on a projection surface in an imaging device, wherein the projection image is formed by a preset reference image being projected on the projection surface, and reference identification points are preset on the reference image; acquiring secondary image identification points in the secondary imaging image; acquiring a distortion coefficient according to the change of the location parameters of the secondary image identification points in the secondary imaging image based on the location parameters of the reference identification points in the reference image; and modifying an image to be projected according to the distortion coefficient. Thus, the processed projection image is less distorted or not distorted at all so as to improve an image projection device's adaptive capability to the environment.

Description

Processing method and device based on projection image Technical field

Embodiments of the present invention relate to the field of image processing, and in particular, to a processing method and apparatus based on a projected image.

Background technique

Projection display is a display method widely used by people in daily work and life. It is popular with many advantages such as adjustable size of projected image, free and flexible selection of projection surface, and free mobility of projection equipment. . The traditional projection display generally adopts a fixed type of ceiling-mounted or table-mounted, and has a fixed projection surface. With the development of technology, the projector is getting smaller and smaller and more and more flexible and movable, such as a wearable projection device. It can be placed on the human body, which increases the uncertainty in the projection environment. The angle change between the projection device and the projection surface and the unevenness of the projection surface can cause distortion of the projected image. How to deal with such image distortion, especially important. The prior art provides a projection device, including a projector, a projection surface, and a camera. The projector projects an interactive interface on a medium, and the user interacts with the interface through a specific gesture, and the gesture is captured and captured by the camera. The computing device recognizes and drives the device to complete the corresponding action according to the recognition result. However, since the specific position of the projection surface is not known in advance, the angles of the projector and the projection surface may change frequently, and the complicated projection environment may result in the projected image. A large projection distortion is generated to distort the displayed image.

Summary of the invention

Embodiments of the present invention provide a processing method and apparatus based on a projected image, which corrects image distortion caused by the projection image when the projection environment is unstable and the projection image is distorted.

In a first aspect, the present invention provides a processing method based on a projected image, comprising the following method:

Obtaining a secondary image of the projection image on the projection surface in the imaging device, the projection image being an image formed by the projection device projecting the preset reference image onto the projection surface, the preset reference image Providing a reference identification point, wherein the reference identification point is formed to form an image identification point in the projection image; and the image point formed by the primary image identification point in the secondary imaging image is a secondary image identifier point;

Obtaining a secondary image identification point in the secondary imaging image;

Obtaining a distortion coefficient according to a change of a position parameter of the secondary image identification point with respect to a position parameter of the reference identifier; a position parameter of the secondary image identification point is the secondary image identification point in the second a position parameter in the image, the position parameter of the reference mark is a position parameter of the reference mark in the reference image;

Distortion correction is performed on the image to be projected according to the distortion coefficient.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the preset reference image is a reference infrared grid line, where the reference identifier point is in different directions in the reference infrared grid line Correspondingly, the projection device is an infrared projection device, and the imaging device is an infrared filter camera device;

The acquiring the secondary image of the projection image on the projection surface in the imaging device comprises:

Obtaining a secondary infrared grid line formed by the infrared filter camera by the projection image on the projection surface;

The acquiring the secondary image identification points in the secondary imaging image includes:

Obtaining intersections of grid lines in different directions in the secondary infrared grid lines, the different directions of the network The intersection of the grid lines is the secondary image identification point.

In conjunction with the first aspect, or the first possible implementation of the first aspect, in a second possible implementation of the first aspect, the determining, according to the secondary image identification point, in the secondary image Obtaining the distortion coefficient of the position parameter relative to the change of the position parameter of the reference identification point in the reference image comprises: obtaining the distortion coefficient by using the following formula

Where R is the distortion coefficient,

a matrix formed in the affine transformation of the reference image, wherein r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are parameters in an affine change, and the x _i and the y _i are And the (x _i , y _i ) is a coordinate value of the secondary image identification point; the x _i ′ and the horizontal coordinate value and the ordinate value in the secondary image. The y _i ' is an abscissa value and an ordinate value of the reference identification point in the reference image, and the (x _i ', y _i ') is a coordinate value of a reference identification point; the affine The transformation process performs a linear transformation of the parameters r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ and the secondary image identification point (x _i , y _i ), and is coupled with the translation parameter r ₁₃ , r ₂₃ The reference identification point (x _i ', y _i ') is described; the formula (2) is a linear equation group composed of the transformation of the formula (1), and the affine transformation distortion coefficient R is calculated by the formula (2).

In conjunction with the second possible implementation of the first aspect, in a third possible implementation manner of the first aspect, the modifying the image to be projected according to the distortion coefficient includes:

The distortion correction is performed on the projected image by the following formula:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the image to be projected according to the distortion coefficient R

The abscissa value and the ordinate value, the x _i and y _i are the abscissa value and the ordinate value of the original pixel coordinate point (x _i , y _i ) of the image to be projected.

In combination with the first aspect and the first possible implementation of the first aspect to any one of the third possible implementations of the first aspect, in a fourth possible implementation of the first aspect, After the obtaining the distortion coefficient based on the change of the position parameter of the secondary image identification point based on the change of the position parameter of the reference image, the method further includes: correcting the reference image according to the distortion coefficient;

The distortion correction is performed on the reference image by the following formula:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the reference image according to the distortion coefficient R

An abscissa value and an ordinate value, the x _k and y _k being an abscissa value and an ordinate value of the original pixel coordinate point (x _k , y _k ) of the reference image.

With the fourth possible implementation of the first aspect, in a fifth possible implementation manner of the first aspect, after the correcting the reference image according to the distortion coefficient, the method further includes:

Obtaining a motion parameter of the projection device when projecting the reference image onto the projection surface to form a projected image;

Obtaining a motion compensation parameter according to the motion parameter;

Performing inverse motion compensation on the distortion-corrected image according to the motion compensation parameter; the distortion-corrected image includes a corrected image to be projected, or a corrected reference image.

In conjunction with the fifth possible implementation of the first aspect, the sixth possible implementation in the first aspect In the mode, before the obtaining the motion compensation parameter according to the motion parameter, the method further includes:

Performing motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

The obtaining motion compensation parameters according to the motion parameter includes:

A motion compensation parameter is acquired in conjunction with the motion parameter and the preliminary motion compensation value.

In conjunction with the fifth possible implementation of the first aspect, in a seventh possible implementation manner of the first aspect, the obtaining the motion compensation parameter according to the motion parameter includes:

Motion estimation is performed on the secondary imaging image in conjunction with the motion parameter to obtain a motion compensation parameter.

In conjunction with the sixth possible implementation of the first aspect, in an eighth possible implementation manner of the first aspect, the obtaining the motion compensation parameter by combining the motion parameter and the preliminary motion compensation value includes:

The motion parameter is a first motion compensation value directly acquired by the acceleration sensor;

Performing motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

The first motion compensation value and the preliminary motion compensation value are weighted and averaged to obtain a motion compensation parameter, which is specifically implemented by the following formula:

M=α*M ₁ +(1-α)*M ₂ (5)

Where M is a motion compensation parameter, M ₁ is a preliminary motion compensation value, and M ₂ is a first motion compensation value,

Said

Said

for

Said

for

The tx _i and ty _i are positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image, and the N is the secondary image identifier in each frame. Number of points;

for

Said

for

The tx _j and ty _j are positional differences between the abscissa and the ordinate of the reference mark in the reference image in the adjacent two frames, and the H is the number of the reference mark points in each frame; An empirical value of the specific gravity is taken between the M ₁ and the M ₂ , and the value of α is in the range of 0 ≤ α ≤ 1.

In conjunction with the seventh possible implementation of the first aspect, the ninth possible implementation in the first aspect In one mode, the motion parameter is a component value u and v of the projection device sensed by the acceleration sensor in the X-axis and Y-axis directions, and the motion parameter is used to estimate the motion of the secondary imaging image. The motion compensation parameter M is obtained by the following formula:

Where tx _i and ty _i are the positional differences between the abscissa of the secondary image identification point and the ordinate in the adjacent two frames in the secondary imaging image, and the N is the second in the secondary imaging image of each frame The number of points of the secondary image identification point, where u and v are the motion parameters, that is, the component values of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, β is

with

The empirical value of the ratio between the values, the value of β is 0 ≤ β ≤ 1.

With reference to the fifth possible implementation of the first aspect to any one of the ninth possible implementation manners of the first aspect, in the tenth possible implementation manner of the first aspect, The parameter performs motion compensation on the distortion-corrected image to be projected, which is specifically implemented by the following formula:

Wherein said

with

For the abscissa value and the ordinate value of the image pixel point coordinates corrected by the distortion coefficient R of the image to be projected,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

In conjunction with the fifth possible implementation of the first aspect to any one of the tenth possible implementations of the first aspect, in an eleventh possible implementation of the first aspect, The compensation parameter M performs motion compensation on the distortion-corrected reference image, which is specifically implemented by the following formula:

Wherein said

with

An abscissa value and an ordinate value of the image pixel point coordinates corrected according to the distortion coefficient R for the reference image,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

In combination with the first aspect and the fifth possible implementation of the first aspect to any one of the eleventh possible implementations of the first aspect, in a twelfth possible implementation of the first aspect The imaging device is at least two to form a stereo vision system;

The acquiring the secondary image identification point in the secondary imaging image specifically includes:

Obtaining a three-dimensional parameter in the projected image by the stereo vision system; and fitting the three-dimensional parameter into a plane, vertically mapping the three-dimensional parameter to the plane, to obtain a two-dimensional coordinate point, the two-dimensional coordinate point Identify points for the secondary image.

In conjunction with the twelfth possible implementation of the first aspect, in a thirteenth possible implementation manner of the first aspect, after the performing distortion correction on the image to be projected according to the distortion coefficient, the method further includes: Adjusting the focal length of the projection device according to the distance D from the projection device to the projection surface, specifically by the following formula:

Wherein the s ₀ is a liquid crystal size of the projection device, and the s ₁ is an empirical reference value of the image size,

That is, the z value in the three-dimensional coordinate values of the N secondary image identification points is taken and the z values are averaged to obtain the distance D.

In a second aspect, the present invention provides a processing apparatus based on a projected image, the apparatus comprising:

An imaging acquiring unit, configured to acquire a secondary imaging image of the projection image on the projection surface in the imaging device, wherein the projection image is an image formed by the projection device projecting the preset reference image onto the projection surface, a reference identification point is disposed in the preset reference image, and the reference identification point is projected to form an image identification point in the projection image; the image formed by the primary image identification point in the secondary imaging image The point is a secondary image identification point;

a marker point acquiring unit, configured to acquire a secondary image identification point in the secondary imaging image;

a distortion coefficient acquiring unit, configured to acquire a distortion coefficient according to a change of a position parameter of the secondary image identification point with respect to a position parameter of the reference identifier; a position parameter of the secondary image identification point is the secondary image Identifying a position parameter of the point in the secondary imaging image, where the position parameter of the reference identification point is a position parameter of the reference identification point in the reference image;

The image to be projected correction unit is configured to perform distortion correction on the image to be projected according to the distortion coefficient R.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the preset reference image is a reference infrared grid line, where the reference identifier point is in different directions in the reference infrared grid line The intersection of grid lines;

The imaging acquiring unit is configured to acquire a secondary infrared grid line formed by the infrared image camera by a projection image on the projection surface;

The image identification point acquiring unit is configured to acquire intersections of grid lines in different directions in the secondary infrared grid lines, and intersection points of the grid lines in the different directions are the secondary image identification points.

With reference to the second aspect, or the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the distortion coefficient acquiring unit is configured to calculate the distortion coefficient by using the following formula

Where R is the distortion coefficient,

a matrix formed in the affine transformation of the reference image, wherein r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are parameters in an affine change, and the x _i and the y _i are And the (x _i , y _i ) is a coordinate value of the secondary image identification point; the x′ _i and the coordinate value of the secondary image identification point in the secondary imaging image. The y′ _i is an abscissa value and an ordinate value of the reference identification point in the reference image, and the (x′ _i , y′ _i ) is a coordinate value of a reference identification point; the affine The transformation process performs a linear transformation of the parameters r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ and the secondary image identification point (x _i , y _i ), and is coupled with the translation parameter r ₁₃ , r ₂₃ The reference identification point (x' _i , y' _i ) is described; the formula (2) is a linear equation group formed by the transformation of the formula (1), and the affine transformation distortion coefficient R is calculated by the formula (2).

In conjunction with the second possible implementation of the second aspect, in a third possible implementation of the second aspect, the image to be projected correction unit is configured to perform distortion correction on the image to be projected by using the following formula:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the image to be projected according to the distortion coefficient R

The abscissa value and the ordinate value, the x _i and y _i are the abscissa value and the ordinate value of the original pixel coordinate point (x _i , y _i ) of the image to be projected.

With reference to the second aspect, and the first possible implementation manner of the second aspect, to any one of the third possible implementation manners of the second aspect, in a fourth possible implementation manner of the second aspect, The apparatus further includes: a reference image correction unit configured to correct the reference image according to the distortion coefficient R; the reference image modification unit is configured to perform distortion correction on the reference image by:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the reference image according to the distortion coefficient R

An abscissa value and an ordinate value, the x _k and y _k being an abscissa value and an ordinate value of the original pixel coordinate point (x _k , y _k ) of the reference image.

In conjunction with the fourth possible implementation of the second aspect, in a fifth possible implementation of the second aspect, the device further includes:

a motion parameter acquisition unit, configured to acquire a motion parameter of the projection device when projecting the reference image onto the projection surface to form a projection image;

a compensation parameter obtaining unit, configured to obtain a motion compensation parameter according to the motion parameter;

And a motion compensation unit, configured to perform inverse motion compensation on the distortion-corrected image according to the motion compensation parameter; the distortion-corrected image includes a corrected image to be projected, or a corrected reference image.

With reference to the fifth possible implementation of the second aspect, in a sixth possible implementation manner of the second aspect, the device includes:

a motion compensation estimating unit, configured to perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

The compensation parameter acquisition unit includes a third acquisition sub-unit for acquiring the motion compensation parameter in conjunction with the motion parameter acquired by the motion parameter acquisition unit and the preliminary motion compensation value acquired by the motion complement estimation unit.

With reference to the fifth possible implementation of the second aspect, in a seventh possible implementation manner of the second aspect, the compensation parameter acquiring unit includes: a fourth acquiring subunit, configured to combine the motion parameter acquiring unit The acquired motion parameters perform motion estimation on the secondary imaging image to obtain motion compensation parameters.

With the sixth possible implementation of the second aspect, in an eighth possible implementation manner of the second aspect, the first motion compensation parameter is directly obtained according to the acceleration sensor, and the first motion compensation parameter is used as the Motion parameters

The motion compensation estimating unit is configured to perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value, and perform a first motion compensation value obtained by the motion parameter acquiring unit and the obtained preliminary motion compensation value. The weighted average obtains the motion compensation parameter, and the motion compensation parameter is obtained by the following formula:

M=α*M ₁ +(1-α)*M ₂ (5)

Where M is a motion compensation parameter, M ₁ is a preliminary motion compensation value, and M ₂ is a first motion compensation value,

Said

Said

for

Said

for

The tx _i and ty _i are positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image, and the N is the secondary image identifier in each frame. Number of points;

for

Said

for

The tx _j and ty _j are positional differences between the abscissa and the ordinate of the reference mark in the reference image in the adjacent two frames, and the H is the number of the reference mark points in each frame; An empirical value of the specific gravity is taken between the M ₁ and the M ₂ , and the value of α is in the range of 0 ≤ α ≤ 1.

In conjunction with the seventh possible implementation of the second aspect, in a ninth possible implementation manner of the second aspect, the motion parameter acquiring unit is configured to acquire the projection device sensed by the acceleration sensor on the X axis and Component values u and v of the displacement in the Y-axis direction, taking the u and v as motion parameters;

The compensation parameter acquisition unit is configured to perform motion estimation on the secondary imaging image according to the motion parameter acquired by the motion parameter acquisition unit to obtain a motion compensation parameter M, and specifically calculate the motion compensation parameter M by using the following formula:

Where tx _i and ty _i are the positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image in the adjacent two frames, and the N is the secondary image identification point in each frame The number of points taken, where u and v are the motion parameters, that is, the component values of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, β is

with

The empirical value of the ratio between the values, the value of β is 0 ≤ β ≤ 1.

With reference to the fifth possible implementation of the second aspect to any one of the ninth possible implementation manners of the second aspect, in a tenth possible implementation manner of the second aspect, the motion compensation unit For performing motion compensation on the distortion-corrected image to be projected by the following formula:

Wherein said

with

For the abscissa value and the ordinate value of the image pixel point coordinates corrected by the distortion coefficient R of the image to be projected,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

With reference to the fifth possible implementation of the second aspect to any one of the tenth possible implementation manners of the second aspect, in the eleventh possible implementation manner of the second aspect, the motion compensation The unit is configured to perform motion compensation on the distortion-corrected reference image by the following formula:

Wherein said

with

An abscissa value and an ordinate value of the image pixel point coordinates corrected according to the distortion coefficient R for the reference image,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

In combination with the second aspect and the fifth possible implementation of the second aspect to any one of the eleventh possible implementations of the second aspect, in a twelfth possible implementation of the second aspect The image identification point obtaining unit further includes:

a stereo vision processing sub-unit for acquiring three-dimensional parameters in the projected image;

a ninth obtaining subunit, configured to fit the three-dimensional parameters acquired by the stereoscopic processing subunit into a plane, and vertically map the three-dimensional parameters to the plane to obtain two-dimensional coordinate points, the two-dimensional coordinate points Identify points for the secondary image.

In conjunction with the twelfth possible implementation of the second aspect, the thirteenth possible aspect of the second aspect In an implementation manner, the device further includes:

The focal length adjusting unit is configured to adjust the focal length of the projection device according to the distance D from the projection device to the projection surface, which is specifically implemented by the following formula:

Wherein the s ₀ is a liquid crystal size of the projection device, and the s ₁ is an empirical reference value of the image size,

That is, the z value in the three-dimensional coordinate values of the N secondary image identification points is taken and the z values are averaged to obtain the distance D.

In the embodiment of the present invention, the distortion coefficient is obtained according to the position parameter of the secondary image identification point in the secondary imaging image and the position parameter of the reference identification point in the reference image, and then the image to be corrected is corrected by the distortion coefficient, so that the processed image is processed. The distortion of the projected image becomes small or even undistorted, thereby improving the environmental adaptability of the image projection apparatus.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention, Those skilled in the art can also obtain other drawings based on these drawings without paying for creative labor.

1 is a flowchart of a method for processing a projection image according to Embodiment 1 of the present invention;

2 is a flowchart of a method for processing a projection image according to Embodiment 2 of the present invention;

3 is a schematic diagram of implementation details of a projection image processing method according to Embodiment 2 of the present invention;

4 is a supplementary flowchart of an embodiment of a method based on projection image processing according to Embodiment 2 of the present invention;

5 is a supplementary flowchart of another implementation manner of a method based on projection image processing according to Embodiment 2 of the present invention;

FIG. 6 is a structural block diagram of a projection image processing apparatus according to Embodiment 3 of the present invention; FIG.

FIG. 7 is a detailed block diagram of a structural block diagram of a projection image processing apparatus according to Embodiment 3 of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

As shown in FIG. 1 , an embodiment of the present invention provides a projection image processing method, which includes the following steps:

S101. Acquire a secondary imaging image of the projection image on the projection surface in the imaging device, where the projection image is an image formed by the projection device projecting the preset reference image onto the projection surface, the preset image A reference identification point is set in the reference image, and the reference identification point is projected to form an image identification point in the projection image; the image point formed by the primary image identification point in the secondary imaging image is twice Like a logo point.

In an embodiment of the invention, the projection device projects a preset reference image onto the projection surface to form a projection image, the imaging device takes the projection image to form a secondary imaging image, and the computing device acquires the secondary imaging image from the imaging device, and the computing device A preset reference image has been stored in advance; a reference identification point is set in the preset reference image, and the reference identification point is projected on the projection surface by the projection device. In the specific implementation process, the reference marker points may be selected according to the rule interval distance, or may not be selected according to the rule interval distance, and all the logo points may be in a square grid shape or a diamond mesh. The grid shape may also be other regular or irregular shapes, and may be determined according to the positional relationship between the projection device and the projection surface and the degree of gradation of the projection surface; in a specific implementation process, the reference marker point is a reference identifier. The two-dimensional coordinate value of the point.

S102. Acquire a secondary image identification point in the secondary imaging image.

In the embodiment of the present invention, after the secondary imaging image is acquired, the secondary image identification point in the secondary imaging image is acquired therefrom, and the secondary image identification point is a secondary image identification point in the projection image. The image point formed in the image is formed. In a specific implementation process, the acquired secondary image identification point is a two-dimensional coordinate value of the secondary image identification point.

S103. Acquire a distortion coefficient R according to a position parameter of the secondary image identification point in the secondary imaging image based on a change of a position parameter of the reference identification point in the reference image.

In the embodiment of the present invention, the computing device extracts the preset reference image that has been stored in advance, extracts the two-dimensional coordinate value of the preset reference identification point, and obtains the coordinate value according to the coordinate value of the reference identification point. According to the position parameter in the reference image, the position parameter in the secondary imaging image is obtained according to the obtained two-dimensional coordinate value of the secondary image identification point, and the position parameter and the secondary image identification point of the reference identification point in the reference image are referenced. The positional parameters in the secondary imaged image are compared to obtain the distortion coefficient, ie the distortion coefficient R, where the deformation is actually an affine transformation, ie R can be represented as a 2x3 matrix:

Optionally, in a specific implementation process, the manner of specifically obtaining R in the embodiment of the present application may be obtained by using the following formula:

The two-dimensional coordinate value (x, y) of each secondary image identification point acquired from the secondary imaging image can be transformed by the affine transformation R into the two-dimensional coordinates of the corresponding reference identification point in the preset reference image. The value (x', y'), in the above formulas (1) and (2), r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are 6 unknown variables, (x', y' a reference point in the reference image, that is, a known point, (x, y) is a secondary image identification point in the secondary imaging image acquired by step S101 and step S102, which is also a known variable, thereby passing the steps S101 and step S102 can obtain six equations by taking corresponding points of three pairs of reference mark points and secondary image mark points, and six unknown variables r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ can be solved by combining six equations. , r ₂₂ , r ₂₃ , that is, the R value is obtained.

S104: Perform distortion correction on the image to be projected according to the distortion coefficient R.

In the embodiment of the present invention, the acquired distortion coefficient R is applied to the image to be projected, and the image to be projected is subjected to distortion correction. In a specific implementation process, n pixel points may be taken in the image to be corrected, each The coordinate position value of the pixel is (x _i , y _i ), and the corresponding pixel in the corrected image is

Optionally, in a specific implementation process, the specific modification process in the embodiment of the present application may be represented by formula (3):

Where R is a known variable and (x _i , y _i ) is a known coordinate value arbitrarily taken on an image to be projected.

The coordinate value of the corrected pixel point corresponding to the known coordinate value (x _i , y _i ).

Optionally, on the basis of the foregoing embodiment, the reference image may be corrected according to the acquired distortion coefficient R, and the reference image is specifically modified by the following formula:

Wherein said

(x _k , y _k ) is the original pixel coordinates of the reference image in order to correct the reference image according to the distortion coefficient R.

In the embodiment of the present invention, by acquiring the secondary imaging image, the secondary image identification point is acquired from the secondary imaging image, and the position parameter of the secondary image identification point in the secondary imaging image and the reference identification point are compared in the reference image. The position parameter in the middle obtains the distortion coefficient R, and then the distortion coefficient R is applied to the image to be corrected, that is, the image to be corrected is pre-corrected so that the output image is not distorted.

In another embodiment of the present invention, the preset reference image is a reference infrared grid line, that is, the infrared grid line is used as the reference image, so that the projection device can project the reference image to the projection surface without interruption in real time and does not interfere with the visible light. The imaging ensures that the correction of the reference image and the image to be projected is also performed in real time without interruption, so that the projected image recognized by the human eye is an undistorted image from beginning to end.

As shown in FIG. 2, an embodiment of the present invention provides an implementation process of an image processing method, which includes the following steps:

S201. Acquire a secondary imaging image of the projection image on the projection surface in the infrared filter imaging device, where the secondary imaging image is a secondary infrared grid line; the projection image is a preset reference infrared image by the infrared projection device. a projection image formed on the projection surface, the projection image being a primary infrared grid line; wherein the preset reference image is provided with a reference identification point, where the reference identification point is specifically in the reference infrared grid line An intersection of grid lines of different directions; the reference marker point forms an image identification point in the projected image;

Specifically, the infrared projection device projects the reference infrared grid line onto the projection surface to form a projection image, and the infrared filter camera captures the projection image to form a secondary infrared grid line, and the computing device acquires the secondary infrared network from the infrared filter camera device. a grid line, and a preset reference infrared grid line has been stored in advance in the computing device; the intersection point of the grid lines in different directions in the reference infrared grid line is the reference identification point, In a specific implementation process, the shape of the reference infrared grid line may be a square grid shape, a diamond grid shape, or other regular or irregular shapes, which may be according to an infrared projection device and a projection surface. The positional relationship and the degree of gradation of the projection surface are determined; in a specific implementation process, the reference marker point is a two-dimensional coordinate value of the reference marker point.

S202. Acquire a secondary image identification point in the secondary infrared grid line, where the secondary image identification point is a primary image identification point in the projected image in the acquired secondary infrared grid line. The formed image point; wherein, the intersection point of the grid lines in different directions in the secondary infrared grid line in the secondary imaging image is a secondary image identification point; in a specific implementation process, the acquired secondary image identifier The point is the two-dimensional coordinate value of the secondary image identification point.

S203. Acquire a distortion coefficient R according to a position parameter of the secondary image identification point in the secondary imaging image based on a change of a position parameter of the reference identification point in the reference image.

In the embodiment of the present invention, the computing device extracts the preset reference infrared grid line that has been stored in advance, and extracts the two-dimensional coordinate value of the preset reference marker point, that is, refers to different directions in the infrared grid line. The two-dimensional coordinate value of the intersection point of the grid line; and obtaining the position parameter in the reference infrared grid line according to the two-dimensional coordinate value of the reference identification point, according to the obtained two-dimensional coordinate value of the secondary image identification point To the positional parameter in the secondary infrared grid line, the positional parameter of the reference identification point in the reference infrared grid line and the position parameter of the secondary image identification point in the secondary infrared grid line are compared, thereby obtaining The deformation coefficient is the distortion coefficient R. The deformation here is actually an affine transformation, that is, R can be expressed as a 2×3 matrix:

Optionally, in a specific implementation process, the manner of specifically obtaining R in the embodiment of the present application may be obtained by using the following formula:

The two-dimensional coordinate value (x, y) of each secondary image identification point acquired from the secondary infrared grid line can be transformed by the affine transformation R to the corresponding reference identifier in the preset reference infrared grid line. The two-dimensional coordinate value of the point (x', y'), in the above formulas (1) and (2), r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are 6 unknown variables, ( x', y') is a reference mark in the reference infrared grid line, that is, a known point, and (x, y) is a secondary image mark point in the secondary infrared grid line acquired by step S201 and step S202. It is also a known variable, and thus six equations can be obtained by taking the corresponding points of the three pairs of reference identification points and secondary image identification points through steps S201 and S202, and six unknown variables can be solved by combining six equations. ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , that is, an R value is obtained.

S204, correcting the image to be projected according to the distortion coefficient R.

In the embodiment of the present invention, the acquired distortion coefficient R is applied to the image to be projected, and the image to be projected is subjected to distortion correction. In a specific implementation process, n pixel points may be taken in the image to be corrected, each The coordinate position value of the pixel is (x _i , y _i ), and the corresponding pixel in the corrected image is

Optionally, in a specific implementation process, the specific modification process in the embodiment of the present application may be represented by formula (7):

Where R is a known variable and (x _i , y _i ) is a known coordinate value arbitrarily taken on a known reference infrared grid line and a known image to be projected.

The coordinate value of the corrected pixel point corresponding to the known coordinate value (x _i , y _i ).

Optionally, on the basis of the foregoing embodiment, the reference image may be corrected according to the acquired distortion coefficient R, and the reference image is specifically modified by the following formula:

Wherein said

(x _k , y _k ) is the original pixel coordinates of the reference image in order to correct the reference image according to the distortion coefficient R.

In the embodiment of the present invention, after the secondary infrared grid line is acquired, the secondary image identification point in the secondary infrared grid line is obtained therefrom, and in the specific implementation process, the secondary image identifier may be acquired. The two-dimensional coordinate value of the point.

In the embodiment of the present invention, the infrared grid line is used as the preset reference image, and the intersection point of the grid lines in different directions in the infrared grid line is taken as the reference identification point, so that the infrared projection device can continuously project the reference image in real time. At the same time, the imaging of the visible light is not disturbed, so that the reference image and the image to be projected can be corrected in real time without interruption, so that the display image can be presented without distortion even if the projection environment is complicated.

Optionally, based on the foregoing embodiment, the method may further include:

There are at least two imaging devices to form a stereoscopic vision system; as shown in FIG. 3, in the embodiment of the present invention, two imaging devices may be configured to form a binocular stereo vision system; the two imaging devices will capture the captured image. The formed secondary imaging image is transmitted to the computing device, and the computing device processes the secondary imaging image transmitted by the two imaging devices by the binocular stereo vision method. In a specific implementation process, the processing step is: extracting the secondary imaging The quadratic image in the image identifies a three-dimensional parameter of the point, the three-dimensional parameter is fitted into a plane by least squares method, and the three-dimensional parameter is vertically mapped onto the fitting plane to obtain the secondary image identification point. A two-dimensional coordinate point on the projection surface, that is, a secondary image identification point described in the embodiment of the present invention.

Further, the binocular stereo vision system may be a binocular stereo vision system with parallel optical axes, in which there may be two cameras for taking in a projected image, and the computing device acquires a secondary image from the camera and Extracting the three-dimensional parameters of the secondary image identification point in the secondary imaging image, that is, the three-dimensional coordinate value of the secondary image identification point, as follows: the distance between the centers of the two camera lenses is e, two imaging The focal length of the head is f, the midpoint of the center line of the two camera lenses is taken as the origin, the lens center of the camera 1 is directed to the X-axis of the lens center of the camera 2, the optical axis of the parallel camera is the Z-axis, and the XYZ-axis direction conforms to the right-hand rule. To establish a coordinate system, the coordinates of a point (x, y, z) in the coordinate system can be calculated by the following formula:

Where (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are the imaging positions of the (x, y, z) on the imaging surface of the camera 1 and the camera 2, respectively, so that we can find all the secondary image identification points in The position in the three-dimensional space is provided with N secondary image identification points, and the three-dimensional coordinate values are {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ), ..., (x ^N , respectively). y ^N , z ^N )}.

Then, {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ), ..., (x ^N , y ^N , z ^N )} can be fitted to a plane by least squares method:

Ax+By+Cz+D=0 (12)

Where C≠0, since the parallel camera optical axis direction is the Z axis, the projection surface cannot be parallel to the Z axis,

Then formula (6) can be written as:

z=a ₀ x+a ₁ y+a ₂ (13)

Where a ₀ , a ₁ , a ₂ are the parameters to be sought, and the least squares method requires the following formula to be the smallest:

To minimize S, you should satisfy:

which is:

Finishing the above formula can be obtained:

Thus, we get a space plane expressed by equation (7), and then {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),..., (x ^N , y ^N , z ^N )} is vertically projected onto the fitting plane to obtain the two-dimensional coordinates {(x ^1' ), (y ^1′ ), (x ^2′ , y ^2′ ),... (x ^N' , y ^N' )}, which is the coordinate value of the secondary image identification point in the secondary imaging image. The specific two-dimensional coordinate value obtained after the vertical projection of the above three-dimensional coordinate value can be realized by the following calculation process:

From the above formula, a known fitting plane Ax+By+Cz+D=0 has been obtained, so that {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),... , (x ^N , y ^N , z ^N )} three-dimensional coordinates projected perpendicularly to the plane

among them

get

Then, map it to a two-dimensional coordinate {(x ^1' , y ^1′ ), (x ^2′ , y ^2′ ),..., (x ^N′ , y ^N′ )}, where the coordinate origin needs to be determined. And the X-axis direction (the Y-axis is perpendicular to the coordinate origin and the X-axis). Alternatively, in the specific implementation process, the coordinate origin and the X-axis direction can be obtained as follows:

The specific mark is made in the preset reference image, so that the specific mark point is easily detected in the reference image. In the specific implementation process, the reference image is selected as the grid image, and the word "meter" is made in the grid image. The specific mark of the type, as shown in Fig. 6, makes a specific mark in the projected mesh image, and projects the mesh image with the "meter" mark, the two meshes with the "meter" mark The point is easily detected in the image, and the middle "m" shaped mark can be used as the origin of the coordinate system in the subsequent calculation, and the direction from the point to the other "meter" mark point is taken as the X-axis direction.

Set two "meter" font points through the formula

Calculated 3D coordinate value

with

Take

For the origin, to

direction

For the X-axis and the Y-axis, the coordinate origin and the X-axis are perpendicular, and the unit length is the same as the original three-dimensional coordinate system, thereby establishing a two-dimensional coordinate system. will

Mapping to the two-dimensional coordinate system yields {(x ^1' , y ^1' ), (x ^2' , y ^2' ),..., (x ^N' , y ^N' )}:

x ^i' = l ₁ · cos θ, y ⁱ = l ₁ · sin θ, where i = 1, 2, ..., N (19)

among them:

Where l ₁ represents the three-dimensional coordinate point on the fitted plane

The coordinate origin to the established two-dimensional coordinate system

the distance;

The l ₂ represents a three-dimensional coordinate point on the fitting plane

The coordinate origin to the established two-dimensional coordinate system

the distance;

The l ₃ represents a three-dimensional coordinate point on the fitting plane

Another m-shaped point outside the origin of the established two-dimensional coordinate system

the distance;

The θ represents the origin of the coordinate

For the apex, point

To the point

Lines and points

To the point

The angle between the straight lines.

In the embodiment of the present invention, considering the complexity of the projection environment, the projection surface may be presented in a three-dimensional state of unevenness. In this case, the projection surface presented in the three-dimensional state must be fitted into a two-dimensional plane to obtain the final secondary image identifier. The two-dimensional coordinate value of the point, and based on this, the distortion coefficient R is extracted to correct the reference image and the image to be projected, thereby making the image projected even in a complicated projection environment not distorted.

Optionally, in the embodiment of the present invention, at least two imaging devices may also be at least two camera devices equipped with an infrared filter camera.

On the basis of all the foregoing embodiments of the present invention, optionally, as shown in FIG. 4, the solution may further include the following implementation manners, and the specific steps are as follows:

S301. Acquire a motion parameter when the projection device projects the reference image onto the projection surface to form a projected image.

In a specific implementation, the motion parameters may be taken by an acceleration sensor that should be in close proximity and rigidly coupled to the projection device such that the acceleration sensor can accurately capture the motion of the projection device to achieve the most accurate motion parameters. In a specific implementation process, the motion parameter may be a motion compensation directly obtained by the acceleration sensor, or may be a motion direction information obtained by the acceleration sensor.

S302. Obtain a motion compensation parameter according to the motion parameter. As shown in FIG. 5, the method may be implemented by the S302a or S302b method:

S302a, performing motion estimation on the secondary imaging image to obtain a preliminary motion compensation value, and combining the motion parameter and the preliminary motion compensation value to obtain a motion compensation parameter M;

When the motion parameter is the first motion compensation M ₂ directly acquired by the acceleration sensor, performing motion estimation on the secondary imaging image: specifically, taking two adjacent frames of the secondary imaging image, which are a standard frame and a current frame. And taking the coordinate value of the corresponding pixel in the standard frame and the current frame, and finding the coordinate position difference of the corresponding point, and finally averaging all the coordinate position differences to obtain the preliminary motion compensation value M ₁ , combining the motion parameter M ₂ and the preliminary motion The compensation value M ₁ acquires the motion compensation parameter M.

Optionally, in the embodiment of the present invention, the method for acquiring the motion compensation parameter M in combination with the motion parameter M ₂ and the preliminary motion compensation value M ₁ may be: the motion parameter, that is, the M ₂ first motion compensation value, and the preliminary The motion compensation value M ₁ is weighted and averaged to obtain a motion compensation parameter M, which is specifically implemented by the following formula:

M=α*M ₁ +(1-α)*M ₂ (21)

Wherein said

Said

Said

Said

The (tx _i , ty _i ) is a position difference of a secondary image identification point in the adjacent two frames in the secondary imaging image, and the N is a reference point of the secondary image identification point in each frame. Number of points;

Said

The (tx _j , ty _j ) is a position difference of a reference identification point in the reference image in two adjacent frames, where M is a number of points of the reference identification point in each frame; An empirical value of the specific gravity between the M ₁ and the M ₂ , wherein the value of α ranges from 0 ≤ α ≤ 1.

Optionally, the embodiment of the present invention may also be:

S302b: Perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value, and obtain a motion compensation parameter M according to the motion parameter and the preliminary motion compensation value.

When the motion parameter is the motion direction information obtained by the acceleration sensor, in a specific implementation process, The motion direction information is a component value (u, v) of the displacement of the projection device sensed by the acceleration sensor in the X and Y directions, and the motion estimation parameter is obtained by performing motion estimation on the second imaging image. M.

Optionally, in the embodiment of the present invention, performing motion estimation on the secondary imaging image in combination with the motion direction information to obtain the motion compensation parameter M may be implemented by using the following formula:

Taking the value of the acceleration sensor in the X and Y directions, and setting its value to (u, v), its angle with the X direction is:

It is assumed that there are N grid points in the secondary imaging image, that is, there are N secondary image identification points, and the displacement of each identification point obtained by the position difference of two adjacent frames is (tx _i , ty _i ), i=1 , 2, ..., N, then the angle θ _i and the size L _{i of} each marker point displacement with the X direction are:

The displacement direction of the grid point corrected by the acceleration sensor is:

θ' _i =α·θ _i +(1-α)·θ _uv (25)

Where α is an empirical value, such as α = 0.5.

The components of the final motion compensation M in the X and Y directions are:

which is:

Where (tx _i , ty _i ) is a position difference of the secondary image identification point in the adjacent two frames in the secondary image, and the N is a point of the secondary image identification point in each frame Quantity, where (u, v) is a motion parameter, that is, a component value of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, α is

with

The empirical value of the ratio between the values, the value of the α is 0 ≤ α ≤ 1.

S303. Perform inverse motion compensation on the distortion-corrected image according to the motion compensation parameter M. The distortion-corrected image includes a corrected image to be projected, or a corrected reference image.

Optionally, in the embodiment of the present invention, performing inverse motion compensation on the image to be projected that is subjected to distortion correction may be implemented by using the following formula:

among them

The pixel coordinates of the image after motion compensation for the pixel coordinates of the corrected image to be projected.

Optionally, in the embodiment of the present invention, performing inverse motion compensation on the distortion-corrected reference image may be implemented by using the following formula:

Wherein said

Image pixel point coordinates for correcting the reference image according to a distortion coefficient R, wherein

The image pixel coordinates obtained by inverse motion compensation of the image pixel coordinate coordinates after distortion correction, wherein (tx, ty) is the motion compensation parameter M.

In the embodiment of the present invention, considering the complexity of the projection environment, the projection device may move or In the case of shaking, thereby causing the projected picture to be shaken, the embodiment of the present invention obtains the most accurate motion compensation parameter M by the combination of the acceleration sensor and the motion estimation, and performs the inverse motion of the corrected reference image and the image to be projected after the distortion correction. Compensation to ensure that the output image is not shaken in the case of a complex projection environment.

Based on all the foregoing embodiments of the present invention, optionally, the embodiment of the present invention may further include:

Projecting the distortion-corrected image or the distortion-corrected and inverse-motion-compensated image onto the projection surface includes: adjusting the focal length of the projection device according to the distance D from the projection device to the projection surface, and causing the projection device to project an image onto the projection surface The size is in line with the best visual reception, which is achieved by the following formula:

Wherein the s ₀ is a liquid crystal size of the projection device, and the s ₁ is an empirical reference value of the image size,

That is, the z value in the three-dimensional coordinate values of the N secondary image identification points is taken and the z values are averaged to obtain the distance D.

The three-dimensional coordinate value of the secondary image identification point is obtained by the stereo vision system.

In a specific implementation process, at least two imaging devices are configured to form a stereo vision system; in conjunction with the flowchart 3, in the embodiment of the present invention, two imaging devices may be configured to form a binocular stereo vision system; The imaging device transmits the secondary imaging image formed by the captured projection image to the computing device, and the computing device processes the secondary imaging image transmitted by the two imaging devices by the binocular stereo vision method. In the specific implementation process, the processing step And extracting a three-dimensional parameter of the secondary image identification point in the secondary imaging image.

Further, the binocular stereo vision system may be a binocular stereo vision system with parallel optical axes, in which there may be two cameras for capturing projected images, and the computing device is from the camera. Obtaining a secondary imaging image and extracting a three-dimensional parameter of the secondary image identification point in the secondary imaging image, that is, a three-dimensional coordinate value of the secondary image identification point, as follows: the distance between the centers of the two camera lenses is e, and the two cameras are The focal length is f, the midpoint of the center line of the two camera lenses is taken as the origin, the lens center of the camera 1 is directed to the X-axis of the lens center of the camera 2, the optical axis of the parallel camera is the Z-axis, and the XYZ-axis direction conforms to the right-hand rule, and the coordinates are established. For the system, the coordinates of a point (x, y, z) in the coordinate system can be calculated by the following formula:

Wherein (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are (x, y, z) imaging positions of the imaging surface of the camera 1 and the camera 2 in the stereoscopic processing sub-unit 302a, respectively. In this way, we can find the position of all the secondary image identification points in three-dimensional space, with N secondary image identification points, the coordinates are {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),...,(x ^N ,y ^N ,z ^N )}.

Based on the Z coordinate of these points, the approximate distance of the projection surface from the projector can be estimated:

The focal length of the visible light projector is automatically adjusted according to the distance d, so that the image size projected by the visible light projector onto the projection surface is suitable for wearing and viewing. The following table is a reference value of the projection distance and the projected image size obtained empirically.

Table 1. Reference values for projection distance and projected image size

After obtaining the projection distance d (unit: m), the focal length f (unit: m) that the projector needs to set can be obtained by the following formula:

Where s ₀ is the projector liquid crystal size (unit: inch), s ₁ is the size (in inches) of the projected image obtained according to the distance d query table 1, the value of f is limited by hardware, must be located in [f Between _min , f _max ], that is, the actual value is max(min(f, f _max ), f _min ). In order to avoid the focal length changing due to the change of the distance, it can be specified that the new focal length calculation is performed only when the absolute value of the difference between the current distance and the previous adjustment of the focal length is greater than a given threshold (for example, 0.5 m). With settings.

In the embodiment of the present invention, on the basis of the foregoing embodiment, the distance d between the projection surface and the imaging device is obtained by averaging the Z coordinate values of the secondary image identification points in the three-dimensional space, and automatically according to the distance d. Adjust the focal length of the projection device so that the image size projected by the projection device onto the projection surface is optimally viewed.

Embodiment 2

An embodiment of the present invention provides a processing apparatus based on a projected image. As shown in FIG. 6, the apparatus includes:

An imaging acquiring unit 301, configured to acquire a secondary imaging image of the projection image on the projection surface in the imaging device, where the projection image is an image formed by the projection device projecting the preset reference image onto the projection surface, a reference identification point is disposed in the preset reference image, and the reference identification point is projected to form an image identification point in the projection image; the primary image identification point is formed in the secondary imaging image. The image point is a secondary image identification point;

The image identification point obtaining unit 302 is configured to acquire a secondary image identification point in the secondary imaging image;

a distortion coefficient acquisition unit 303, configured to acquire a distortion coefficient according to a change of a position parameter of the reference image identifier according to a position parameter of the secondary image identification point; a position parameter of the secondary image identification point is the secondary image Identifying a position parameter of the point in the secondary imaging image, where the position parameter of the reference identification point is a position parameter of the reference identification point in the reference image;

The image to be projected correction unit 304 is configured to perform distortion correction on the image to be projected according to the distortion coefficient R.

In the embodiment of the present invention, the projection device projects the preset reference image to the projection surface to form a projection image, the imaging device picks up the projection image to form a secondary imaging image, and the imaging acquisition unit 301 acquires the secondary imaging image from the imaging device, and images the image. The preset reference image has been stored in the acquisition unit 301 in advance; the reference identification point is set in the preset reference image, and the reference identification point is projected on the projection surface by the projection device to form an image identification point; The reference identification point may be selected according to a regular interval, or may not be selected according to a regular interval. The pattern represented by all the identification points may be a square grid shape, a diamond grid shape, or other rules or not. The regular shape may be determined according to the positional relationship between the projection device and the projection surface and the gradual degree of the projection surface; in a specific implementation process, the reference identification point is a two-dimensional coordinate value of the reference identification point.

After the imaging acquisition unit 301 acquires the secondary imaging image, the image identification point acquiring unit 302 acquires the secondary image identification point in the secondary imaging image from the secondary imaging image, and the secondary image identification point is once in the projection image. The image point formed by the marker point in the secondary imaging image, in a specific implementation process, the acquired secondary image identification point is a two-dimensional coordinate value of the secondary image identification point.

The distortion coefficient acquisition unit 303 extracts the preset reference image that has been stored in advance, extracts the two-dimensional coordinate value of the preset reference identification point, and obtains the reference image in the reference image according to the coordinate value of the reference identification point. The position parameter is obtained according to the secondary image identification point acquired by the image identification point acquiring unit 302. The two-dimensional coordinate value obtains its positional parameter in the secondary imaging image, and the distortion coefficient acquisition unit 303 compares the positional parameter of the reference identification point in the reference image with the positional parameter of the secondary image identification point in the secondary imaging image. , thereby obtaining a distortion coefficient, that is, a distortion coefficient R.

Specifically, the distortion coefficient obtaining unit 303 can be configured to calculate the distortion coefficient by the following formula:

Where R is the distortion coefficient,

a matrix formed in the affine transformation of the reference image, wherein r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are parameters in an affine change, and the x _i and the y _i are And the (x _i , y _i ) is a coordinate value of the secondary image identification point; the x′ _i and the coordinate value of the secondary image identification point in the secondary imaging image. The y′ _i is an abscissa value and an ordinate value of the reference identification point in the reference image, and the (x′ _i , y′ _i ) is a coordinate value of a reference identification point; the affine The transformation process performs a linear transformation of the parameters r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ and the secondary image identification point (x _i , y _i ), and is coupled with the translation parameter r ₁₃ , r ₂₃ The reference identification point (x' _i , y' _i ) is described; the formula (2) is a linear equation group formed by the transformation of the formula (1), and the affine transformation distortion coefficient R is calculated by the formula (2).

Correspondingly, the image to be projected correction unit 304 can be used to perform distortion correction on the image to be projected by the following formula:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the image to be projected according to the distortion coefficient R

The abscissa value and the ordinate value, the x _i and y _i are the abscissa value and the ordinate value of the original pixel coordinate point (x _i , y _i ) of the image to be projected.

On the basis of the apparatus in the above embodiment, a reference image correcting unit 305 may be further included for correcting the reference image according to the distortion coefficient R; the reference image correcting unit is configured to refer to the reference by the following formula Image distortion correction:

Where R is the distortion coefficient,

with

Image pixel coordinate point for correcting the reference image according to the distortion coefficient R

An abscissa value and an ordinate value, the x _k and y _k being an abscissa value and an ordinate value of the original pixel coordinate point (x _k , y _k ) of the reference image.

As an optional implementation manner, the imaging acquiring unit 301 is configured to acquire a secondary infrared grid line of a projection image on the projection surface in the infrared filter camera;

The image identification point acquisition unit 302 is configured to acquire intersections of grid lines in different directions in the secondary infrared grid lines, and intersection points of the grid lines in the different directions are the secondary image identification points.

The specific reference image in the device of the embodiment of the present invention is a reference infrared grid line, in the specific application, because the image to be projected correction unit 304 and the reference image correction unit may perform real-time uninterrupted correction on the image to be corrected. The reference identification point is an intersection of the grid lines in different directions in the reference infrared grid line, that is, the infrared grid line is used as the reference image, so that the projection device can project the reference image to the projection surface in real time without disturbing the visible light. The imaging ensures that the correction of the reference image and the image to be projected is also performed in real time, so that the projected image recognized by the human eye is an undistorted image from beginning to end.

The image recognition point acquisition unit 302 may also be inaccurate. The method may further include: a stereo vision processing sub-unit 302a for acquiring in the projected image Three-dimensional parameters.

The image identification point acquisition sub-unit 302b is configured to fit the three-dimensional parameters acquired by the stereoscopic vision processing sub-unit 302a into a plane, and vertically map the three-dimensional parameters to the plane to obtain a two-dimensional coordinate point. The two-dimensional coordinate point is the secondary image identification point.

There are at least two imaging devices in the stereoscopic processing sub-unit 302a; in conjunction with FIG. 3, in the embodiment of the present invention, there may be two imaging devices, that is, a binocular stereo vision system; the imaging acquisition unit 301 will capture the captured The secondary imaging image formed by the projection image is transmitted to the image identification point acquisition unit 302, and the stereoscopic vision processing sub-unit 302a processes the secondary imaging image transmitted by the two imaging devices by the binocular stereo vision method. The processing step is: extracting a three-dimensional parameter of the secondary image identification point in the secondary imaging image, fitting the three-dimensional parameter into a plane by least squares, and vertically mapping the three-dimensional parameter to the fitting plane The two-dimensional coordinate point of the secondary image identification point on the projection surface, that is, the secondary image identification point described in the embodiment of the present invention is obtained.

Further, the binocular stereo vision system may be a binocular stereo vision system with parallel optical axes, in which there may be two cameras for taking in a projected image, and the computing device acquires a secondary image from the camera and Extracting the three-dimensional parameter of the secondary image identification point in the secondary imaging image, that is, the three-dimensional coordinate value of the secondary image identification point, as follows: the distance between the centers of the two camera lenses is e, and the focal lengths of the two cameras are f, The midpoint of the center line of the two camera lenses is the origin. The center of the lens of the camera 1 is directed to the X axis of the lens center of the camera 2, the optical axis of the parallel camera is the Z axis, and the XYZ axis direction conforms to the right hand rule. The coordinate system is established at the coordinate. The coordinates of a point (x, y, z) in the system can be calculated by the following formula:

Where (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are the imaging positions of the (x, y, z) on the imaging surface of the camera 1 and the camera 2, respectively, so that we can find all the secondary image identification points in The position in the three-dimensional space is provided with N secondary image identification points, and the three-dimensional coordinate values are {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ), ..., (x ^N , respectively). y ^N , z ^N )}.

Then, {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ), ..., (x ^N , y ^N , z ^N )} can be fitted to a plane by least squares method:

Ax+By+Cz+D=0 (8)

Where C≠0, since the parallel camera optical axis direction is the Z axis, the projection surface cannot be parallel to the Z axis,

Then formula (8) can be written as:

z=a ₀ x+a ₁ y+a ₂ (9)

Where a ₀ , a ₁ , a ₂ are the parameters to be sought, and the least squares method requires the following formula to be the smallest:

To minimize S, you should satisfy:

which is:

Finishing the above formula can be obtained:

Thus, we get a spatial plane expressed by equation (9), and then {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),..., (x ^N , y ^N , z ^N )} is vertically projected onto the fitting plane, and the two-dimensional coordinates of these points in this plane {(x ^1' , y ^1' ), (x ^2' , y ^2' ),..., (x ^N' , y ^N' )}, which is the coordinate value of the secondary image identification point in the secondary imaging image. The specific two-dimensional coordinate value obtained after the above-mentioned three-dimensional coordinate value is vertically projected can be realized by the following calculation process:

From the above formula, a known fitting plane Ax+By+Cz+D=0 has been obtained, so that {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),... , (x ^N , y ^N , z ^N )} three-dimensional coordinates projected perpendicularly to the plane

among them

get

After that, map it to two-dimensional coordinates.

{(x ^1' , y ^1′ ), (x ^2′ , y ^2′ ),...,(x ^N′ , y ^N′ )}, where the coordinate origin and the X-axis direction need to be determined (the Y-axis passes through the coordinate origin) It is perpendicular to the X axis. Alternatively, during the specific implementation process, the coordinate origin and the X axis direction can be obtained as follows:

Make specific marks in the preset reference image so that specific points are easily detected in the reference image In the specific implementation process, the reference image is selected as a grid image, and a specific mark with a "meter" shape is made in the grid image, as shown in FIG. 6, a specific mark is made in the projected grid image. Projecting such a grid image with the "meter" type mark, the two grid points with the "meter" type mark are easily detected in the image, and the middle "meter" type mark can be used as a follow-up Calculate the origin of the coordinate system from which the direction of the other "meter" shaped mark points is taken as the X-axis direction.

Set two "meter" font points through the formula

Calculated 3D coordinate value

with

Take

For the origin, to

direction

For the X-axis and the Y-axis, the coordinate origin and the X-axis are perpendicular, and the unit length is the same as the original three-dimensional coordinate system, thereby establishing a two-dimensional coordinate system. will

Mapping to the two-dimensional coordinate system yields {(x ^1' , y ^1' ), (x ^2' , y ^2' ),..., (x ^N' , y ^N' )}:

x ^i' = l ₁ · cos θ, y ^i' = l ₁ · sin θ, where i = 1, 2, ..., N (15)

among them:

Where l ₁ represents the three-dimensional coordinate point on the fitted plane

The coordinate origin to the established two-dimensional coordinate system

the distance;

The l ₂ represents a three-dimensional coordinate point on the fitting plane

The coordinate origin to the established two-dimensional coordinate system

the distance;

The l ₃ represents a three-dimensional coordinate point on the fitting plane

Another m-shaped point outside the origin of the established two-dimensional coordinate system

the distance;

The θ represents the origin of the coordinate

For the apex, point

To the point

Lines and points

To the point

The angle between the straight lines.

In this embodiment, considering the complexity of the projection environment, the projection surface may be presented in a three-dimensional state of unevenness. In this case, the projection surface presented in the three-dimensional state must be fitted into a two-dimensional plane to obtain the final secondary image identification point. The two-dimensional coordinate values, and based on this, extract the distortion coefficient R to correct the reference image and the image to be projected, thereby making the image projected even in a complicated projection environment not distorted.

All the above embodiments have distortion correction of the distortion of the projected image, so that the image after distortion correction is no longer distorted, but in practical applications, wearable projection devices have emerged due to the popularity of wearable devices. When the wearable projection device is working, it is likely that the projected image will be shaken as the human body moves, or the slight jitter of the human body may adversely affect the projection effect of the wearable projector. On the basis of the embodiment, in consideration of de-jittering of an image, the embodiment of the present invention may further include the following devices:

a motion parameter acquiring unit 306, configured to acquire a motion parameter when the projection device projects the reference image onto the projection surface to form a projected image;

a compensation parameter obtaining unit 307, configured to obtain a motion compensation parameter according to the motion parameter;

The motion compensation unit 308 is configured to perform inverse motion compensation on the distortion-corrected image according to the motion compensation parameter; the distortion-corrected image includes a modified image to be projected, or a corrected reference image.

The motion parameter obtaining unit 306 may be further configured to directly acquire the first motion compensation value according to the acceleration sensor, and use the first motion compensation parameter as the motion parameter; the motion parameter acquiring unit 306 may further be configured to acquire the sensed sensor. The component values u and v of the projection device being displaced in the X-axis and Y-axis directions use the u and v as motion parameters.

Embodiments of the present invention may further include a motion compensation estimating unit configured to perform the secondary imaging image The motion estimation obtains a preliminary motion compensation value; as an optional implementation manner, the compensation parameter acquisition unit 307 can be used to combine the motion parameters acquired by the motion parameter acquisition unit 306, that is, the first motion compensation parameter, and obtain the motion compensation unit by the motion compensation unit. The initial motion compensation value obtains the motion compensation parameter.

Specifically, the motion compensation estimation unit may be configured to perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

The motion parameter obtaining unit 306 may be configured to perform weighted averaging of the first motion compensation value and the preliminary motion compensation value obtained by the motion complement estimation unit to obtain a motion compensation parameter, where the motion compensation parameter is specifically calculated by using the following formula:

M=α*M ₁ +(1-α)*M ₂ (17)

Where M is a motion compensation parameter, M ₁ is a preliminary motion compensation value, and M ₂ is a first motion compensation value,

Said

Said

for

Said

for

The tx _i and ty _i are positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image, and the N is the secondary image identifier in each frame. Number of points;

for

Said

for

The tx _j and ty _j are positional differences between the abscissa and the ordinate of the reference mark in the reference image in the adjacent two frames, and the H is the number of the reference mark points in each frame; An empirical value of the specific gravity is taken between the M ₁ and the M ₂ , and the value of α is in the range of 0 ≤ α ≤ 1.

As another optional implementation manner, the compensation parameter obtaining unit 307 may be further configured to perform motion estimation on the secondary imaging image according to the motion parameters u and v acquired by the motion parameter acquiring unit 306 to obtain a motion compensation parameter.

Specifically, the compensation parameter obtaining unit 307 may be further configured to perform motion estimation on the secondary imaging image by using the motion parameter acquired by the motion parameter acquiring unit 306 to obtain a motion compensation parameter M, where the motion compensation is calculated by using the following formula. Parameter M:

Where tx _i and ty _i are the positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image in the adjacent two frames, and the N is the secondary image identification point in each frame The number of points taken, where u and v are the motion parameters, that is, the component values of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, β is

with

The empirical value of the ratio between the values, the value of β is 0 ≤ β ≤ 1.

After any of the above embodiments, the motion compensation unit 308 in the apparatus may include a first motion compensation sub-unit for performing motion compensation on the distortion-corrected image to be projected by the following formula:

Wherein said

with

For the abscissa value and the ordinate value of the image pixel point coordinates corrected by the distortion coefficient R of the image to be projected,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

Optionally, the motion compensation unit 308 in the apparatus may further include a first motion compensation subunit, configured to perform motion compensation on the distortion corrected reference image by using the following formula:

Wherein said

with

An abscissa value and an ordinate value of the image pixel point coordinates corrected according to the distortion coefficient R for the reference image,

with

The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

The parameter M is compensated for the motion.

In the embodiment of the present invention, considering the complexity of the projection environment, the projection device may be moved or shaken, thereby causing the projection picture to be shaken. In the embodiment of the present invention, the most accurate motion compensation is obtained by combining the acceleration sensor and the motion estimation. The parameter M performs inverse motion compensation on the corrected reference image and the image to be projected corrected by the distortion, thereby ensuring that the output image is not shaken in the case where the projection environment is complicated.

On the basis of the above embodiments, the projected image projected after the distortion correction processing and the de-shake processing is an image without distortion and no jitter, which is suitable for human eyes to view, but if the projection device moves vertically with the projection plane, Still taking a wearable projection device as an example, when the human body carries the wearable projection device vertically moving toward the projection plane, the image is not distorted and shaken, but when the human body is close to the projection plane, the projected image is for the human eye. Too large, and when the projection plane of the human body is far away, the projected image is too large for the human eye. Therefore, on the basis of the above-described embodiment of distortion correction and debounce, the image identification point acquisition unit 302 can also acquire The distance D of the projection device to the projection surface; the device of the embodiment of the present invention may further include: a focus adjustment unit 309, configured to adjust the focal length of the projection device according to the distance D of the projection device acquired by the image identification point acquisition unit to the projection surface, specifically by the following Formula implementation:

Wherein the s ₀ is a liquid crystal size of the projection device, and the s ₁ is an empirical reference value of the image size,

That is, the z value in the three-dimensional coordinate values of the N secondary image identification points is taken and the z values are averaged to obtain the distance D. The Z value in the three-dimensional coordinate value of the secondary image identification point is acquired by the stereoscopic vision processing sub-unit 302a.

Specifically, in the embodiment of the present invention, the stereo vision processing sub-unit 302a may be two imaging devices to form a binocular stereo vision system; the two imaging devices transmit the secondary imaging images formed by the captured projection images. For the computing device, the computing device processes the secondary imaging image transmitted by the two imaging devices by the binocular stereo vision method. In a specific implementation process, the processing step is: extracting the secondary image identifier in the secondary imaging image The three-dimensional parameters of the point.

Further, the binocular stereo vision system may be a binocular stereo vision system with parallel optical axes, in which there may be two cameras for taking in a projected image, and the computing device acquires a secondary image from the camera and Extracting the three-dimensional parameter of the secondary image identification point in the secondary imaging image, that is, the three-dimensional coordinate value of the secondary image identification point, as follows: the distance between the centers of the two camera lenses is e, and the focal lengths of the two cameras are f, The midpoint of the center line of the two camera lenses is the origin. The center of the lens of the camera 1 is directed to the X axis of the lens center of the camera 2, the optical axis of the parallel camera is the Z axis, and the XYZ axis direction conforms to the right hand rule. The coordinate system is established at the coordinate. The coordinates of a point (x, y, z) in the system can be calculated by the following formula:

Wherein (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are (x, y, z) imaging positions of the imaging surface of the camera 1 and the camera ₂ , respectively, in the stereoscopic vision processing system. In this way, we can find the position of all the secondary image identification points in three-dimensional space, with N secondary image identification points, the coordinates are {(x ¹ , y ¹ , z ¹ ), (x ² , y ² , z ² ),...,(x ^N ,y ^N ,z ^N )}.

Based on the Z coordinate of these points, the approximate distance of the projection surface from the projector can be estimated:

The focal length of the visible light projector is automatically adjusted according to the distance d, so that the image size projected by the visible light projector onto the projection surface is suitable for wearing and viewing. The following table is a reference value of the projection distance and the projected image size obtained empirically.

Table 1. Reference values for projection distance and projected image size

After obtaining the projection distance d (unit: m), the focal length f (unit: m) that the projector needs to set can be obtained by the following formula:

Where s ₀ is the projector liquid crystal size (unit: inch), s ₁ is the size (in inches) of the projected image obtained according to the distance d query table 1, the value of f is limited by hardware, must be located in [f Between _min , f _max ], that is, the actual value is max(min(f, f _max ), f _min ). In order to avoid the focal length changing due to the change of the distance, it can be specified that the new focal length calculation is performed only when the absolute value of the difference between the current distance and the previous adjustment of the focal length is greater than a given threshold (for example, 0.5 m). With settings.

In the embodiment of the present invention, on the basis of the foregoing embodiment, the distance d between the projection surface and the imaging device is obtained by averaging the Z coordinate values of the secondary image identification points in the three-dimensional space, and automatically according to the distance d. Adjusting the focal length of the projection device, so that the size of the projected image is correspondingly scaled according to the vertical movement of the human body relative to the projection plane, so that the scaled projected image size is suitable for human eyes to watch, thereby enhancing the user experience.

Claims (28)

1. A processing method based on a projected image, comprising the following method:

   Obtaining a secondary image of the projection image on the projection surface in the imaging device, the projection image being an image formed by the projection device projecting the preset reference image onto the projection surface, the preset reference image Providing a reference identification point, wherein the reference identification point is formed to form an image identification point in the projection image; and the image point formed by the primary image identification point in the secondary imaging image is a secondary image identifier point;

   Obtaining a secondary image identification point in the secondary imaging image;

   Obtaining a distortion coefficient according to a change of a position parameter of the secondary image identification point with respect to a position parameter of the reference identifier; a position parameter of the secondary image identification point is the secondary image identification point in the second a position parameter in the image, the position parameter of the reference mark is a position parameter of the reference mark in the reference image;

   Distortion correction is performed on the image to be projected according to the distortion coefficient.
2. The method according to claim 1, wherein the preset reference image is a reference infrared grid line, and the reference marker point is an intersection of grid lines in different directions in the reference infrared grid line; Correspondingly, the projection device is an infrared projection device, and the imaging device is an infrared filter camera device;

   The acquiring the secondary image of the projection image on the projection surface in the imaging device comprises:

   Obtaining a secondary infrared grid line formed by the infrared filter camera by the projection image on the projection surface;

   The acquiring the secondary image identification points in the secondary imaging image includes:

   Obtaining intersections of grid lines in different directions in the secondary infrared grid lines, the different directions of the network The intersection of the grid lines is the secondary image identification point.
3. The method according to claim 1 or 2, wherein said positional parameter in said secondary imaged image according to said secondary image identification point is in said reference image relative to said reference identification point Obtaining the distortion coefficient by the change of the position parameter specifically includes: obtaining the distortion coefficient by using the following formula

   

   

   

   Where R is the distortion coefficient,

   

   a matrix formed in the affine transformation of the reference image, wherein r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are parameters in an affine change, and the x _i and the y _i are And the (x _i , y _i ) is a coordinate value of the secondary image identification point; the x _i ′ and the horizontal coordinate value and the ordinate value in the secondary image. The y _i ' is an abscissa value and an ordinate value of the reference identification point in the reference image, and the (x _i ', y _i ') is a coordinate value of a reference identification point; the affine The transformation process performs a linear transformation of the parameters r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ and the secondary image identification point (x _i , y _i ), and is coupled with the translation parameter r ₁₃ , r ₂₃ Reference point (x _i ', y _i '); formula (2) is a system of linear equations formed by the transformation of equation (1), and the affine transformation parameter r ₁₁ , r is calculated by the formula (2) ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ values to obtain the affine transformation distortion coefficient R.
4. The method according to claim 3, wherein the correcting the image to be projected according to the distortion coefficient comprises:

   The distortion correction is performed on the projected image by the following formula:

   

   Where R is the distortion coefficient,

   

   with

   

   Image pixel coordinate point for correcting the image to be projected according to the distortion coefficient R

   

   The abscissa value and the ordinate value, the x _i and y _i are the abscissa value and the ordinate value of the original pixel coordinate point (x _i , y _i ) of the image to be projected.
5. The method according to any one of claims 1 to 4, wherein after the distortion coefficient is obtained based on a change in a position parameter of the secondary image identification point based on a position parameter of the reference identification, The method further includes: correcting the reference image according to the distortion coefficient;

   The distortion correction is performed on the reference image by the following formula:

   

   Where R is the distortion coefficient,

   

   with

   

   Image pixel coordinate point for correcting the reference image according to the distortion coefficient R

   

   An abscissa value and an ordinate value, the x _k and y _k being an abscissa value and an ordinate value of the original pixel coordinate point (x _k , y _k ) of the reference image.
6. The method according to claim 5, wherein after the correcting the reference image according to the distortion coefficient, the method further comprises:

   Obtaining a motion parameter of the projection device when projecting the reference image onto the projection surface to form a projected image;

   Obtaining a motion compensation parameter according to the motion parameter;

   Performing inverse motion compensation on the distortion-corrected image according to the motion compensation parameter; the distortion-corrected image includes a corrected image to be projected, or a corrected reference image.
7. The method according to claim 6, wherein before the obtaining the motion compensation parameter according to the motion parameter, the method further comprises:

   Performing motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

   The obtaining motion compensation parameters according to the motion parameter includes:

   A motion compensation parameter is acquired in conjunction with the motion parameter and the preliminary motion compensation value.
8. The method of claim 6 wherein said obtaining motion compensation parameters based on said motion parameters comprises:

   Motion estimation is performed on the secondary imaging image in conjunction with the motion parameter to obtain a motion compensation parameter.
9. The method according to claim 7, wherein the obtaining the motion compensation parameter in combination with the motion parameter and the preliminary motion compensation value comprises:

   The motion parameter is a first motion compensation value directly acquired by the acceleration sensor;

   Performing motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

   The first motion compensation value and the preliminary motion compensation value are weighted and averaged to obtain a motion compensation parameter, which is specifically implemented by the following formula:

   M=α*M ₁ +(1-α)*M ₂ (5)

   Where M is a motion compensation parameter, M ₁ is a preliminary motion compensation value, and M ₂ is a first motion compensation value,

   

   Said

   

   Said

   

   for

   

   Said

   

   for

   

   The tx _i and ty _i are positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image, and the N is the secondary image identifier in each frame. Number of points;

   

   for

   

   Said

   

   for

   

   The tx _j and ty _j are positional differences between the abscissa and the ordinate of the reference mark in the reference image in the adjacent two frames, and the H is the number of the reference mark points in each frame; An empirical value of the specific gravity is taken between the M ₁ and the M ₂ , and the value of α is in the range of 0 ≤ α ≤ 1.
10. The method according to claim 8, wherein the motion parameter is a component value u and v of the displacement of the projection device sensed by the acceleration sensor in the X-axis and Y-axis directions, the combination The motion parameter performs motion estimation on the secondary imaging image to obtain a motion compensation parameter M, which is specifically implemented by the following formula:

    

    

    

    Where tx _i and ty _i are the positional differences between the abscissa of the secondary image identification point and the ordinate in the adjacent two frames in the secondary imaging image, and the N is the second in the secondary imaging image of each frame The number of points of the secondary image identification point, where u and v are the motion parameters, that is, the component values of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, β is

    

    with

    

    The empirical value of the ratio between the values, the value of β is 0 ≤ β ≤ 1.
11. The method according to any one of claims 6 to 10, wherein the motion-compensated image to be projected is subjected to motion compensation according to the motion compensation parameter, which is specifically implemented by the following formula:

    

    Wherein said

    

    with

    

    For the abscissa value and the ordinate value of the image pixel point coordinates corrected by the distortion coefficient R of the image to be projected,

    

    with

    

    The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

    

    The parameter M is compensated for the motion.
12. The method according to any one of claims 6 to 11, wherein the distortion-corrected reference image is motion-compensated according to the motion compensation parameter M, specifically by the following formula achieve:

    

    Wherein said

    

    with

    

    An abscissa value and an ordinate value of the image pixel point coordinates corrected according to the distortion coefficient R for the reference image,

    

    with

    

    The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

    

    The parameter M is compensated for the motion.
13. The method according to any one of claims 1 to 12, wherein the imaging device is at least two to constitute a stereoscopic vision system;

    The acquiring the secondary image identification point in the secondary imaging image specifically includes:

    Obtaining a three-dimensional parameter in the projected image by the stereo vision system; and fitting the three-dimensional parameter into a plane, vertically mapping the three-dimensional parameter to the plane, to obtain a two-dimensional coordinate point, the two-dimensional coordinate point Identify points for the secondary image.
14. The method according to claim 13, wherein after the performing distortion correction on the image to be projected according to the distortion coefficient, the method further comprises: adjusting a focal length of the projection device according to a distance D from the projection device to the projection surface, specifically It is implemented by the following formula:

    

    Wherein the s ₀ is a liquid crystal size of the projection device, and the s ₁ is an empirical reference value of the image size,

    

    That is, the z value in the three-dimensional coordinate values of the N secondary image identification points is taken and the z values are averaged to obtain the distance D.
15. A processing device based on a projected image, characterized in that the device comprises:

    An imaging acquisition unit for acquiring a secondary image of a projection image on a projection surface in an imaging device An image obtained by projecting a preset reference image onto the projection surface by a projection device, wherein the preset reference image is provided with a reference identification point, and the reference identification point is projected Forming an image identification point in the projection image; the image point formed by the primary image identification point in the secondary imaging image is a secondary image identification point;

    a marker point acquiring unit, configured to acquire a secondary image identification point in the secondary imaging image;

    a distortion coefficient acquiring unit, configured to acquire a distortion coefficient according to a change of a position parameter of the secondary image identification point with respect to a position parameter of the reference identifier; a position parameter of the secondary image identification point is the secondary image Identifying a position parameter of the point in the secondary imaging image, where the position parameter of the reference identification point is a position parameter of the reference identification point in the reference image;

    The image to be projected correction unit is configured to perform distortion correction on the image to be projected according to the distortion coefficient R.
16. The device according to claim 15, wherein the preset reference image is a reference infrared grid line, and the reference marker point is an intersection of grid lines in different directions in the reference infrared grid line;

    The imaging acquiring unit is configured to acquire a secondary infrared grid line formed by the infrared image camera by a projection image on the projection surface;

    The image identification point acquiring unit is configured to acquire intersections of grid lines in different directions in the secondary infrared grid lines, and intersection points of the grid lines in the different directions are the secondary image identification points.
17. The apparatus according to claim 15 or 16, wherein said distortion coefficient acquisition unit is configured to calculate said distortion coefficient by the following formula

    

    

    

    Where R is the distortion coefficient,

    

    a matrix formed in the affine transformation of the reference image, wherein r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ are parameters in an affine change, and the x _i and the y _i are And the (x _i , y _i ) is a coordinate value of the secondary image identification point; the x _i ′ and the horizontal coordinate value and the ordinate value in the secondary image. The y _i ' is an abscissa value and an ordinate value of the reference identification point in the reference image, and the (x _i ', y _i ') is a coordinate value of a reference identification point; the affine The transformation process performs a linear transformation of the parameters r ₁₁ , r ₁₂ , r ₂₁ , r ₂₂ and the secondary image identification point (x _i , y _i ), and is coupled with the translation parameter r ₁₃ , r ₂₃ The reference identification point (x _i ', y _i ') is described; the formula (2) is a linear equation group composed of the transformation of the formula (1), and the affine transformation distortion coefficient R is calculated by the formula (2).
18. The apparatus according to claim 17, wherein the image to be projected correction unit is configured to perform distortion correction on the image to be projected by the following formula:

    

    Where R is the distortion coefficient,

    

    with

    

    Image pixel coordinate point for correcting the image to be projected according to the distortion coefficient R

    

    The abscissa value and the ordinate value, the x _i and y _i are the abscissa value and the ordinate value of the original pixel coordinate point (x _i , y _i ) of the image to be projected.
19. The apparatus according to any one of claims 15 to 18, further comprising: a reference image correction unit configured to correct the reference image according to the distortion coefficient R; the reference image correction The unit is configured to perform distortion correction on the reference image by the following formula:

    

    Where R is the distortion coefficient,

    

    with

    

    Image pixel coordinate point for correcting the reference image according to the distortion coefficient R

    

    An abscissa value and an ordinate value, the x _k and y _k being an abscissa value and an ordinate value of the original pixel coordinate point (x _k , y _k ) of the reference image.
20. The device of claim 19, wherein the device further comprises:

    a motion parameter acquisition unit, configured to acquire a motion parameter of the projection device when projecting the reference image onto the projection surface to form a projection image;

    a compensation parameter obtaining unit, configured to obtain a motion compensation parameter according to the motion parameter;

    And a motion compensation unit, configured to perform inverse motion compensation on the distortion-corrected image according to the motion compensation parameter; the distortion-corrected image includes a corrected image to be projected, or a corrected reference image.
21. The device of claim 20 wherein said device comprises:

    a motion compensation estimating unit, configured to perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value;

    The compensation parameter acquisition unit is configured to acquire a motion compensation parameter according to the motion parameter acquired by the motion parameter acquisition unit and the preliminary motion compensation value acquired by the motion complement estimation unit.
22. The apparatus according to claim 20, wherein the compensation parameter acquisition unit is configured to perform motion estimation on the secondary imaging image in combination with the motion parameter acquired by the motion parameter acquisition unit to obtain a motion compensation parameter.
23. The device according to claim 21, wherein the motion parameter acquiring unit is configured to directly acquire a first motion compensation value according to the acceleration sensor, and use the first motion compensation parameter as the motion parameter;

    The motion compensation estimating unit is configured to perform motion estimation on the secondary imaging image to obtain a preliminary motion compensation value, and perform a first motion compensation value obtained by the motion parameter acquiring unit and the obtained preliminary motion compensation value. The weighted average obtains the motion compensation parameter, which is calculated and obtained by the following formula. The motion compensation parameter:

    M=α*M ₁ +(1-α)*M ₂ (5)

    Where M is a motion compensation parameter, M ₁ is a preliminary motion compensation value, and M ₂ is a first motion compensation value,

    

    Said

    

    Said

    

    for

    

    Said

    

    for

    

    The tx _i and ty _i are positional differences between the abscissa and the ordinate of the secondary image identification point in the secondary image, and the N is the secondary image identifier in each frame. Number of points;

    

    for

    

    Said

    

    for

    

    The tx _j and ty _j are positional differences between the abscissa and the ordinate of the reference mark in the reference image in the adjacent two frames, and the H is the number of the reference mark points in each frame; An empirical value of the specific gravity is taken between the M ₁ and the M ₂ , and the value of α is in the range of 0 ≤ α ≤ 1.
24. The apparatus according to claim 22, wherein the motion parameter acquisition unit is configured to acquire component values u and v of the displacement of the projection device sensed by the acceleration sensor in the X-axis and the Y-axis direction, Said u and v as motion parameters;

    The compensation parameter acquiring unit is specifically configured to perform motion estimation on the secondary imaging image in combination with the motion parameter acquired by the motion parameter acquiring unit to obtain a motion compensation parameter M, and specifically calculate the motion compensation parameter M by using the following formula:

    

    

    

    Where tx _i and ty _i are the positional differences between the abscissa of the secondary image identification point and the ordinate in the adjacent two frames in the secondary imaging image, and the N is the second in the secondary imaging image of each frame The number of points of the secondary image identification point, where u and v are the motion parameters, that is, the component values of the displacement of the projection device in the X and Y directions sensed by the acceleration sensor, β is

    

    with

    

    The empirical value of the ratio between the values, the value of β is 0 ≤ β ≤ 1.
25. The apparatus according to any one of claims 20 to 24, wherein the motion compensation unit comprises: specifically for performing motion compensation on the distortion-corrected image to be projected by the following formula:

    

    Wherein said

    

    with

    

    For the abscissa value and the ordinate value of the image pixel point coordinates corrected by the distortion coefficient R of the image to be projected,

    

    with

    

    The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

    

    The parameter M is compensated for the motion.
26. The apparatus according to any one of claims 20 to 25, wherein the motion compensation unit is further configured to perform motion compensation on the distortion-corrected reference image by the following formula:

    

    Wherein said

    

    with

    

    An abscissa value and an ordinate value of the image pixel point coordinates corrected according to the distortion coefficient R for the reference image,

    

    with

    

    The abscissa value and the ordinate value of the image pixel point coordinates after the inverse motion compensation of the image pixel point after the distortion correction is performed, wherein

    

    The parameter M is compensated for the motion.
27. Apparatus according to any one of claims 15 to 26, wherein said image identification The point acquisition unit also includes:

    a stereo vision processing sub-unit for acquiring three-dimensional parameters in the projected image;

    a marker point acquisition subunit, configured to fit the three-dimensional parameters acquired by the stereo vision processing subunit into a plane, and vertically map the three-dimensional parameters to the plane to obtain two-dimensional coordinate points, the two-dimensional coordinates The point is the secondary image identification point.
28. The device of claim 27, wherein the device further comprises:

    The focal length adjusting unit is configured to adjust the focal length of the projection device according to the distance D from the projection device to the projection surface, which is specifically implemented by the following formula:

    

    

    The distance D is averaged.

Images