CA2225400A1 - Method and system for image combination using a parallax-based technique - Google Patents

Abstract

A system for generating three-dimensional (3D) mosaics from a plurality of input images representing an imaged scene. The plurality input images contain at least two images of a single scene, where at least two of the images have overlapping regions. The system combines the images using a parallax-based approach that generates a 3D mosaic comprising an image mosaic representing a panoramic view of the scene and a shape mosaic representing the three-dimensional geometry of the scene. Specifically, in one embodiment, the system registers the input images along a parametric surface within the imaged scene and derives translation vectors useful in aligning the images into a two-dimensional image mosaic. Once registered, the system generates a shape mosaic representing objects within the scene.

Description

W O 97/01135 PCTrUS96/1048 METHOD AND ~Y~ ;~ FOR IMAGE COMBINATION USING
A PARALIl~X-BASED TECHNIQUE

~ The invention relates lto image proce.sqing systems, and more5 particularly, the invention relates to an image proce.qqing system that comhin~s multiple images into a mo~3~ic using a parallax-based technique.
Until recently, image proce.q.qin~ systems have generally processed images, such as frames of video, still photographs, and the like, on an individual, image-by-image basis. Each individufJl frame or photograph is 10 typically processed by filt~rin~, w~ g, and applying various parametric transformations. In order to form a panoramic view of the scene, the individual images are comhine-7 to form a two-~7imenqion~7l mosf7ic, i.e., an image that cont~inq a plurality of individual images. ~nnitinnAl image processing is performed on the mosaic to ensure that the seams between 15 the images are invisible such that the mos~7ic looks like a single large image.
The Ali~nment of the images and the z7n7niti~)nAl proce~ing to remove seams is typically slccomrli~he-~7 manually by a te~hni~ if7n 77~ing a computer workstSltion, i.e., the image ~ nment and comhinf7tion processes are computer aided. In such computer aided image proce~~ing systems, the 20 technician mAnll~7lly selects processed images, manually aligns those images, and a computer applie-3 various image c~mhining processes to the images to remove any seams or gaps between the images. ManiplllAtion of the images is typically accomrli.~hed using various computer input devices such as a mouse, trAl~.khAll, keyboard and the like. Since mAnlls~l mosaic 25 generation is costly, those skilled in the art have developed automated systems for generating image mosaics.
In ~llt,omAted systems for constructing mosaics, the information vwithin a mosaic is generally e~pressed as two-~limen.cioT-Al motion fields.
The motion is represented as a planar motion field, e.g, an afflne or 30 projective motion field. Such a system is ~li.cl lose-l in U.S. patent applicAt.ion serial nnmber 08/339,491, ~ntitlerl "Mosaic Based Image Proce.~.~ing System", filed November 14, 1994 and herein incorporated by reference. The image proces.~ing approach ~ rloserl in the '491 application automatically comhine.s mlll~iple image firames into one or more 35 two--limen~ nAl mosaics. Huwev~ that system does not account for parallax motion that may c ause errors in the displAcPm~nt fields representing motion in the mo~

CA 0222~400 1997-12-19 In other types of image procçqqing systems, multiple images are analyzed in order to recover photogr~mm.qtic inform~ti~ such as relative orientation estim~tion, range map rec..ve~.y and the like without generating a mosaic. These image analysis techniques assume that the internal camera parameters (e.g., focal length pixel resolution, aspect ratio, and image center) are known. In ~lltom~te~l image procç.qqing systems that use ~lignment and photogrammetry, the ~ nm~nt and photogr~mm~t.ic process involves two steps: (1) est~hli-qhin~ corresp~n~nce between pixels within various images via some form of area- or feature-based m~tching scheme, and (2) analyzing pixel displ~c~mqnt in order to recover t_ree-dimensional scene inform~ti--n The invention is a method of procç.q.qing a plurality of images to generate a three-dimensional (3D) mosaic of a scene comprising the steps of providing a plurality of images of the scene registering said images to construct said 3D mosaic cont~ining an image mosaic and a shape ~nosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a 3D geometry of the scene.
The invention is also a system for generating a 3D mosaic of a scene from a plurality of images of the scene, comprising means for storing said plurality of images and a registration processor, crnnected to said storing means, for registering said images to construct said 3D mosaic cont~ining an image mosAic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a 3D
geometry of the scene.
The te~rhing.q of the invention can be readily Imderstood by c-)nqi-l~ring the following ~ ile(l description in conjunction with the ~ccomr~nying drawings, in which:
FIG. 1 depicts a block diagram of an im~ing system incorporating an image proce.sqing system of the invention;
FIG. 2 schematically depicts the input images and output mosaics of the system of FIG. 1;
FIG. 3 is a geometric representation of the r~l~tion~qhip amongst a reference image generated by a reference camera, an inspection image generated by an inspection camera, and an albi~lal ~ parametric surface within a scene imaged by the cameras;
FIG. 4 is a flow chart of a P-then-P routine for registering images and extracting parallax inform~tion from the registered images;

CA 0222~400 1997-12-19 FIG. 5 is a flow chart of a P-and-P routine for registering images and extracting parallax inform~tion from the registered images;
FIG. 6 is a functional block diagram of an image processing system of the invention;
FIG. 7 is a flow chart of a pose estim~tion routine;
~ FIG. 8 is a flow chart of a 3D corrected mosaic construction routine;
FIG. 9 is a two--lim~n.qi-n~l geometric represent~tior~ of the plane OMP of FIG. 3 where the scene contains an object that occludes points within the image; and FIG. 10 depicts an experim~nt~1 set-up for estim~t~ing h~ight~ of objects within a scene using the system of the invention .
FIG. 11 depicts a block diagram of an application for the inventive system that synth~ es a new view of existing 3D mos~ic~;
An image proc~qing system that combines a plurality of images representing an imaged scene to form a 3D mosaic, where the 3D mosaic cont~in~ an image mosaic representing a panoramic view of the scene and a shape mosaic repr~s~nting the 3D geometry of the scene is first described.
The shape mosaic ~lefine~q a r~l~tionqhip between any two images by a motion field that is decomposed into two-rlim.onqion~1 image motion of a two-dimçn.cion~1, par~metric surface and a residual parallax field. Although many techniques may be useful in generating the motion fields and the parametric tr~n.ql~tion parameters, the following disclosure discusses two illustrative processes. The first process, known as plane-then-parallax (P-then-P), initi~11y registers the images along a parametric surface (plane) in the scene and then determines a parallax field representing the 3D
geometry of the scene. I'he second illustrative process, known as plane-and-parallax (P-and-P), simultaneously registers the images and determines the parallax field. With either process, the results of registration are trAn.q1s3t.ion parameters for achieving image ~1ignm~nt along the parametric surface, a parallax field representing the 3D geometry (motion) of the scene with respect to the parametric surface, and a planar motion field representing motion within the parametric surface. These results can be used to combine the input images to form a 3D mosaic.
Image motion of a parametric surface is, in essence, a conventional representation of a 2D mosaic. Motion of the parametric surface is generally expressed as a parametric motion field that is estim~ted using one of the many available techniques for directly estim~ting two-llim~nqional motion fields. Generally speaking, a direct approach is sufficient for ~ ning CA 0222~400 1997-12-19 W O 97/01135 _4 PCTAJS96/10485 and c~nhining a plurality of images to form a two--limen.qionAl mosaic.
Such a two--lim~n-qinnAl mosaic represents an Alignm~nt of a two-dimensional parametric surface within a scene captured by the image sequence. This parametric surface can either be an actual surface in the scene within which lie most objects of the scene or the parametric surface can be a virtual surface that is arbitrarily selected within the scene. AU
objects within the scene generate what is known as parallax motion as a camera moves with respect to the parametric surface. This parametric motion is repre.s~nte~l by a parallax motion field (also referred to herein as aparallax field). The parallax field has value for objects within the scene that do not lie in the plane of the surface. Although objects lying in the plane of the surface are represented in the parallax field, those objects have zero parallax. More particularly, the parallax field represents the objects that lie in front of and behind the parametric surface and the distance ~height) of these objects from the surface, i.e., the 3D geometry of the scene. As such, using the parallax field in comhinAti--n with the parametric surface and its planar motion field, the system can generate a 3D reconstruction of the scene up to an arbitrary colline~tio~ If camera calibration parameters such as focal length and optical center are known, then this 3D
reconstruction of the scene is Euclidean.
FIG. 1 depicts a block diagram of the image proce.qqing system 100 as it is used to generate 3D mosaics from a plurality of images. The system is, in general, a general purpose computer that is programmed to function as an image proces.qing system as described herein. The system further 25 . co~t~inq one or more cameras 104n that image a scene 102. In the illustrative system two cameras, cameras 104l and 1042. are shown. Each c~mera, for simplicity, is assumed to be digital video camera that generates a series of frames of rligiti~etl video informAtion Alternatively, the cameras could be still cameras, conventional video cameras, or some other form of imAging sensor such as an infrared sensor, an ultrasonic sensor, and the like, whose output signal is separately ~ligiti~ell before the signal is used asan input to the system 100. In any event, each c~mera 104l and 104z generates an image having a distinct view of the scene. Specifically, the images could be selected frames from each camera im~in~ a different view of the scene or the images could be a series of frames from a single camera as the camera pans across the scene. In either case, the input signal to the image proce~ing system of the invention is at least two images taken from different viewpoints of a single scene. Each of the images partially overlaps CA 0222F,400 1997- 12- 19 the scene depicted in at least one other image. The system 100 combines the images into a 3D mosaic and presents the mosaic to an output device 106. The output device could be a video compression system, a video storage and retrieval ~,y~lelll, or some other application for the 3D mos~ic FIG. 2 st~hem~tically depicts the input images 200n to the system 100 and the output 3D mosaic 202 generated by that system in response to the input images. The input images, as me~tionetl above, are a series of images of a scene, where each image depicts the scene from a Llr~ t viewpoint. The system 100 a~igns the images and c~mhines them to form an image mosaic 204, e.g, a two-tlim~.n.~i-n~l mosaic having the images ~ n~l along an arbitrary parametric surface ~ n-ling through all the images. ~ ning the images to form the image mosaic requires both the parametric tr~n.~l~tion parameters and the planar motion field. In ~ n to the image mosaic, the system generates a shape mosaic 206 that contains the motion field that relates the 3D objects within the images to one another and to the parametric surface. The shape mosaic cont~in~ a parallax motion field 208. The planar motion field represents motion within the parametric surface that appears in the images from image to image, while the parallax flow field represents motion due to parallax of 3D objects in the scene with respect to the parametric surface.
A. Determinin~ A R~ Parallax F~eld C--n.qi~lPr two camera views, one denoted as the "lef~rellce" camera and the other denoted the "inspection" camera (e.g, respectively cameras 1041 and 1042 of FIG. 1). In general, the image procç.~ing system Z5 maps any 3D (3D) point Pl in t;he reference camera coordinate system to a 3D point P2 in the inspection camera coordinate system using a rigid body transformation represented by Equation 1.

P2 = R(P, ) + T2 = R(PI--T, ) The mapping vector is represented by a rotation (R) followed by a tr~n.qk3tion (T2) or by a tr~n.~ tion (Tl)followed by a rotation (R). Using perspective projection, the image coor(1in~tes (x,y) of a projected point P are given by the vector p of Equation 2.
x y = f p (2) CA 0222~400 1997-12-19 W O 97/01135 PCT~US96/10485 where f is the focal length of the camera.
FIG. 3 is a geometric representation of the r~l~ti- nqhip amongst a reference image 302 generated by the reference camera, an inspection 5 image 304 generated by the inspection camera, and an arbitrary parametric surface 300 within the imaged scene. Let S denote the surface of interest (a real or virtual surface 300), P denotes an environrnent~l point (e.g., a location of an object) within the scene that is not located on S, and Oand M denote the center locations (focal points) of each camera. The image 10 of P on the image 302 is p. Let the ray MP intersect the surface S at location Q. A conventional W~illg process, used to align the images 302 and 304 by ~ligning all points on the surface S, warps p', the image of P on the image 304, to q, the image of Q on the image 302. Therefore, the residual parallax vector is pq, which is the image of line PQ. It is 15 imme~ ~ly obvious from the figure that vector pq lies on the plane OMP, which is the epipolar plane p~qqing through p. Since such a vector is generated for any point P in the scene, it can be said that the collection of all parallax vectors forms a parallax displ~c~ment field. Since the parallax displ~cemçnt vector associated with each image point lies along the 20 epipolar plane associated with that image, the vector is referred to as an epipolar field. This field has a radial structure, each vector appearing to çm~n~te from a common origin in the image dubbed the "epipole" (alias focus of ~nqion (FOE)). In FIG. 3 the epipole is located at point "t".
From FIG. 3, it is obvious that the epipole t lies at the intersection of the 25 line OM with the image plane 302. The parallax displ~cement field is also referred to herein simply as a parallax field or parallax motion field.
In det~l...i..i..g the r~qi(lll~l parallax inform~tio~ (e.g, parallax field), it is assumed that the two images are ~lignetl (registered) along the parametric surface using a conventional parametric motion est~im~tion 30 method. These ~liFnm~snt methods are also known in the art as "hierarchical direct methods" of ~ nment or registration. One such method is described in commonly ~q.ci~nerl U.S. patent application serial number 08/339,491 filed November 14, 1994 and herein incorporated by reference.
As shall be discussed in detail below, once the inventive system determines 35 the transform~tion and planar motion field for ~ligninF the two images along the parametric surface, the system d~te~ es the r~.qi~ l parallax information representing the h~ight above or below, the parametric surface of objects within the scene.

CA 0222~400 1997-12-19 W O 97/01135 7 PCT~US96/10485 B. Re~istration of Images Using the general prin( iple.s discussed above to accurately represent a 3D scene, the system must recover both the planar and parallax motions as well as the tr~n~l~ti~n parameters for ~ ning the images.
5 Illustratively, the system u~,es two terhniques either separately, or in sequence, to determine the transformation parameters and the motions within the images. The first te-~hnique is a "seqll~nti~l registration"
approach, in which a plane (parametric surface) within the scene that is imaged by both cameras is first registered using an eight parameter planar 10 transformation. The re~ l parallax motion is then estim~te~l using a separate, seqllenti~lly ~ecllte~l step. The second terhnique is a "simultaneous registration" approach, in which the system simultaneously estimates the parametric transformation as well as the planar and parallax motion fields.
i. Seqll.qnti~l Re~i.c~ration FIG. 4 depicts a flow chart of a routine 400 executed by the system to perform seqll~nti~l registration and determine the parallax field. To register a plane within the scene, the system uses a hierarchical direct registration terhnique. This technique uses a planar flow field model for 20 motion within a plane. Once a plurality of images are input to the system at step 402, the routine performs two seql1~nti~1 steps to determine the tr~n~l~tion parameters and the motion fields; namely, at step 404, the routine derives the planar motion fields and, at step 406, the routine estimates both the tr~n~l~tion parameters and the parallax field. The 25 resulting output 408 from the routine is the r~l~t.i- n~l inform~t.ion regarding the input images, e.g, the tr~n.cl~tion parameters for ~ ning the images along a plane and the planar and parallax motion fields representing the 3D
geometry of the scene.
Specifically, the total motion vector of a point in the scene is 30 expressed as the sum of the motion vectors due to the planar surface motion (up,vp) and the residual parallax motion (ur,vr). As such, this motion vector is represented as Equation 3.

(u,v) = (up~uv) + (ur,vr) (3) Further, the motion field of a planar surface (two-(limen~ion~l) is represented as:

W O 97/01135 8 PCT~US96/10485 up(x) = p~X + P2Y + p5+ p7X + p8Xy Vp(X) = p3X+P4Y+P6 +P7XY+P8Y
(4) where N2,~T2,~--N2ZT2Z
PlN2yT2l--QZ
P2N2~T2,.--QZ
P3 N2yT2y--N2ZT2Z
= f(N2ZT2~ - Qy) f(N2ZT2.~--n~
P6 f (Q,--N2xT2z) P8 f (--Ql--N2!.T2z ) (5) T2X~ T2y~ and T2z denotes the tr~nql~qtion vector between camera views, S2,~, Qy and Qz denotes the angular-velocity vector, f denotes the focal length of the camera. and N2X~ N2y~ and N2z denotes the normal vector to the planar surface from a camera center. The rç.~ l parallax vector is further 10 represented as:
ur(x) = ~(fT2l--xT2z ) (6) v,(X) = r(fT2!--YT2z ) Where the parallax magnitude field is represented by Equation 7.
y = H/PzTl (7) where H is the perpen(lic3ll~r distance of the point of interest from the plane, and P~ is the depth of the point of interest (also referred to in the art20 as range). Tl is the perpendicular fli~t~nce from the center of the first camera (reference) to the plane, and f is the focal length of that camera. At each point in the image, the parallax m~Enit~ e field ~ varies directly with the height of the corresponding 3D point from the reference surface and .

W O 97/01135 9 PCTrUS96/10485 inversely with the depth of the point, i.e., the distance of the point from the camera center.
To determine the total motion field, the seqll~nti~l approach first solves Equation 4 for (up,vp) and then Equation 3 for (ur,vr). To achieve 5 Ali~nment in a coarse-to-fine, iterative m~nn~r, the input images are subsampled to form multi-r~ t;on~l image pyr~mids. Within each level of the pyramid, the measure used as in~ of an image AliFnm~nt match is the sum of the squared di~~ ce (SSD) measure integrated over selected regions of interest on the images. Typically, the system initially selects the 10 entirety of the images as the selected region and, thereafter, re~ iv~ly selects smaller regions until the AliFnm~nt measure is ~ i7ç~ To perfect ~ nm~nt, the Alignm~nt measure is ,..;..;,..i~ç.l with respect to the quadratic flow field parAm~tP!rs (-1efine-1 below). The SSD error measure for F~"gtim~ting the flow field within an image region is:
E(l,U}) = ~(l(X,t)--I(X--u(x),t--1)) (8) where x=(x,y) denotes the spat;al position of a point within an image, I is the multi-resollltionAl ~y~ ..id image intensity, and u(~)=(u(x,y),v(x,y)) denotes 20 the image velocity at a point (x,y) within an image region and {u} denotes the entire motion field within the region. The motion field is modeled by a set of global and local parameters.
To use this technique, the system, at step 410, first constructs a multi-resollltion~l pyramid representation (e.g., Laplacian or Gll~ iAn 25 pyramids) of each of the two input images. Thereafter, at step 412, the routine estimAtes, in a coarse-to-fine mAnn~r~ the motion parameters that align the two images to one another, i.e., although not specifically shown, the routine iterates over the levels of the pyramids to achieve the coarse-to-fine AliFnm~nt Specifically, the routine aligns the images using 30 the fol~ Ui~lg planar motion field computations and ~ ing the SSD at each level of the image pyramids. The routine estimates the eight motion parameters (Pl through P8) and the resulting motion field with reference to a region within a planar surface comprising a substantial number of pixels in the two images (e.g., a "real" or physical surface). In particular, the routine 35 begins with some initial parameter values (typically, zero) and then iterively refines the parameters in order to first minimi7e the SSD error at a coarse image resolution, then ellccç~.~ively at finer image resolutions within CA 0222~400 1997-12-19 W O 97/01135 lo PCTAUS96/10485 the image pyramids. After each step of ~ nm~nt iteration, the transformation based on the current set of parameters is applied to the inspection image in order to reduce the r~qi~llAl displAc~m-?nt between the two images. The reference and inspection images are registered so that the 5 selected image region (e.g., the overlapping region in the physical plane) is Ali~n~l along a "visible" planar surface. The routine queries, at step 414, whether further computational iterations are necessary to achieve ~lignm~nt The decision is based on a comparison of the SSD to a predefined threshold SSD level. If further Ali~nm~qnt is necessary the 10 routine loops to step 412. Once the images are accurately registered, the routine proceeds from step 414 to step 406.
At step 406, the routine uses the planar flow field information to compute the trAn~lAtic-n parameters and the parallax field. At step 418, the value of (up,vp) as expressed in Equation 4 is computed using the 15 estimated values of Pl through P8 computed in step 404. Then, the values of (up,vp) and the expression of (ur,vr) in Equation 6 are substituted into the SSD function (Equation 8). Equation 8 is then ...;..;..~i~ell to solve for the direction trAn.~lAti-n T2(X~y~z) and the parallax vector field ~. The routine iterates from step 420 as the parameters are computed to minimi~e the 20 SSD. Although not ~xpli~itly shown, the routine also iterates through the pyramid levels to achieve sufficient tr~n.~l~tif n parameter accuracy. Once the SSD is minimi~ed to a sufficient level, the routine generates, at step 408, as an output the trAn.~l~tion parameters, the planar motion field, and the parallax motion field. Note that these values are generated llsing Z5 the various levels of image pyramids and, as such, these parameters and motion fields are generated as multi-resollltio~Al pyramids. Thus, the parameter and motion field pyramids can be directly used to produce a multi-resolutional pyramid of the 3D mosaic.
~he system generally uses this seqll~nt;~l registration process to 30 align images that depict a scene col.ts.;..ing a well~ fin~-l planar surface.The result of the process is a set of tr~n.~l~t.ion parameters for Ali~ning the images along the plane to produce a 2D mosaic and a motion fields representing 3D geometry in the scene. In other words, the system generates the parameters used to produce a 3D mosaic using a two step 35 process: registration to a plane (step 404), then determine the parallax inform~tion (step 406). This two step process has been dubbed the plane-then-parallax (P-then-P) method.
ii. Simultaneous Registration W O 97101135 ~ PCTAJS96/10485 FIG. 5 depicts a routine 500 for simultaneously registering two images and generating a parallax field. In the simultaneous registration approach, the system simultaneously solves for both (up,vp) and (ur,vr) in the total motion vector as r~fine~ in Eqll~tinn~ 3 and 6.
Using the simultaneous registration approach, a "real~ planar surface is not necessary; thus, the images can be registered to a ~virtual"
planar surface lying ~ ~;]y within the images. As such, this approach is more flexible than the seqllentiiRl registration approach.
Routine 500 begins in the same m~nn~r as routine 400, in that, steps 402 and 410 input images and then construct multi-resolllt;on~l pyramids thelerl.~ . Thereafter, the routine, in step 504, computes the tr~n.~l~t.ion parameters, the planar motion field and the parallax field. The results are output in step 510.
More specifically, the e2~pres.~iong for (u,v) in Eqll~til-n~ 3 and 4 are substituted into equation 5 to obtain a complete objective filnc~;on The resulting function is then ~ i7e-1, in step 504, with respect to SSD to simultaneously solve for the planar motion par~meters (P1 through P8), direction of tr~n~l~tion T2(y,y,z), and the parallax field ~ at each level of the image pyramid. As such, this process is iterated using the multi-resollltion~l image pyr2lmids in a coarse-to-fine f~Rhion Results obtained at the coarse level of the pyramid are used as an initial estimate for a computation at the ne~t level. At each level of the pyramid, the computation is iterated to minimi7e SSD. Howeve~, the results at each iteration are stored to form a multi-resolutional pyramid of the computation results, i.e., the process fi~rms multi-resollltion~l pyramids for the tr~n~ t.ion parameters and :motion fields. After each iteration through step 506, the estimated motion parameters are n~ e~l, at step 508, such that the planar registration parameters correspond to a virtual plane which gives rise to the smallest parallax field (e.g, the average plane of the 3D scene imaged by the two cameras). The result, generated at step 510, is a set of tr~n.ql~t.ion parameters for Ali~ning the images along the plane to produce a 2D mosaic, a planar motion field representing motion with the plane, and a parallax vector field representing objects in the scene that do not lie in the plane. In other words, the system generates a 3D ~ nment llsing a one step process: simultaneous registration to a plane and deter~nin~tion of the parallax information. This one step process has been dubbed the plane-and-paralla~: (P-and-P) method.
ui. Comhin~tit)n of Seqllenti~l and Simultaneous Image Registration CA 0222~400 1997-12-19 W O 97/01135 PCT~US96/10485 FIG. 6 depicts a filnrtion~l block diagram of the system 100 of the i~lv~ ion . The input images are temporarily stored in image storage 600.
First, the system 100 uses seqllPnti~l registration to register the reference and inspection images and provide an estimate of the parallax field 5 (P-then-P registration processor 602 operating in accordance with routine 400 of FIG. 4). Secondly, the system uses simultaneous registration to provide further image ~ nmPnt and an accurate parallax field (P-and-P registration processor 604 operating in accordance with routine 500 of FIG. 6). If either processor 602 or 604 generates a flow field 10 for the parametric surface that is deemed accurate to within a pretlP.finP-l measure of accuracy (e.g, a minim~l SSD), the system ceases proce.qqing the images and begins generating a 3D mosaic. For a~mrle~ in a scene c-nt~ining simple shaped objects, the P-then-P proc~fiqing may be enough to accurately generate a 3D mosaic of the scene. More compliç~te-1 scenes 15 having many parallax objects may require both forms of image procP.$qing to generate an accurate 3D mosaic. Generally, ~lignment. quality is tested by computing the magnitude of the normal flow field between the inspection and reference images. Regi--n.q in the parametric surface having normal flow above a prelefinP-l threshold (e.g., 0.5) are labeled as lln~ nPIl and 20 further processed by a subsequent processor. If these regions obtain a ;..;...us size, the system deems the images ~ ne~l and the next processor is not executed.
The output of the two processors are the tr~n.ql~ti-)n parameters and the motion fields. A 3D mosaic generator 606 combines the input images with the tr~n.ql~tion parameters and motion fields to produce a 3D mosaic.
As ~PfinP~l above, the 3D mosaic contains an image mosaic and a shape mosaic, where the image mosaic is a panoramic view of the scene represented by the images and the shape mosaic represents the 3D
geometry of the scene within the panoramic view.
The 3D mosaic can then be used in various ~ nqion.q and applications (reference 608) of the basic system discussed above. These P~tPn.qion.q and application of the system are discussed in detail below.
C. ~,~t~nqion.q of the Invention There are a number of optional processes that enh~nce the usefulness of the invention. The f~rst is a pose estimation routine that provides a simple ter-hnique for relating a new image taken from a new viewpoint to an P~i~ting mosaic. The second P~tenqi~n is a te-~hnique for generating a new 3D mosaic by c-)mhining an P~jqtin~ mosaic with a new CA 0222~400 1997-12-19 image of the scene represented by the mosaic. The third e~enqir~n is a technique for detecting and proce.q~ing occlusions v.~ithin a 3D mosaic.
i. Pose Estimation FIG. 7 is a flow chart of a pose esti~n~tion routine 700. Given both lefelellce and inspection images, the system aligns the images and determines a parallax field using the P-then-P process and/or the P-and-P
process as discussed above to form a lefelellce image mosaic and a ~efelellce shape mosaic. The reference mosaics then serve as am initial repres~nt~ti-~n of the 3D scene. The reference image and shape mos~ics are input at step 702 and then collvel led into multi-resollltioI ~l pyramids at step 704. If the reference mosaics are provided as pyramids, then step 704 can be di~l~led. Given a new image of the scene taken from a new viewpoint (step 706) that has been c~ v~lled into an image pyramid at step 708, the routine computes, at step 710, the pose of the new viewpoint with respect to the reference view used to construct the reference mosaics.
The pose of the new image is represented by eleven pose parameters;
namely, eight planar motion parameters (Pl through P8) and three tr~n.ql~t.ion parameters (T2(y,y,z)). To compute the pose parameters, the system again uses the direct lhierarchical te~hnique used above to register the images, and iterated through step 712 until the SSD achieves a pre~l~fine-l value. Specificall~y, given the parallax field y, Equation 8 is i7e~1 using Equation 3 to estimate the eleven pose parameters. As with the registration approaches described above, the new image is f3li~n.oA
using the coarse-to-fine registration process over an image pyr~mid of both the 3D representation of the scene and the new image. The outcome of the pose estimation routine are tr~n~l~tion parameters, a planar motion field, and a parallax motion field for the new image. With these results, the new image can be integrated into the 3D mo~ic as discussed below.
ii. 3D Corrected Mos~ic Generation Given a reference 3D mosaic (e.g., an ~i.cting mosaic) relating a plurality of images of a scene to one another and using pose estimation, the system can update the existing mosaic with new image inform~tion as it becomes available. This process of integrating inform~tion from new images into an existing mos~ic is known as correcting the 3D mosaic.
As discussed above 3D mosaics contain two parts; namely, a mosaic image repres~nting an assemblage of the various images of a scene into a single (real or virtual) camera view, and a parallax map (shape mosaic) corresponding to that view. Note that the parallax map is itself a mosaic CA 0222~400 1997-12-l9 W O 97/01135 -14- PCTAUS96tlO485 produced by arranging the parallax maps relating various images to one another. To construct a 3D corrected mosaic, a new image is registered with the existing mosaic and then the new image information is merged into the ~i.eting mosaic. The merged image and mosaic become a corrected 5 mosaic that then becomes the existing mosaic for the next new image.
FIG. 8 depicts a flow chart of a routine800 for constructing 3D
corrected mosaics. At step 802, the system is supplied an ~ ting 3D
mosaic, then the mosaic assembly process proceeds as follows.
1. At step 804, a camera provides a new image of the scene 10 represented in the existing mosaic. The new image is taken from a new viewpoint of the scene.

2. At step 806, the routine uses the pose estim~tion process to compute the eleven pose par~meters that register the ç~i~tinF mosaic to the new image.

3. At step 808, the routine creates a synthetic image taken from the new viewpoint by reprojecting the ~ ting image mosaic using the estimated pose parameters. The reprojection is accompli~hed by folvv~d W~illg the existing image mosaic using Equation 3. To avoid generating any holes in the synthetic image arising from rch ~d image w ~ lg, the routine conve~tiol-~lly super-samples the second image as described in Wolberg, Digital Image Warping, IEEE Computer Society Press, Los Alamitos, California (1990). The reprojection must also be sensitive to occlusion within the image. Occlusion detection and proc~ ing is described below.
It should be noted that the new image can also be registered to the existing image mosaic and then the new image warped to the ~ ting image mosaic. However, to accomplish this W~illg, parallax information concerning the new image is needed to accurately warp the new image and capture the 3D geometry of the scene. To generate the necessary parallax inforTn~tion, either the previous image merged into the mosaic is temporarily stored in memory and used as a reference image to generate the parallax information with respect to the new image, or two new images and there respective parallax field is provided at step 804. In either instance, if the new image is to be warped to the mosaic, the new image must be provided with a parallax field. This is not necessary when the existing mosaic is warped to the new image. FIG. 8 only depicts the process for warping the mosaic to the image.

CA 0222~400 1997-12-19 W O 97/01135 PCT~US96/10485

4. At step 810, the routine merges the synthetic image into the new image to create a new mosaicr This new mosaic is supplied along path 814 as the existing mosaic. The synthetic image c-nt~in.~ image regions not present in the new image. These new regions are added to the new image

5 and extend its boundaries to create the new 3D mosaic. Note, in the merging process, to achieve a smooth mosaic construction, the routine can temporally average the inten.~ities of common regions in the synthetic image and the new image.
To construct the shape mosaic for the new viewpoint, the system 10 r~wald warps the ~xi.~ting s]hape mosaic to the new image coordinate system in much the same way as the existing image mosaic was reprojected. Given the pose parameters between the ~.xi~tinF image mosaic and the new image, the shape mosaic of those portions of the ~xi~t.ing 3D
mosaic not visible in new image, but only visible in f~xi~ting mosaic can al:so 15 be estim~te~l The reprojected shape mosaic is merged with this ~rlrlit:ir~n~l parallax information to complete the 3D mosaic as viewed from the new viewpoint.
5. At step 812, the routine displays the corrected 3D mosaic as seen from the new viewpoint. As such, new image information is accurately ZO incorporated into an existing 31D mosaic.
iii. Occlusion Detection and Proce.~ing Due to occlusion, in creation of the synthetic image (as flet~ile~
above) more than one image p,oint in the second image may project to the same point in the synthetic image. As shown in FIG. 9 points P, Q and R aU
25 project to the same point the inspection image 304. If the depth of each point relative to inspection image 304 were known, then it would be known that points P and Q are occluded by point R. In other words, from the viewpoint of the inspection camera, the points P and Q are occluded by the corner of the box 900. How~vel-, the parallax map does not contain the 30 necessary information to deduce the relative depth of each point.
Nonetheless, the relative depth information can be derived from the relative locations of the image points p, q, and r in the reference image 302. These points in image 302 must lie in an epipolar line 902 within the image. By connecting the focal points O and M of each image (camera) with a line 904, 35 an epipole m is defined on line 902. Given that focal point M is nearer the scene than focal point M, the order of the points from point m on line 902 itl~ntifies the occluded points. l[n this .ox~mple, point r proceeds points p and q and, as such, point R occludes points P and Q. If, however, focal point O is CA 0222~400 1997-12-19 nearer to the scene than focal point M, then the ordering of the occluded points is reversed and the occluded points are nearest to point m on line 902. The system uses this relatively simple te-hnique for d~ ...i..i..g oc~ l points within an image. Once reco~ni~e~, the oc~ l(lçl points can 5 be ~l~lete~ from the image, filtered, or otherwise processed such that potential artifacts generated by the occluded points are avoided.
D. Applications for 3D mosaics The foregoing description discussed the image procç.~ing system used to comhinç at least two images into a single 3D mosaic. Such a 10 dynamic system for representing video information has many applic~t.ion~, some of which are discussed below.
i. Object Height Estimation In general, parallax flow vectors vary directly with height and inversely with depth, where depth is the distance of an object from the 15 camera. As such, 3D mos~iç.s generated from aerial views of objects on the ground can be used to estimate the height of the objects above the earth's surface. To elimin~te depth and efitim~te object height from the parametric surface (the earth's surface), the inventive system is adapted to use a characteristic property of aerial view images (her~.in~fter referred to as an 20 aerial view property). Sperific~lly, the depth from a camera to an object is typically much greater than the height of the object from the ground. In nadir aerial images, the depth of all points is apl~loxi..~tely the same so that a weak perspective projection can be used to estimate object height .
VVhereas, in an oblique view, there can be con~i~çrable depth variation 25 across a given image. How~ver, for any single point in an oblique aerial image, the depth of that image point is appro~im~tely the same depth of a virtual 3D point obtained by ~xtPntling a line of sight ray from the camera and intersecting it with the ground plane. Therefore, the system can factor out the depth in the parallax Eqll~tion.~ 4 and 5 by estim~tinF an equation 30 of for ground plane.
This specific application for the inventive image processing system uses the magnitude of the displacement vectors to infer the m~gnit.l~ of the height and the direction of the flow vectors to infer the sign of the height.
The sign of the height in(li(~.~tes whether the point is above or below the 35 plane. For the seqnenti~l approach, the m~nit:ll(le of the displacement vector Y2 = ~(XW - x, )2 + (Yw _ Yl ~2 for the case where the tr~n~l~tion is parallel to the image plane is given by Equation 9.

W O 97/01135 -17- PCTÇUS96/10485 r2 = PT ~ + 1;, (9) The m~nitude of the displ~cem.ont vector for the case where Tl~O is given 5 by Equation 10.
l~.H
r2= ~ r (10) where YF = ~(X., -XF) +(YW -YF) is the distance of the point (xw,yW) from 10 the focus of expansion (FOE).
Since NTpl = f, using the aerial view property eqll~tionR 9 and 10 can be reduced to Equation 11.

I = S2HN2Ty~ (11) where I=Y2/YF is a measurement obtained from the ~lignment parameters and the est.im~te-l parallax vector and S is a proportionately factor that dependsupon the trFn~l~tion vector Tl and the distance T.,I . Equation 11 can be lew~il,len as Equation 12 to solve for the height H of any image point 20 in a scene.

(K,x + K2y + K3) (12) where K is an unknown vector having components K1=SN2X, K2=SN2X and 25 K3=fSN2z To best utilize the height estimation computation, the intrinsic camera parameters such as focal length and image center should be determined using standard te- hniques. H~ w~ve~-, in some applications, it is not possible to calibrate the camera parameters nor obtain these 30 parameters apriori. As such, a nllmber of alternative estimation methods have been developed to estimate height when either the focal length and/or image center are known or unknown.

CA 0222~400 1997-12-19 If the focal length and center are both unknown and height of at least three points are known (reference points), then Equation 12 can be solved to linearly estimate vector K.
If focal length and camera center are both known, a normal plane is 5 inferred using Equation 12. This equation related the quadratic registration parameters to the tr~n.ql~tion~ rotation and normal of the plane, but the tr~n-ql~tion direction is computed dur ng the quasi-parametric residual estimation. ThetrAn.ql~tio~ direction together with Equation 10 provides a linear set of eight equations cont~inin~ six unknowns; namely, normal 10 vector N and rotation vector Q. Since the tr~n.ql~tion used in Equation 9 is T2, while the tr~qnql~tion computed for the parallax flow vectors is Tl, the quadratic transformation ~lefin~ by parameters Pl through P8 is inverted.
Alternatively, the inverse quadratic transformation can be directly estimated by inter-r~h~n~ing the two images dur~ng parallax vector 15 eqtim~tion. The tr~nql~t.ion vector is determined up to a pre~ o.fin~ sçflling factor. As such, the height of at least one point is neetl~l to determine the height of each other point in the 3D mosaic or any constituent image (real or virtual).
To determine the height of image points when focal length is unknown 20 and image center is known, Equation 12 is solved using the known height of two points in the scene. Since focal length is unknown, it is not possible to utilize all eight parameters given by Equation 8. Howev~r, the linear parameters Pl through p4 do not depend on the focal length and, as such, Equation 10 pert~ining to these parameters can be used. On inspecting 25 these equations, when Tz=0, the normal component Nz cannot be determined. However, the components Nx and Ny can be determined up to a scaling factor. Since the focal length is unknown, this result is also true for the case when Tz~0. As such the tr~n.ql~tion vector is a scaled version of the vector [fTx ~y Tz]. Therefore, whether Tz is zero or not, the method is 30 capable of det~...;..;..g at least one component of the vector N and subsequently the vector K The method uses the height of at least two image points and Equation 12 to determine vector K and the height of any point in the mosaic.
The foregoing te.hnique has been used experim~n~lly to determine 35 the height of a number of objects lying upon a plane (ground plane). FIG. 10 schematically depicts the experimental set-up. Specifically, a camera 1000 was mounted on a tripod 1002 ~l~x;...~te to a flat plane 1004 upon which various shaped objects 1006 were pl~ce-l The objects ranged in height from -W O 97/01135 19 PCT~US96110485 1 inch to 4.9 inches above the plane. Initially, an image was taken with the camera at an angle of 3~ degrees from hori~ont~l and a~oki~ t~ly 69 inches above the plane, i.e., position 1. Next, the camera was moved fol~/v~l in the y-z plane by ap~lox;...~tely 4 inches to position 2. At 5 position 2, a second image waLs captured. The foregoing height estim~tiC)n technique was used to register the image from position 2 (inspection image) to that of the image taken at position 1 (reference image) and then determine the height~ of the various objects. Without knowing the focal leng~h and camera center and knowing the height of three points, the 10 method det~lllPilled the height; of the entire scene. When compared to the actual height of each object, the largest st~nfl~rd deviation for the estimated height was 0.2 inches.
In a second experiment, the inspection image was generated by moving the camera to position 3, e.g., a~plf xi~ tely 4 inches laterally along 15 the x axis from position 1. The method was ag~in used to register the images and generate a height map. The result of this experiment showed a largest standard deviation of C!.27 in(.hqs.
The foregoing height estimation processes were ~ c~ ed in the context of three scenarios; namely, (1) when no camera informAtion is 20 known, (2) when camera focal length is known and camera center is unknown (or vice versa), and (3) when both camera center and focal length are known. In the third scenario, the assumption regarding the aerial view property is not relevant and is not assumed. In the first and second scenarios, the assumption was used Nonet.hele~, the foregoing equations 25 can be slightly modified to avoid using this assumption to solve for the h~.ight The foregoing technique used to find object height within a mosaic of two images can be ~ten~l~rl to determine height inform~tion in mosaics comprised of more than two images. The multiframe te(~;hrlique uses a 30 batch method that registers all the images to a single reference image and a single reference plane that ex~tends through all the images. Once all the images are ~lign~(l along the plane, the method computes the re~
parallax displ~cçm~nt vectors between each pair of image frames. The height map is inferred from the sequence of estimated residual parallax 35 displacement vectors.
To accomplish this computation, Equation 11 is l~w.;l ~en as Equation 13.

CA 0222~400 1997-12-19 W O 97/01135 PCT~US96/10485 li = H(K,x + K2y + K3 ) (13) Where Ii and Si each vary from frame to frame, while the right-hand 5 side of the equation is constant over the entire image sequence. As such, the ratio of Ii and Si is an invariant quantity across the mos~ic For a sequence of N inspection frames and given the height of three image points relative to the image plane, the method solves 3N linear eqll~tion.~ cont~ining N+3 unknown values; namely, N Si terms and the 10 vector K First the method finds a solution for the N+3 unknown values and then uses the values to solve Equation 13 to estimate the height of other points in the mosaic. If the focal length and/or image center is known, then the eqll~t~ can be solved using only one or two known image height values.
The image height compllt~t.ion can be comhined with the 3D
corrected mosaic routine to produce topographic mapping inform~ticn For example, the foregoing height estim~ti~n system is used to produce a height map of terrain, and the 3D corrected mosaic routine uses the same images generated used to generate the height map to produce a 3D corrected mosaic. Thereafter, a new view, e.g., pcrp~n~ r to the terrain, can be synt.hesi7ed and the height map can be corrected (altered to conform to the new view). As such, the height map can be generated from any arbitrary viewpoint of the scene. Consequently, images that are captured at an oblique angle of a scene can be converted into an image of the scene from an orthogonal viewpoint and height inform~ti-~n can be generated fiom that new viewpoint.
ii. Synthetic View Generation (Tweening) Generally speaking, given an existing 3D mosaic representing a 3D
scene and the pose of a new viewpoint with respect to that mosaic, the system can derive a synthetic image of the scene. As such, by capturing a scene using di~e~ t cameras having different viewpoints of the scene, the system can synt.he~i~e images that are a view of the scene from viewpoints other than those of the cameras.
FIG. 11 depicts a haldw~u.2 arrangement of camera(s) within a 3D
studio 1100 used to generate a 3D mosaic representation of the studio. The studio is merely illustrative of one type of 3D scene that can be recorded by the system. It, of course, can be replaced with any 3D scene. The 3D

CA 0222~400 l997-l2-l9 W O 97/01135 -21- PCTrUS96/10485 mosaic generation process, as discussed above, uses a plurality of images of the scene to produce one or more mos~i~R representing the scene. As such, a two f~im~n~i-)n~l grid 1102, ~l~.fining a plurality of one foot by one foot squares, is used to define camera positions within an area ~ xi~ te to the S studio. In general, the specific size of the grid squares, i.e., the number of camera positions, will vary depending upon the complexity of the scene.
Also, the shape of the grid will vary depending upon the type of scene being recorded, e.g., some scenes, such as a sporting event, may be circumscribed by the grid.
To produce the images for the mosaic(s), a camera 1104 records an image (or a series of images, e.g., video) from each of the grid squares. The images are typically recorded at various camera pan, tilt, rotate and zoom positions for each grid square to generate the plurality of images from a plurality of viewpoints. The image proce.c~in~ system described above generates a 3D mosaic from the various images recorded at each camera location. .qimil~rly, 3D mosaics are generated for the other camera locations at each of the grid points. For ~mpl~, 3D mosaics 1106, 1108, 1110 (only the image mosaic portion is depicted) represent the scene as recorded from g~id loc~tionq 1112, 1114, and 1116. These 3D mo.s~i~q are merged to generate a synthetic image 1118 repre~enting the scene as viewed from, for example, la,cation 1120. The image generated at the synthetic viewpoint is not a areal" camera viewpoint, but rather is synthesized from information cont~ined in the various mo.s~içs.
The system of the invention generates the synthetic image using one of t~,vo processes. The first pra,cess used to generate a synthetic image view of the scene, warps each of the individual mosaics (e.g., mosaics 1106, 1108, and 1110) to the location of l;he synthetic ~iewpoint (e.g., location 1120).
Thus, as each 3D mosaic is generated for each grid point, the 3D mosaic is stored in memory (mosaic storage 1122) with respective to its associated grid point. Given a new viewpoint location, the mosaics are recalled from memory to generate a synthetic image representing the scene from the new viewpoint. Depending upon the compl~ity of the scene being imaged, the system may recall each of the 3D mosaics in memory or some subset of those mosaics, e.g., only recall those mosaics that are nearest the new view location. Using new view generator 11249 each recalled 3D mosaic is warped to the new viewpoint Location (e.g, location 1120) and the mosaics are merged to form the new view image 1118. Image merging is typically ~ccomrlished by averaging the pixels of the various mosaics used to form CA 0222~400 l997-l2-l9 W O 97/01135 -22- PCT~US96/10485 the new view image. H~wev~., other forms of image merging are known in the art and can be applied to these 3D mosaics. The result generated by the new view generator is a new view (e.g., image 1118) of the scene 1100.
The second process warps each camera view 3D mosaic to the S location of a previously generated 3D mosaic. Illustratively, the 3D
mosaic 1106 from camera location 1112 iS pro~ f e~l first, the mosaic 1108 produced from camera location 1114 is then warped to the coordinate system of location 1112, and lastly, the mosaic 1110 produced by camera 1118 is warped to the coordinate system of location 1112. As such, a composite 3D mosaic of the scene (not specifically shown) is generated by comhining (merging) the various 3D mosaics as viewed from a reference coordinate system (e.g, location 1112). Of course, any coordinate system can be used as the reference coordinate system. Also, rl~.pPn~ling upon the scene being imaged, less than all the 3D mosaics generated at each grid point may be used to produce the composite mosaic. Thereafter, any synthetic view of the scene can be produced by W~illg the composite 3D
mosaic to the coordinate system of the synthetic view, e.g, location 1120.
The result is a new view (image 1118) of the scene.
iii. Scene Change Detection The system of the invention can be used to monitor a scene through a moving im~ing device (e.g, camera) and detect changes in the scene.
The system corrects for changes that are due to parallax and viewpoint changes and, thc er(~le, is less sensitive to false scene changes than prior art systems.
Specifically, the system detects change by c- mhining a sequence of images to form a 3D mosaic (or a corrected 3D mosaic). For any image in the sequence of images, or for any new images that are to be added to the 3D mosaic, the system compares the selected image to both a previous and a next image in the sequence using the PthenP process, the P-and-P
process, or pose estimation. The afinal" areas of cha~ge that represent "real" moving objects are those that appear in both the comparisons to the previous and next images. The system deems all other areas of change to be due to viewpoint changes, i.e., parallax. This simple heuristic operates quite well in ~qliminflt.in~ many areas of change which are viewpoint dependent such as specularities and occlusions.
iv. Other applications 3D mosaics can be used in applications where 2D mosaics presently find use. Specifically, since image re~ n~l~ncy is removed by comhining W O 97101135 -23- PCT~US96/10485 seql7~ncefi of images into mosaics, mosaics find use in video tr~n.~mi.~.~ion, video storage and retrieval, an~d video analysis and manipulation. By using mosaics, less video data need be transmitted, stored, or analyzed. As such, the 3D mosaics generated by the system of the invention will find use in 5 many appliç~tion~ where image information needs to be çffi~ nt.ly m~nipl71~ter7., stored, and/or transmitted.
Although various embor7.ime~tq which incorporate the te~rhing~ of the invention have been shown and described in detail herein, those skilled in the art can readily devise many other varied emborlime~t,s that still 10 incorporate these te~hings.

Claims (14)

WHAT IS CLAIMED IS:

1. A method of processing a plurality of images to generate a three-dimensional (3D) mosaic of a scene comprising the steps of:
providing a plurality of images of the scene; and registering said images to construct said 3D mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three dimensional geometry of the scene.

2. The method of claim 1 wherein said registering step further comprises the steps of:
registering each image in said plurality of images along a parametric surface to produce registered images;
determining, in response to said registered images, translation parameters and a parametric motion field useful in Aligning the images along the parametric surface; and generating a parallax field representing parallax of objects within the scene.

3. The method of claim 2 further comprising the step of converting said plurality of images into a plurality of multi-resoutional pyramids, where each image pyramid contains a plurality of levels; and wherein said registering and determining steps are iterated over each of said levels within said multi resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.

4. The method of claim 1 wherein said registering step further comprises:
simultaneously registering said images in said plurality of images along a parametric surface to produce registered images, determinining, in response to said registered images, translation parameters and a parametric motion field useful in Aligning the images along the parametric surface, and generating a parallax field representing parallax of objects not lying within said parametric surface.

The method of claim 1 further comprising the steps of:
converting said image mosaic and said shape mosaic into multi-resolutional pyramids;
converting a new image into a multi-resolutional pyramid; and determining pose parameters for relating the new image with the image mosaic and the shape mosaic, where the pose parameters contain translation parameters, a planar motion field, and a parallax motion field for the new image.

6. The method of claim 6 further comprising the steps of:
providing an existing 3D mosaic;
determining pose parameters for a new image with respect to said existing 3D mosaic;
warping said existing 3D mosaic to image coordinates of said new image to create a synthetic image, where said synthetic image represents a view of the 3D mosaic from the coordinates of the new image; and merging said synthetic image into said new image to produce a new 3D mosaic that is a combination of said new image and said existing 3D
mosaic.
providing a next image that sequentially follows said new image; and detecting changes between said new image, said existing 3D mosaic, and said next image, where said changes represent motion within the scene without detecting parallax due to viewpoint change as said motion.

7. The method of claim 1 further comprising the steps of:
detecting points within said 3D mosaic that are occluded within the scene by objects in the scene; and image processing the detected occluded points such that said occluded points do not produce artifacts in said 3D moscaic.

8. The method of claim 1 further comprising the step of:
estimating a height of points within said 3D mosaic relative to said parametric surface, where said height of said points form a height map that represents the height of object points within said scene.

9. The method of claim 1 further comprising the steps of:
providing a plurality of 3D mosaics representing a scene from different viewpoints, where 21 3D mosaic has been generated at each viewpoint;
warping said plurality of 3D mosaics to a reference coordinate system;
merging said plurality of 3D mosaics to form a composite 3D mosaic;
providing coordinates for a new viewpoint of said scene;
determining parameters to relate said new viewpoint coordinates to said composite 3D mosaic; and warping said composite 3D mosaic to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the composite 3D mosaic taken from the new viewpoint.

10. The method of claim 1 further comprising the steps of:
providing a plurality of 3D mosaics representing a scene from different viewpoints, where a 3D mosaic has been generated at each viewpoint;
providing coordinates for a new viewpoint of said scene;
determining parameters to relate said new viewpoint coordinates to a plurality of the 3D mosaics;
warping said plurality of 3D mosaics to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the 3D mosaic taken from the new viewpoint; and merging said plurality of 3D mosaics to form said synthetic image.

11. The method of claim 1 wherein said registering step further comprises the steps of:
performing a plane-then-parallax process including the steps of registering each image in said plurality of images along a parametric surface to produce initially registered images; determining, in response to said initially registered images, initial translation parameters and a initial parametric motion field useful in initially aligning the images along the parametric surface; and generating an initial parallax field representing parallax of objects within the scene; and simultaneously registering, using said initial translation parameters, initial parametric motion field and initial parallax field, said images in said plurality of images along said parametric surface to produce final registered images, determining, in response to said final registered images, final translation parameters and a final parametric motion field useful in aligning the images along the parametric surface, and generating a final parallax field representing parallax of objects within the scene.

12. A system for generating a three-dimensional (3D) mosaic of a scene from a plurality of images of the scene, comprising:
means for storing said plurality of images;
a registration processor, connected to said storing means, for registering said images to construct said 3D mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimentional geometry of the scene.

13. The system of claim 12 wherein said registration processor further comprises:

a plane-then-parallax registration processor for aligning said images along a parametric surface that extends through the plurality of images to produce translation parameters and a parametric motion field used to align the images within the image mosaic and then for determining a parallax field representing objects within the scene.

14. The system of claim 12 wherein said registration processor further comprises:
a plane-and-parallax registration processor for alinging said images along a parametric surface that extends through the plurality of images to produce translation parameters and a parametric motion field used to align the images within the image mosaic and for determining a parallax field representing objects within the scene.