CA2231374C - Method and apparatus for automatic electronic replacement of billboards in a video image - Google Patents

Abstract

Apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means operative to measure the Field of View (FOV) of the TV camera relative to a known reference position.

Description

l~TEIOD AND APPARATUS FOR AUTOMATIC ELECTRONIC
REPLAOEMENT OF BILLBO~DS IN A VIDEO IlVIAGE

The present invention relates to a method and al~palaLus for autom~tic~lly replacing billboards in a video image.

The present invention has particular use in electronic repl~cem~nt of billboards in a st~ m or other venue but can be used to provide acculate data relating to camera ol;e.l~lion for other ~ul~oses.
In previous systems it has been proposed to electronically replace billboards in a st~ m which are viewed by a viewer on television. The bill~oaluls in the st~ lm are televised by a TV c~.e.~ and the boards are clccLlonically altered so that the TV viewer at home sees a dirr~ nL board to the spect~tor in the st~ m or other venue.

The known systems such as those described in US 5,266,933, an apparatus and method for electronically altering video images is disclosed.
The appalatus and method disclosed in the US patent and also in US
patent 5,353,392 whilst theoretically allowing repl~eln~nt of billboards do not solve the many practical problems e!-~....t,.~d in real e.~ u~ ents. Most of these problems are related to the recognition and repl~celnent processes.

Relying entirely on pattern recognition techniques which utilize only the video signal to identify and localise billbûards for replacement introduces major problems which affect the practical value of such a system.

Clearly, any pattern recognition scheme, including those described in US 5,264,933 and US 5,353,392 must rely on useful visible reaL~lr~s in the innage that can be co~ d with pre-defined descriptions. Such features should be located inside the billboard or at its neighbourhood.

In realistic situations, the visibility of these features might change, continuously or otherwise from practically zero to a some threshold visibility which allows the yat~l~ recognition sr.hl~-m~- to work ~r()~
These changes can occur in the direction of growing or lcJ.~ visibility.

Such situations inrl~de:
~ Acceleration or de-acceleration of camera motion cing a huge amount of blur.
~ l;Yr~c~;~e zooming-in or ~ ~ing-out of the billboard.
~ FYc~cive occlusion by players.
1~ ~ En~.~ or ex~ting a ca.. ~ 's field of view by any co,..bi.lalion of pan, tilt and zoom operations.
~ Any combination of the above mentioned meeh~ni~m~

Th~erG.~, in practical situations, a co..~ o~ repl~r~mP~t of 20 billboards, is not possible. Even if an i~lt,llu~l~d ~ was allowed, it would require a delay of at least a few S ~ l~lS to decide whether the resulting replacement interval is ~r~pt~hle or not. Such a delay is usually not pe~ ed in live bro~c~ting of sports events.

2~ Replacing arbitrary billboards introduces further problems. A
seamless replacement requires to identify the foreground objects oc~ ling the billboard in order to inhibit replacement at places of occlusion.
Foreground objects mainly consist of players but also the ball or other objects Consider now a player with a red shirt, occluding a part of a 30 similarly red portion of a billboard. Colour cO~ a~ cannot be used robustly to identify occlusion. Furthermore, since the player is a non-rigid object, motion or shape information cannot be used accurately enough to ~u~allLGe perfect repl~r~ment.

Another problem which may arise in practical situation is resolution of billboard identity. Consider two identical billboards positioned at two dirr~len~ locations in the arena. Suppose dirr.,lG~-t repl~r~m~-nt billboards are assigned to each of these physical billboards, then one must be able to tell which one is which. This can prove to be ~ ,...cly .lifficlllt 10 ~speci~lly if no unambiguous ~ Ur~S are visible.

This invention describes a robust system for billboard rep~ mPnt based on some or all of the following key rl(,--~P~

15 ~ Pan, tilt, zoom and focus SG~ISOI~ f,d to the camera, which enable after a proper set up procedure to e~ the presence and location of billboards in any given video field.
~ Image processing methods and their embodiment which enable to refine the sensors' estim~tes 20 ~ Physical billboards which are coloured p.<,~lly to enable the efflcient def~cti~-n of occlusion by chroma-key techniques.
~ Colour variation or a pattern within the physical billboard for further enhancing the performance of the image processing methods ~ The plesent invention has a first object to provide a method and apparatus which enables identification of the location of a billboard or other static object in a stadium or other venue in any weather conditions with any p~nning speed of the camera and with any other change in camera parameters.

S The present invention therefore provides apparatus for automatic electronic replacement of a billboard in a video image inclll~ing an ~utom~tic camera orientation mea~u~ wnL a~ alu:, inrlu-lin~ motion l--~ur~ enl means O~l a~ e to ~ F~ the Field of View (FOV) o~ the TV camera relative to a known lcr~ ce positic!n The ~l~selll invention also y~r~l~bly provides ~ ~alus for ~ulolllatic clcc~ .~ic repl~ . . .e..l of a billboard in a video image, inrlurlin~
image proc~c~ means for procç~cin~ video signals ~ n~ led by the TV
camera, in which said proces~in~ means int~lu-l~s caL1,1ali".~ means for~5 periodically ~ o.~ ;r~lly c~ tin~ the motion l~-U~Ul~ means, Lus in which the motion ~ulc.~ent means i~ des means for .C~ulillg the pan tilt, zoom or focus of the camera relative to known lGÇ~ .GIlce positions and app~alus in which the motion ~ ul~ c.-L means includes means for m~uling the pan tilt, zoom and focus of the camera~0 relative to known reÇ~I~.lce positions.

The ~ sen~ invention thel erc,lG uses ~lyl~alllic recalibration to correct for residual sensor errors or abberations in an i~ elrGct model and for sensor drift over time. Thus it is possible in accordance with the 2~ present invention to use less stable sensors and the a~alus and method in accordance with the present invention can ~c~commodate movement in the carnera position. The image correction process for calibration of the sensors elimin~tPs the nPcPsci~y to keep the sensors stable by merh~ni means by recalibration automatically with reference to the video image.

In the initial set up procedure corrections can be incorporated for calibration for billboards which are, for example, not in the centre of the Field of View - for ex~mple a billboard which is in the top left-hand corner of the screen can be adjusted by, for example, 3 pixels to take into S account abberations in the camera.

Further problems which arise in the prior art systems are firstly when the billboard is either sllbst~nti~lly totally oa~luclP~ or secon-lly, is ocrln~l~ by an object, such as a player, of the same colour as the real 10 sign on the billboard.

This can firstly, as explained above, lead to non-~,cog-liLion of the billboard and also secQnt11y~ to ~liffic~llty in s~ti~f~ctory repl~cernPnt of the billboard.
In the first case the real billboard can have already been rep~ P l in the video image but if the camera zooms into a close up or if a dirr~ nl camera is used for the close up then the lock may be lost due to only a very small portion of the billboard being in view. In the second case the 20 player may have on a strip which is the same colour as the billboard. The prior art systems propose to distinguish the billboard from the player on the basis of movement if the colours are the same and to analyse the "moving~ pixels to determine occlusion. This is reasonable in theory but fails in practice since not all payers are moving at all times. Thus, if a 25 number of players move in front of a billboard and one player remains after the others have moved on, the electronics will not be able to distinguish on movement grounds. Since colours are distorted by floodlighting, shadows, differences in reflectivity and different lighting conditions for a foreground player and a background billboard, there will 30 be occasions in practice where the system fails In such cases either the original billboard may re-appear on the video image or the repl~rem.ont billboard will not be accurately oc~ de~l It is again possible to introduce a delay in the video L~ cion S to enable the electronic signal processing to be more accurate but this does not solve the practical problem where a plurality of players move in dirr~lel-~ directions to ocdude a billboard. The ..~ce~. y delay is considered nn~l~F~l~bl~ and will in any case not solve all of the above problems.

Inacco~ eewitha~r~ lborl;..~ 1 ofthe~ .llinvention it is pn~posed to replace the real billboards with cl~ a-key panels or with ~ l areas forming ch~ llla-key panels.

Chroma-key is esse.~ lly an occlusion te~hn-que allowing, for example, a news reader to stand and move about in front of a chroma-key board, usually coloured blue or another suitable colour. The news reader (foreground) is distinguished from tihe chroma-key board (bac~,loulld) by colour dirre,ellliation and can thus move in front of the repl~ t background with normal occlusion of the foreground and bac~;glvulld.
This terhnique is very well known in television studio systems and is described in llUll~el-OUS US patents, including US 2,974,190 and 4,200,980 Recently, several systems which combine camera sensors with chroma-key for the purpose of coorflin~ting the movement of the graphics backgrounds with those of the camera, have been described and demonstrated.

[Ref. K. Haseba et al., Real-timing compositing system of a real camera image and a computer graphic image, International Bro~lc~ting Convention, 16-20 Sep. 1994, Conference publication No. 397, IEE 1994, pp. 656-660].

S In principle, such an arrangement could be used for billboard replacement where the sensors unambiguously solve the recognition problem and the chroma-key billboard helps to handle occlusion ~ ly.
However due to some major dir~.~.~ces, this arr~n~f ,.~ ..l should be enh~n~ l. These e~ .-r~ are the basis of the ~.~,se.-l invention.
In a virtual set aprlir~tion~ the c~,~ a is typically 2-10 metres away from the ro~uu-ld and the entire field of view is usually replaced.
In co,l,~a.;son, a billboard may be several hundred metres from the camera and therefore a repl~ce~nsnt system using sensors is much more 15 sl-sc~ptihle to sensor errors:

~ Due to the large focal ~ t~nr,es, the same sensor ac;u-a~;y will translate to larger geometric registration errors.
Consider a rotary encoder of 81000 pluses/revolution, then the angular precision is 0.0044 degrees or 75 micro-radians.
The r~ ability is twice as bad. Con~i~er a shooting range of 100 m with a field of view of 4 metres, then the FOV is 40 milli-radian. The error translates to 768 *? 150 / 20000 = 2.88 pixels.
25 ~ Since the field of view includes many stationary objects (includin~g billboards) which are not replaced, the human - observer will be much more sensitive to the registration errors Additional errors may originate from lens distortion, rotation axis which does not pass through the focal point, non-zero roll angle, etc.

Chroma-key is b5~.ei~-~lly a techniflue for studios where the min~tion is carefully ~le~ei~nç~ and controlled and the controls of the chroma-keyer are carefully adjusted for the specific arrangement of blue-screen colour and illl-min~tion.

In a sports event, the conditions may be highly non-ideal and lGquil~ some morlific~tion to the chroma-key algo~ ls. In particular, the keyer p~A.~.,t~ ~ ~ should be ~ A to the spe~ fic billboards being re~lac~ due to rh~n~s in ill~....i.~l;on across the arena.
Accol~iu~ly in the ~-~se.~L invention it is p ~osc~ to use clu~ a-key panels and to replace these in the video image by the repl~r~m~nt billboards.

Since it is .. ~oc~.esS~. y for ~rceL ocrllleion that the players or other ocrl--riing obJects are of dirr~ r~ colour to the ~ "lla-key panels, it is proposed in a further l"Gr~,-.ed embotlimerlt to provide cl~ull-a-key panels in which the colour of the panel may be changed, for ex~mrle, by using a rotating billboard structure which is known in the art. One side could, 20 for example, be blue and ~nother green. Green may be ~.~re able in a sports en~,ir~ll.lle.~ since players tend not to wear green as this would not CCillL~ with the backgl~lulld sports surface.

In a further preferred embodiment and in particular where a 25 plurality of billboards require replacement, a patterned chroma-key board is used. The pattern may be of any suitable shape but is preferably sçlect~l to be suitable for the size and shape of the billboard or series of billboards and also to the anticipated video conditions. Thus if a lbillboard is only able to be viewed from a long ~ t~nce then a different pattern will 30 be selected to a billboard which is to be viewed in close up.

The pattern may comprise different colours or may be dirr~,~eilt shades of the same colour. The pattern may comprise vertical and honzontal lines or may comprise a decorative pattern, a discernible advertisement, company logo or other suitable wording which may be S more aesthetically acce~lal~le.

The use of a pattern allows further discrimin~tion of the position of the camera and may allow movement of the camera from a fixed position The camera ol;enlalion data can be tr~n~mitt~ together with the video signal and will identify the position of the billboard in any w~lLel, li~htin~ or occlusion conditions. No ~ ,.ce is ICyUil~d to any feature within the sports venue to identify the position of the billboard.

l~i The camera sensols can be accurate to a few pixels or in physical terms to a~ mately 1 cm at a range of about 100 metres thereby enabling accurate repl~cem~nt of any billboard. The recalibration can be carried out co,lli"uously or only periodically, particularly if an initial adjll~tmPnt of the calibration of billboards not in the centre of the FOV is 20 recorded on set up.

By use of the chroma-lcey techniques there is no requirement to transmit any occlusion data since this can be readily inserted at a receiver and the occlusion inserted in the normal manner.
2~
In a preferred arrangement within a stadium or other sports venue real billboards with normal advertising material will be situated on one side of the stadium to be viewed by a first plurality of cameras and chroma-key billboards will be situ~t~ on another or the opposite side to be viewed by a second plurality of cameras This, for eY~ le, the home nation may view the norrnal billboards, with the inte~ tional TV audience seeing only substit~lte~l boards S The present invention also provides a method for electronically replacing a billboard in a video image display, ge.lcl~ted by a camera, comprising the steps of:
a. i~1rntific~tion of the position of a ~L~ u~ar billboard in a st~ m or other venue, said identifir~ti~-n step co...~
specifying on the video display the billboard to be replaced by i~ tifir~tion of its four CO~ S at a first camera position;
b. stonng the i~1entifir~tion illfo~ ;on;
c. ~I.o~.;IC.;.~ the mov~,.l.e,.L of the carnera in pan, tilt and zoom;
d. storing the monitored movement of the camera on a field by field basis; and e analysing the size and position of the billboard to be r~placed from the i~Çc,.,~ation recorded in its first known position and the stored mov~.. e~ of the camera to provide inforrnation relating to the size, ~.~ e and position of the billboard in the ~ Sent video field;
f storing in a billboard repl~rern~nt store a reFl~celn~nt billboard eO be used in replacement of the billboard in the stadium;
g electronically altering the size and perspective of the replacement billboard in accordance with the camera motion information to confonn to the size and perspective of the billboard to be replaced in the present video frame; and h electronically replacing the billboard in the present video frame by the replacement billboard.

In a preferred embodiment the step of analysing the size and position of the billboard to be replaced comprises a further step of 5 analysing a plurality of video scan lines to provide fine adjustment information relating to the exact size, perspective and position of the billboard to be replaced.

In a further yr~r~ d embodiment the billboard to be repl~r~l is 10 blank and is of colour suitable for chl~-,-a-key rep~ e-m~-nt Such colour may be of a blue or green shade, due to the fact that these colours are rarely found in human skin and hair.

In a still further ~lcÇ~ d emb~1im~rlt the cl~ a-key billboard is 15 pa~ .,ed with a pattern of a suitable shape for the ~ul~ose of f~cilit~tin~
the fine adjl-~tment process mentioned above. The step of analysis of the size and position of the billboard comprises the analysis of the pattern to ascertain the exact position of the billboard.

In a further embodiment the correction of the sensor-based prediction by the analysis of the pattern will be controlled by a figure of merit (accuracy estim~t~.) for the analysis, which will be computcd automatically.

In a further embodiment the step of e!ectronically replacing the billboard in the present video field by the replacement billboard include - the step of superimposing occluding objects by use of the chroma-key techniques.

In a further preferred embodiment the billboard to be replaced can be changed to best match the colours and shades of colours on the players costumes, for the purpose of providing a good contrast between the billboards and the players. For ex~mrle, if these cost--mes contain shades of blue, then a green billboard may be selected.
The backing colour can be select~l between blue, green and red.
In order for the cl~ a-keyer to r~le~ t~s all parameters nec~ss~ry to ~-rO~ proper image CO~IlyO~ , the system requires a sample of the bac~gn~u.ld colour as ,~L.~.~cc. This step can be done ~J~o...~;r~lly by 10 sr~nnin~ the image and detectin~ the purest and br ~lht~-st colour.
Advanced d~lollla-keyers enable the user to m~nll~lly select the area to be sampled In a further ~.~,f~ ,d emlb~ the cluv-lla-key a~..lus will 1~ have a multiplicity of set up con~ n~, each colle~ dillg to a dirf~
region of the stadium. The camera pan, tilt and zoom il.roll"~lion will allow to load the colres~onding set up conditions.

In a further embodiment, the fine adjustment i~lÇo,lllation will be 20 used to co...~ drift elTors of the se.~so-~. In a practical situations, the sensor error will have a ~;gnifir~nt portion which is at te.ll~o~
frequencies which are much lower than the video field rate. Thus these sensor-induced errors can be reliably estim~d from good video field and subtracted from subsequent mea:,ul~ment.
The present invention also provides apparatus for carrying out the method of electronically replacing the billboard as specified hereinbefore.

Embodiments of the ~ selll invention will now be described, by 30 way of example with reference to the accompanying drawings in which :-.

Figure 1 shows a stadium or other venue illustrating the ~a alus according to the ~r~sf ,l~ invention;
Figure 2 shows the video image of the st~ m as seen by the camera in a first position;
Figure 3 illustrates a stadium with billboards in several different positions;
Figure 4 illustrates a zoomed camera shot of a billboard illu~lld~ g the problem with prior art systems;
Figure 5 shows a ~ ..ed cl--(,nla-key billboard for use with the 10 ~lcselll invention;
Figure 6 shows in block dia~ ;c form ~ ;Uill~ associated with the camera arrange.~ l of Figure 1 for l~ ;.)g video data and camera o.;e,.-l~tion data;
Figure 7 shows in block dia~li1.. i.lic form receiver cil~;uil ~r for 15 co-operation with the ~ fl eil~;UiLl~r of Figure 6;
Figure 8 shows a flow diagram for the operation of the cir.;uiL.~ of Figure 7;
Figure 9 shows an arrangement for a billboard setup data store;
Figure 10 shows an arrangement for billboard setup data;
Figure 11 shows a flow diagram for perspective l-~n~ro----~Lion co..ll)ulalion;
Figure 12 shows an arrangement for camera intrinsic store;
Figure 13 shows equations for dynamic recalibration;
Figure 14 shows a flow diagram for dynamic recalibration;
Figure 15 shows the process of recalibration;
- Figure 16 illustrates the problems relating to siting of billboards at different locations within a stadium with differing lighting conditions;
Figure 17 shows a graph of minimum and maximum levels for U
30 & V illustrating the operation of a chroma-keyer;

Figure 18 shows a billboard with occluding obJect illustrating the principle of adjustment of the chroma-key colour for a billboard;
Figure 19 shows an exemplary remote receiver for reception of billboard coordinate data and perfect chroma-key colour, occlusion being S effected by chroma-key techniques;
Figure 20 shows an alternative arrangement for the billboard set up data store illustrating an alternative embo~lim~nt Figure 21 shows a flow diagram for dynamic set up procedure for camera p~nnin~ for use with the billboard set up data store of Figure 20;
10 and Figure 22 shows a flow diagram for dynamic set up procedure for camera tilting for use with the billboard set up data store of Figure 20.
With ~re~ ce now to Figures 1 to 4 the prin~irle of the ~l~,se.
invention is now e~rl~in~-A
In a stadium or other venue 10 billboards 14,16,18 are in~t~ A at the side of a pitch represented by markings 12. These billboards are viewable by a camera 20. Billboards 15,17,19 may be present on the opposite side of the stadium for viewing by a further camera 21. The 20 st~ m terraceslseating are shown diagr~mm~ically by the lines 11.

Camera 21 may in a preferred example be a normal TV video camera and will transmit its output video signal directly to a first feed which may serve the local population. Although we refer to camera 20 25 or 21, it may be clearly understood that there could be a plurality of cameras on each side of the stadium providing differing views.

Camera 21 in a preferred embodiment will televise boards 15,17,19 which will be transmitted to the local population in an ~Inçh~nged manner.

Camera 20 will in this ~lGÇt;l,~d embodiment lldllSlllil a feed to an international audience. Camera 20 is equipped with orientation sen~in~
means which preferably comprises one or more of the following:
pan measurement means 24;
tilt measurement means 25;
zoom measurement means 26; and focus measurement means 28.

Suitable sensc,l:j may CO~ G the Virtual Reality FnC~er from RADAMEC EPO, Bridge Road, Chertsey, Surrey KT16, 8LJ, Fn~]~nr~

Depen-lent on the allowed mobility of the camera only one, several or all of these may be required. For example, if camera 20 is fixed in pan and tilt and focus and can only zoom, as in the case of some remotely controlled .. ~ ~1 C~llle~dS then only the zoom ~dl-~t~r need be mea~ur~l.

Most cameras in sports stadiums can zoom, tilt and pan and it is ~sumec~ that these parameters are measured for each camera as now 20 explained. The focus is assumed to be fixed but in similar .l~anl~r the parameter could be added if required.

Figure 2 shows the video image as seen by a viewer and in particular by the operator of the equipment. The camera 20 is zoomed, 25 panned and/or tilted to ~centre~ the billboard 14 in a suitable position and at a reasonable size. With reference to Figure 7 each billboard is then - viewed at a receiver and its position is marked preferably by using a touch screen 700, or keyboard mouse 702 and marking the four CO~ i. The positions are stored in a store 704.

For billboards higher in the stadium such as 30 -(Figure 3) a correction factor for the camera may be stored dependent on the tilt position of the camera.

S Each billboard position is stored in store 704 together with the camera parameter information at the reference position for the camera 20 obtained from the camera parameter information which is correct at the time that the billboard position is stored.

The following procedure is ~r~r~ldbly ~ t~A for each of the Callle.~LS and for each of the target billboard:

1. Point the camera at the target to obtain a stable lln--crluA~1 view of the target. Adjust the zoom to get a large view of the target yet keeping the whole target within the field of view.
2. While the camera is not moving trigger an ~r~ tion device, to grab a picture of the target, as well as the co.,..,~,o"ding of the sensors.
3. Mark the corners of the target, on the video image.
Preferably a corner detector is used to pinpoint the corners of the target at sub-pixel precision.

This camera parameter information is obtained (Figure 6) from the 2~ sensors mounted on the camera and the camera movement is referenced to a first or fixed reference position for each parameter. The movements of the camera are sensed and the signals are fed into a combiner circuit 24 and then to a transmit buffer 36 from which the combined video and position data signals are transmitted.

W O 97/09823 PCTtGB96/02226 During set up, at the receiver (Figure 7) the receive buffer 706 receives the signals and feeds these to a splitter 708. The video signal is stored and delayed in a suitable store 710 and the camera parameter data is extracted and stored in store 712.

In set up the VDU 700 is used to mark each billboard that may require replacement. The camera 20 is p~nn~-ll etc to move each billboard into a suitable position on the screen and its position is recorded in the billboard store 704 together with the camera ~U~ obtained from store 712 via proce~or 714.

A repl~e-ment billboard store 716 stores a plurality of repl~cem~rt billboards and these are se-lect~hle to be able to ~/lace the ori~in~l billboard.
The repl~cement billboard is in operation inserted into the video signal in a combiner 718 to provide a modified output video signal 720.

The setting up procedure can also identify billboard locations and 20 camera parameters for several cameras by storing a camera ID from a source 30 (figure 6). Thus, billboard position store 704 will store s.,~le lists of billboard data for each camera.

The operation of the system will now be described with reference 25 to a single billboard and a single camera 20.

With reference to Figure 4, it is assumed that billboard 14 enters the field of view in an enlarged form on the left-hand side of the screen as camera 20 pans following zooming from the Figure 3 position.

The camera orientation data is constantly being received by the receiver and the processor 714 will constantly match on a pixel by pixel basis the video image with the known billboard position stored in store 704. As soon as the billboard a~ea,~ in the video image the pixels 5 reprçsenting the billboard will be identified and the replacement billboard pixels which relate to those pixels will be substituted in the combiner 718.
The delay will be minim~l since the identification of the pixels is by an address correlation ~locess which will be virtually i..~ ous.

After a period of time the camera sensors may drift and in this case the repl~r~ln~-nt billboard may not exactly align with the ori~in~l This may only be by one or t~,vo pixels and may not be discernible to the viewer. To corr~ct this two solutions are possible. Firstly, the billboard position can be m~ml~lly restored periodically at a suitable time, for 15 example, when a camera is not active. This ~e~luir~3 the c~operation of the ope,~or.

Secondly, a comparison can be made on a pixel by pixel basis of the billboard ~g~in~t an original stored billboard and an adjustment of the 20 reference camera parameters can be made in billboard position store 704.
This process can be done automatically at either set intervals or when the processor 714 has a suitable time slot.

The essential steps of a preferred recalibration process are to 25 perspectively transform the current video image using the camera data to provide an estimated transformed model. A stored image of the billboard is then compared with the transformed model to provide a residual video field. The residual distortion between the transformed model and the residual video field is resolved to provide updating i"fol ",a~ion for 30 updating the estim~te~l transformation and to thereby provide a calibration correction factor for recalibrating the position of each billboard in the store in accordance with the camera sense information.

~ The repl~eement of each billboard is accomplished by use of the 5 processor 714 (Fig. 7) and the various parameter and billboard stores using a~pl-opl;ate software programmes as now described in rnore detail.

Figure 8 desel;bes the complete process which allows to d~te..~ .e the position of each billboard in the camera's field of view, and render the 10 cc,~ ,dillg par~t of the billboard into the frame buffer. Since the r~n-l~nn~ and later the ~Ill~osiLing of the graphics buffer with the video buffer by means of ~ l,r~-,la-key are known art, we will col~ dle on the billboard position de~l--i-laLion with l~,fc,~lce also to I;i~;u.~s 6 and 7.

15 At the l,cg;.. ~ of each video field, the pan, tilt, zoom and focus sensors (24,25,26) are read 800. These values, combined with billboard data from billboard setup data store 704 and camera data from camera intrinsic parameters store 712, enable the detection and l~co~ ion of all billboards in the camera's FOV, independently of the video signal. The proces~ing of Fig. 1 consists of a loop on all billboards (m) 802,804. For each billboard, its setup data is retrieved 806 from billboard setup data store 704 and used with camera intrinsic parameters 808 to co---~ule the perspective ~ ,sro."-ation 810 from billboard m to current field. The replacement billboard information is then stored (812) in a frame buffer.
Fig. 9 describes the billboard setup data store 900 which consists - of a separate record 902.. 904 for each billboard in the arena. Such a record consists of a static image 906 grabbed in favourable conditions and of the corresponding static setup data 908. The record also consist of 30 dynamic setup data 910 which is computed using the image proce~ing means in a process known as dynamic re-calibration which has been briefly described above and will be further described with reference to Figure 11. An alternative procedure providing static and dynamic calibration is described with reference to Figures 20, 21 and 22.
Fig. 10 describes the setup data (either static or dynamic) 1000 for a single billboard. It consists of the sensc,~s,' re~lin~s 1002 at the setup .c~e, the billboard quadrilateral vertex 1004 coo~iir,~l~s and ~e time-code of the setup in~t~nre 1006.
The metho~ of dy~lall~ic re-calibration can be ~ ;"~A as follows:

Due to Se-~SGl:i' drift and in~r~llracies~ a final calibration table and other practical ~ s, it is i~ ,os:,i1)1e to predict ffle exact ~ tioll of all 15 visible billboards at a given j"~ Ho~ , at many video fields, a billboard's visïbility may be such that an exact geo."ellic position correc~ion can be pe- rO, ~~led. Since that position is closer both temporally and sp~ti~lly to the subsequent video fields, it is l"~,r~lal)le to rely on that"luck shot" by predicting the billboard position relative to its sensors' 20 re~-ling~ and exact quad coordi..a~s. Consider for eY~mp!e a billboard which exits the field of view due to camera p~nnin~. Having a luck shot while it is still highly visible, allows the smooth traclcing of the billboard by sensors only, when its visibility does not allow any image processing means to be applied.
Fig. l l depicts the flow diagram 1100 for perspective l,~,.sÇol.,.ation cor-l~ulation. A setup data selection logic 1102 selects either the static 1103 or the dynamic 1105 setup data from the setup data store 806 as described above This setup data, together with camera 30 intrinsic parameter is used to compute a sensor-based prediction of the perspective transformation 1104, independently of the video signal.

A dynamic re-calibration 1106 based on image pro~,s~ing means is then applied to the prediction. It utilizes the video 1108 and chroma-S key 1110 signals as well as the billboards model image 1112 from the setup data store 806 (Fig. 8) Based on a quality factor derived from the image proces~in~ means, either the sensors-based 1118 or the corrected tran~ ,ation 11 16 are output. If the e~ P~ quality of the geometric correction is high, then the dynamic setup data is ~ t~1 1114.
Figs. 12,13,14 describe the sensor-based prediction of billboard coordinates in the video field. Such a prediction utilizes the sensors reading as well as the camera intrinsic ~a.~.llcters. These parameters are described in Fig. 12 and have to be tabulated for a dense s~mplin~ of the (zoom,focus) space. The mP~ning of these p~ ctcrs is clear from Figure 13 to which reference is now made.
Let the set of measurements given by the pan, tilt, zoom and focus sensors be represented by the vector (P,T,Z,F). The tilt angle is ~ nm~l to be relative to the horizon.
Consider an object point whose image at some setup in~t~nce iS, at frame-buffer coordinates (xs, y5). Let also the sensor mca~ul~,ulent vector at that instance be (Ps~ Ts, Zs, Fs)-At another instance, the prediction instance, let the sensor measurements vector be (Pp, Tp, Zp, Fp). It is required to predict the - location of the object point in frame-buffer coordinates (possibly out of the actual frame-buffer), (xp, yp).

To enable the procedure we define the setup rotation matrix as shown at 600 and the prediction rotation matrix is defined as shown at 602.

Then, the perspective transformation matrix between the two image-plane coordinate systems is given as shown at 604 and 1402 (Fig. 14).

Rsp is a 3*3 matnx with row and column indices ranging from 0 to 2. Rsp~iJ~;] ~erlotes the terrn in row i, column j in the m~tri~ Thus, given the setup image-plane coo~li.~aLes of the object point (u5, v~), the preAict~ location of the object point in image-plane coordinates (up. vp) is given as shown at 606, 1404.

Image-plane to r~ .c buffer cooldil-ate l,~ ro~ lion is achieved as shown at 608, 1406. Aberration co~ ;on is acl~i~ cd as shown at 608, 1406 (Fig. 14) to provide pre~ ted frame buffer billboard coordinates and perspective L~ ro~ alion data.

An effective way of denving these parameters for a specific (zoorn,focus) pair is described in ~J. Weng et al., Calibration of stereo cameras using a non-linear distortion model, IEEE 10th Intl. Conf. Pattern Recognition (1990), pp. 246-253]. The image proces~ means for geometric correction, which allow also the process of re-calibration, is now described with reference to Fig. 1~.

2~i The image processing rr.eans for geometric correction of sensors-based prediction are based on the differential method for motion estim~tion [C. Cafforio and F. Roca, The differential method for motion estim~tion, in: T S. Huang, eg., Image sequence processing and dynamic scene analysis, Spring, Berlin, 1983, pp. 104-124]. Let C be the current video field and let M be the static billboard setup image, perspectively transforrned according to the sensors-based prediction. Here we consider only lnmin~nce images. Ideally, M and C are identical within the support of the billboard quadrilateral. Actual differences may incl~l-1e:
~ Occlusion present in C but not in M.
~ Geometric errors due to sensors and intrinsic camera parameters errors.
~ T l-min~nce changes.

Neglectin~ for the moment any dirr~ ce which is not due to geometric errors, consider a point (x,y) inside the support of the billboard qn~ teral Let (p,q) be the local geometric error then we may write for the l--min~nf~ signals of the respective images:
M(x + p,y + q) = C(x,y) Under the ~ plion that the error is small, one may write a lS Tailor series exp~n~ion:

M~fx + p,y + q) = M(x,y) + P dY + q d--+ (Se~ond order terms) Neglecting the second order terrns and denoting the spatial derivatives dM/dx = H
dM/dy = V
we obtain C(x,y) - M(x,y) = pH + qV

- 25 Also denoting the differences C(x,y) - M(x,y) by D we obtain D = pH + qV

The equation above holds, locally. For a global billboard solution, and small error as~u",p~ion we may use the perspective model [G. Adiv, Del~. Ini~ g Three-Dimensional Motion and Structure from Optical Flow Ccne.~.led by several moving objects, IEl~E Trans. Pattern Analysis and Machine intelligence, 7, pp. 384 - 401, 1985].

alx + ~y + a3 p(x~) =
a7x + a~ + 1 a4x + a~y + a6 a7x ~ ayy + 1 The Co~ffir~ t~ al,.... ,a8 are CO~ JI~1 by ~ E the following e~p~ession:

~ (D(x,y)-p~x,y)h~x,y)-q(x,y) ~X~y))2 (~

Now, the perspective lldll~rc~ tion rnatrix (based on sensors' prediction) is multiplied by:

al ~4 a7 a~ a~ a8 23 a6 1 2~ The matrix obtained can be considered to be the updated prediction of billboard perspective.
In a practical environment the following considerations may apply.
~ Occlusion may cause major problem in this formulation, since if pixels from occluding and moving objects participate in the rninimi7~tion of the e~ ,ssion above they might bias the solution significantly. Preferably, such pixels are discarded from processing by using cl,ro,.la-key panels. A
key signal output by a chroma-keyer, is ~lefe.~ly utili7~1 to discard these pixels.
~ T llmin~nce variations can be minimi7~ by pre-proce~ing - the ~;U~lenl video field, using histogram m~ .hirlg techniques.
~ The prediction-correction process may require 2-3 iterations to converge.
10 ~ Noise i~ u~ y and conve,g~ ce can both be e,.~ ~l by pre-smo~nl.ing the images.
Thus the billboards 14 etc are in accordance with the ~l~,S~
invention cllr~,lla-key boards and ocr3-~ion is by colour ~ ;on using the normal chro-,/a-key techniques. These te~-hni~lues will enable 15 perfect occlusion providing that the players do not wear any colour which is the same as the boa~d. Ihis may not always be possible and it is proposed in acco~a~ce with a particular emb~iment of the ~ s~
invention to use boards which can rotate or otherwise change to a second or third colour. For example, three colours may be blue, green and red 20 which may be s~o-lecte~l when the colour of the players strips are known.

Al~~ Li~ely if it is le~luil~,d to display a billboard in an area of the pitch or a surrounding area then such an area must be selecte~l to be of a known colour which can then be recorded in the chroma-keyer as a 25 chroma-key colour.

In a preferred embodiment chroma-key apparatus can comprise the Ul,TIMAl~E-7 digital video image compositing device from - ULTIMArrE Corp., 205~4 Plummer St., Chatsworth, CA 91311, USA.

W O 97/09823 PCT/GB96/02226 The ~ king colour can be sele-ctecl between blue, green and red.
In order for the chro--la-keyer to calculate all pararneters nt~c~ss~ry to ~elroll,l proper image compo~i*n~, the system requires a sample of the background colour as reference. This step can be done ~-ltom~tically by S sc~nnin~ the image and detecting the purest and brightest colour.
Advanced chroma-keyers enable the user to m~nll~lly select the area to be sampled.

In a particular embo~ it is ~r~osGd to use a ~lh~ ed 10 cl~r~ a-key panel. Calibration of the ca,.-e.a S~ SC~l~, can then readily beaccomrli~hecl by co.-~r~ 0!~ of ~he ~ on a pixel by pixel basis. ~he L.-l on the billboard panel should ~lGre.~bly have critical di.l,e"sions less than ~nti~ip~te~ se~sol~ error ~projected to world cool~lina~s).

In :~u.-.. ~ ~, the above system can O~IG even in eAL ~.~lcly poor weather conditions since the electronic procPs~in~ ;Uil~y kllOWS exacdy where each billboard is ~ tt~ll and does not rely on any analysis of dle video image to detect the billboard. In the event that the video image is so distorted that recalibration cannot be carried out with re~on~ble 20 certainty, then the original camera parameter settinf~.C can continue to be used since the video image as viewed will be of poor c~uality and thus the viewer will not notice an error of one or two pixels in the positioning of the replacement billboard which will require to be displayecl in an equivalent quality which m~t~hes the poor quality video image.
In a further preferred embodiment of the present invention, the problem addressed is that of having billboards situated in dirre,~
positions in a stadium as shown in Figure 16.

In such conditions the lighhng of billboards, 1,3 and 5 will be -different because of the location of lights 7, 9 and 13. Also this li~htin~
can change all the time during the game.

If such billboards are chroma-key boards all of the same colour S then the billboards will all appear to be slightly different colours due to the different lightin~ conditions.

A fixed adj~-stme~t of a global b~ in~ colour might result in partial object background separation by the cllro-..a-keyer.
In the ~ ,sen~ invention it is proposed to provide spatial ~d~pt~tion of the b~ ing colour map so that the chl~,ll,a-keyer can co~ y recognise each billboard. This can be provided by storing in store 704 (Figure 7) a spatial map providing information relating to the colour of 1~ each chlol"a key board.

Thus, the chroma-keyer will co",l~ale the colour in each video location with a specific colour associated with the billboard in that location.
In a preferred embodiment the locations of the billboards may be identified by "p~inting" a slightly enlarged box s~lllu~ g the billboard to identify the location. Such boxes are identified as 1', 3' and ~i" by dotted lines in Figure 16.
The system will track the backing colours over time and therefore - will continuously update, to ensure correct identification, once correctly set up The operation of the system is as follows.

Firstly with r~,rt;,~.-ce to Figure 17, minimum and maximum levels are set for U and V. These should be wide enough to encompass all billboards which are reasonably lit.

S Then for each billboard, as lighting conditions change, an adjustment can be made to its stored values, as shown in Figure 18, which ~C~llmes occlusion of billboard 1 by an object 13. An inner box 1" is defined to ensure only pixels from within 1 are considered. Most c~ linf~ pixels can be discarded as these will be of a ~Lr~ L colour.
Then all pixels (YuV) inside FOV and billboard quad 1 " are n~u~d and an addition to the average (which is b~kin~ colour average W over billboard) is made if :-UmiD S U _ Um~c or Vmin ' V 2 V~

l['he inventors have recognised a further problem which arises from the use of chroma-key billboards in a ~t~linm. Due to the variable liS~htin~ as described above, each billboard will appear on the video image 20 as a slightly different colour. In order to transmit correct occlusion information it iS ~-ec"-s.C~.y to ~ lalllit an occlusion map for each billboard.

In accordance with a preferred embodiment of the present invention 25 it is proposed to transmit, for each billboard, a perfect background colour and to then allow a chroma-keyer in each receiving station to introduce the occluded portions by normal chroma-key procedures.

Consider now the billboard arrangement as shown in Figure 16.
30 Each billboard 1,3 and 5 will, because of its different lightin~ conditions, appear to be a different colour even though this colour may be within the maximum and minimum limits as set out in Figure I7.

In accordance with this preferred embodiment of the present S invention, the tr~ncmitting apparatus (see Figure 6) will transmit a perfect chroma-key colour within the area of the billboard and will also llal,slnit the coordinates of the quadrilateral formed by the billboard.

In this way the receiving station only has to decodç/extract the 10 quadrilateral coordinates of the billboard and then within that q~ ril~tP~ral replace those pixels which are the perfect chroma-key colour by the repl~rem.ont billboard. Those pixels which are not a perfect c;hlo.,.a-key colour are not replS~c~cl In accordance with this system it is not n~.ce.ssZ~ry for the cLlu~{la-keyer at the remote receiving station to be able to recognise dirr~
billboards and to have to store different ch~ la-key values for each billboard. Also it is not ntocçcc~ry to transmit any occlusion inforrnation since occlusion by the chroma-keyer will be relatively simple at each 20 remote location.

With reference to Figure 7, the billboard position and b~r~ing colour store 704 knows the position of each billboard and a control output 7042 from the store is used, in combination with the video output 7102 to 2~ provide inputs for a b~cking colour processor 7044 which can change the colour of the billboard within the coordinates provided by store 704. The - output of processor 7044 is used to control a backing colour store 7046 which changes the colour of the billboard within the required coordinates and can also provide the coordinates to the video output 720 for the 30 remote receiver. These may be transmitted by a standard video data tr~n~rnis.sion system.

An exemplary remote receiver is shown in Figure 19. Video data is received at receiver buffer 1900 and split and delayed 1902, 1904.

Billboard coordinate store 1906 stores the l~ ..ill~l billboard coordinates and in combination with graphics generator 1908 and re}~ .mçnt billboard image store 1910 provides an output signal to a combiner/cllro.l.a-keyer 1912 to produce the desired, ocr~ ded billboard 10 on the screen.

With reference now to Figures 20 to 22 in a further embo~lim~ont the set up data stored in store 704 is modified prior to any event being televised.

S The modification comprises the addition of dynamic set up data as well as the static image and static set up data shown in Figure 9.

The additional data may be used in~te~1 of the dynamic recalibration set up data 910 shown in Figure 9 or could be used in addition.

In a ~r~r~ c;d embodiment it is ~c~lme~ that the additional data is used in~tP~rl of the dynamic recalibration ~loce~lu-~, and this is now described.
As an introduction, the problems associated with replacing billboards with virtual billboards are ~ cllssetl. The same problem is identifying the position, size and perspective of the original billboard and then replacing this with the virtual or replacement billboard.
In a static camera situation there is no real problem once the original co-ordinates have been recorded providing that the camera sensors do not drift subst~nti~lly over time.

However, the inventors have found that during rapid panning or tilting of the camera the co-ordinates 908 of the replacement billboard as - recorded in the store 900 do not coincide with the actual position of the billboard in the stadium or venue. This is because the camera sensors exhibit a degree of hysteresis. This can be compensated for by the dynamic recalibration process already described but this may not be practical in some circnm~t~nceS such as during rapid p~nning with substantial occlusion of the target billboard.

The hysteresis could possibly be countered by a simple ~er~llLage S error built into the movement of the camera but this does not produce very good results because it does not take into concideration the camera angle with respect to each billboard nor does it take into account the variability in the camera ~"lcter sensors with angle.

In the ~lcse.lL invention the-~rol~ in an all~ e embo~li.. enl, in additio;n to static billboard set up data the billboard set up data store 900 stores data for each billboard for each camera at least in relation to left-right pan 2002, right-left pan 2004, up down tilt 2008 and down-up tilt 2006. The data is obtained and stored as now herein described with 15 l~,f~ ce to Figures 20 to 22.

Figure 20 shows the store 900 modified to provide, in addition to the static image data for billboards 1 to M and the static set up data for billboards I to M four further sets of data for each billboard 1 to M and 20 these are mnlsirlieA to provide this data for each camera.

For each camera the position of each billboard is recorded with the camera p~nning from left to right 2002 and for right to left 2004 The panning speed may be selected as the normal speed for the event being 25 televised Thus for example for horse racing it could be low but for motor racing it could be higher. The position of each billboard is then recorded with the camera tilting upwards 2006 and then downwards 2008 across each billboard.

;For each measurement the camera zoom and focus are preferably set at a known level at which the billboard being analysed is in a reasonable view at a reasonable size. The zoom and focus could for example be the same as that during the acquisition of the static set up data for each billboard so that a direct comparison with the static set up data S can be made. In that case only an error correction figure may need to be recorded.

It is ~ fe.~ble not to select too high a ~ g or tilting speed ber~ e at very high speeds the billboards will in any case be blurred and 10 lLe.efol~ accuracy of repl~ment will not be an issue. This procedure is followed for each billboard for each camera and the data is then used during the event to correct the position of each billboard as the camera pans or tilts.

It may be seen that each billboard will be viewed at a dirr~
angle by each camera and also that the output of each of the sensors on each camera may vary dependent on the angle through which the camera must turn to view the billboard. By recording the static data and data relating to ~nning and tilting in both directions the repl~mPnt billboard 20 will be accurately positioned in the exact position of the original or real billboard both for static shots and when the camera is moving.

Dynamic recalibration during the event as previously described will ensure, except during very fast camera movements with large occlusion, 2~ that the replacement billboard is correctly positioned but the use of static and dynamic set up data will also ensure this unless the camera sensors drift substantially during an event. Thus providing that the camera sensors are of a reasonable quality from the point of view of draft they can be of a variable quality with respect to accuracy during p~nning and tilting. By 30 careful selection of camera sensors extremely accurate sensors are therefore not required since any variation with respect to camera movement is compensated for by the storage of dynamic set up data.

The data is obtained as described with reference to Figures 21 and 22 as follows.

Once the static data relating to each billboard image and its static set up data (906, 908, Figure 9) has been o~ cd, sequence 2100 is star~d and the camera is p~nnfYI 21021by the o}~lalor at a desired speed relative to the normal p~nning speed.

The s~,.-so-~ indicate the direction of pan 2106 and ~le~. .~ on the direction the dynamic data is stored in store 2002 or 20()4 in steps 2106 or 2108 by sele~ion of that store. The sequenr~ for both L,R and R-L
1~ stores 2002 and 2004 is similar and will be described for the L-R store but using ,~,fele.lce numerals for both stores.

As the camera pans L-R each billboard is identifit-11 from data stored in store 906 step 2110; 2112. The system asks if the billboard has previously been recoded dynamically step 2114, 2116 and if so it returns to the start of the sequence and repeats steps 2104 2110 until it finds a billboard that has not been dynamically sc~nn~l Once a new billboard has been found the position (co-ordinates) of the billboard during p~nnin~
is recorded and co.llpaled with the static billboard parameters previously stored (908) in s~ep 2118, 2120. Any error is computed (step 2122, 2124) and the errors are stored in L-R and R-L pan stores 2126, 2128 for the billboard. The system asks if all billboards recorded in store 908 have been dynamically s~ nne-1 both for L-R and R-L ~steps 2130, 2132). If not the sequence is continued until the last billboard has been dynamically scanned and then the program is termin~l 2134, 2136.

W O 97/09823 PCT/G~G~'~2226 A similar program sequence shown in Figure 22 is provided for tilting of each camera. Obviously if the cameras are not either allowed to tilt or are unlikely to be tilted to any extent then this seAIl~enre and the recordal of data in stores 2006, 2008 may not be n.oces~ry The sequence is started 2200 and each camera in turn is tilted 2202 and the direction of tilt determined 2204 by the camera sensors.
Depen-lent on whether the c~"~ is tilting up or down dynamic set up data is stored in stores 2006 or 2008 in steps ~o6, ~o8. Both s~-.ences 10 are similar and only the sG~ e-~f~ tilting the camera down will be desc.;l,ed with l~,f,.~,nce then to both se~-e~ s.

Pach billboard is ider~tifi~ ~l0, ~l2 from the static image data and also from the camera parameters ~spe~ri~lly where all real billboards 15 are the same. The program interrogates the billboard data step 2214, ~l6 to see if the billboard has already been interrogated. If it has the program lG~ S but if not the co-ordinate data of the billboard during tilting is co~ ~Gd with the static data step 2218, '~ 0. The error, if any is co,.,~uLed step 2122, 2124 and stored in the stores 2006, 2008 (Figure 20 20) step 2226, '~ 8.

The program then interrogates stores 2006, 2008 to see if all billboards have been dynamically interrogated for tilt errors in both up (step 2230) and down (step 2232) and if so ends the program steps 2234, 25 2236. If not the program continues by commencing at the start of the sequence until all boards have been interrogated.

Normally camera zoom and focus will not require the same type of dynamic set up data to be stored. However, if particular camera 30 aberrations are known then these may be compensated for by use of similar dynamic set up data.

The dynamic data stored in stores 2002-2008 may be used in~k~
of or in conjunction with the dynamic recalibration data obtained as 5 described with reference to Figure 9. Usually however the dynamic set up data will obviate the need for recalibration during most types of event.

During use the system knows by reading the carnera sensol~
whether the billboard is being viewed in a static ~ f-r or is being 10 panned past L-R or R-L or tilted past UP or DOWN. In such cases the position of the billboard is taken from the static data store and ~en if ~JA "' ~ g or tilting is occurring, the ~~c~.ss ~ . y error col l ~ctions are applied.
Once camera movement ceases the static billboard ~ -. . are reverted to.

Claims (8)

THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE
PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS:

1. Apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means receiving inputs from sensors attached to a TV camera and camera lens, said sensors being operative to measure the Field of View (FOV) of the TV camera relative to a known reference position, including image processing means for processing video signals generated by the TV camera, in which said processing means includes calibration means to periodically automatically calibrate the sensor measurement means and in which the calibration means includes fine adjustment image processing means including means for perspectively transforming the current video image to provide an estimated transformed model, means for storing a replica of the billboard image, means for comparing the perspectively transformed video image with the stored replica to provide a residual video field, means for solving residual distortion between the transformed model and the residual video field and means for updating the estimated transformation by means of the residual distortion to provide a calibration correction factor.

2. Apparatus as claimed in claim 1 in which said means for storing the residual distortion includes analysis means based in the spatial derivatives of the current image of the transformed model as well as means for analysing pixel by pixel the image differences of the transformed model and the current image.

3. Apparatus as claimed in claim 1 in which the fine adjustment means includes means for carrying out the fine adjustment in an interactive manner.

4. Apparatus as claimed in claim 2 in which the pixel by pixel analysis means includes means for discarding pixels using colour variation.

5. A method of automatic replacement of a billboard in a video image said method comprising the steps of:
recording the Field of View (FOV) parameters of a TV camera in a first reference position by means of sensors attached to the camera and the camera lens;
storing the image viewed by the TV camera in said first reference position;
measuring the movement of the camera relative to the first reference position and recording the measurements and periodically automatically calibrating the motion measurements by comparison of the present video image produced by said TV camera with the stored video image in said reference position;
calibrating a difference signal for recalibration of the motion sensor measurements and said first reference position and replacing a real billboard in said cameras field of view with a virtual replacement billboard in accordance with said recalibrated first reference position in which said step of periodically automatically calibrating the motion measurements includes a fine adjustment image processing step including perspectively transforming the current video image to provide an estimated transformed model, storing a replica of the billboard image, comparing the perspectively transformed video image with the stored replica to provide a residual video field, solving residual distortion between the transformed model and the residual video field and updating the estimated transformation by means of the residual distortion to provide a calibration correction factor.

6. A method as claimed in claim 5 in which said step of storing the residual distortion includes analysis based in the spatial derivatives of the current image of the transformed model and analysing pixel by pixel the image differences of the transformed model and the current image.

7. A method as claimed in claim 5 in which the fine adjustment step comprises carrying out the fine adjustment in an interactive manner.

8. A method as claimed in claim 6 in which the pixel by pixel analysis step includes discarding pixels using colour variation.