WO2000010328A1

WO2000010328A1 - Method for operating in real time aerial photographs obtained by a high rate sensor, and implementing device

Info

Publication number: WO2000010328A1
Application number: PCT/FR1999/001957
Authority: WO
Inventors: Franck Ripert
Original assignee: Thomson-Csf
Priority date: 1998-08-11
Filing date: 1999-08-06
Publication date: 2000-02-24
Also published as: FR2782427B1; FR2782427A1

Abstract

The invention concerns a method for transmitting to the ground the video pictures shot at very high rate (> 1000 Mbits/s) via a low rate Datalink service (about 10 kbits/s to 10 Mbits/s), which consists in storing (4) on board the reconnaissance aircraft (1) in a high capacity flash memory the videotape, and in transmitting to the ground (in real time or slightly delayed) only the video pictures showing zones of interest.

Description

METHOD FOR REAL-TIME OPERATION OF SHOOTINGS

AERIALS OBTAINED BY A HIGH SPEED SENSOR, AND

IMPLEMENTATION DEVICE.

The present invention relates to a method of real-time exploitation of aerial shots obtained by a high-speed sensor, and to a device for implementing this method.

Originally, the aerial reconnaissance procedures used cameras, and the films taken during the missions were exploited after the return to the base of the reconnaissance planes. Subsequently, we felt the need to have recognition information more quickly, in order to be able to use it without wasting time. For this purpose, it was necessary, on the one hand, to store this information on a support which can be read much more quickly and more easily than a film. To this end, electronic memories with very large capacity have been developed (based on “flash” memories in particular). On the other hand, to transmit the information read in these memories from the flight recognition aircraft to the ground, transmission means commonly known as "Data Link" have been developed. It is now estimated that the lifespan of information acquired during an intelligence mission (i.e. its period of validity) is generally between six and twenty-four hours. It is therefore essential to seek to minimize the time necessary for the transmission of information. The traditional transmission by Data Link is a beginning of solution of this problem, because it makes it possible to have in quasi-real time (after reading of the storage memory) of the collected information. Such a link is suitable for the use of optronic sensors which deliver digital information. However, it does not exempt from keeping the recording on board, in order to avoid the risk of total loss of images which can sometimes result from poor transmission conditions. The known “Data Link” links have very different bit rates and ranges. These bit rates are generally between 10 kbits / s and around 300 Mbits / s, and the ranges are between 100 and 400 km. The digital information rates of current electro-optical sensors can be very high (almost 2000 Mbits / s). In certain aerial reconnaissance applications, it is possible, for example, to use, in the visible field, two matrices having a definition of 1928 x 1084 pixels with two outputs of 38 pixels each coded on 12 bits, the bit rate of the two sensors being 1824 Mbps. Of the- bit of known data links does not allow real-time transmission of such data rates without loss of information. To remedy this, one could consider compressing the information. A compression ratio, without loss of information, of about 2 is commonly accepted. Beyond this value, digital video compression algorithms do not guarantee the required image quality. Consequently, even by using information compression techniques, the known methods do not make it possible to transmit at high speed (greater than 1000 Mbits / s) the aerial reconnaissance images without degrading the quality of these images. In addition, even if we were satisfied with a flow of about

300 Mbits / s, current equipment does not allow this speed to be constantly maintained without transmission errors, in particular for relatively large transmission distances (greater than 100 km). Indeed, to ensure transmission without errors, it is necessary to implement an error correction protocol which decreases the bit rate of useful information, this reduction being all the more important as the transmission conditions are bad, as the distance transmission speed is high and the useful information rate is high, which means that even when the overall information rate (useful information + error correction + service data) is announced for 300 Mbits / s, the rate of useful information (video + embedded data) is actually much lower.

The subject of the present invention is a method of real-time exploitation of aerial shots, obtained by a sensor with a high information rate (which may be greater than 1000 Mbits / s) which can be implemented with Data. Relatively low speed link (significantly less than 300 Mbits / s), without loss of information, with high reliability, even for long transmission distances (which can be greater than 100 km).

The present invention also relates to a device for real-time exploitation of aerial shots obtained by a high-speed electro-optical sensor, a device which is relatively inexpensive, compatible with practically all types of sensors and Data Link. .

The method according to the invention is applied to a mobile connected to an operating base by a “Data Link” type link, this mobile being provided with video means making it possible to film a lens and means for storing the video thus obtained, these storing means being able to be read during their recording, and it is characterized in that during the recording of the video, areas are identified of interest among the shots taken, these markers being recorded with the corresponding video, and which one reads and transmits to the base via the Data Link only the video of the areas of interest thus identified, to exploit basically, in near real time, these areas of interest. The identification of areas of interest can be ordered on board the mobile, or from the base. The present invention will be better understood on reading the detailed description of an exemplary implementation, illustrated by the appended drawing, in which:

FIG. 1 is a simplified diagram of an exemplary device for implementing the method of the invention, ^“ FIG. 2 is a block diagram of the on-board part of the device of FIG. 1,

FIG. 3 is a block diagram of the ground part of the device of FIG. 1, and

FIG. 4 is a flowchart of an example of implementation of the method of the invention.

The present invention is described below with reference to aerial reconnaissance missions, but it is understood that it is not limited to this single application, and that it can be implemented when one has to transmit data, captured at high speed, by a transmission channel having a speed significantly lower than that of the data to be transmitted, and that only a "slice (area of interest for example) of this data is to be exploited in real time or almost real, and that this "slice" must be transmitted without loss of data. This is in particular, but not exclusively, the case for the transmission of filmed images of satellite observation or interplanetary, underwater or medical (intra-corporeal) exploration. Data can be captured by any suitable type of sensor (visible and / or infrared optoelectronic camera, laser, radar, etc.).

In FIG. 1, an aerial reconnaissance airplane 1 has been schematically shown flying over and filming a terrestrial area 2. This airplane 1 comprises, inter alia, a sensor 3, in this case a video camera, connected to a memory 4 with dual access (write and read). This memory 4 can be read at any time, without interfering with the storage of the data arriving from the sensor 3. It is advantageously a solid state memory, for example of the “flash” type. The aircraft 1 also comprises a transmission-reception and management device 5 (described below with reference to FIG. 2) communicating with a corresponding device 6 (described below with reference to FIG. 3) located in a 6A ground operating base. This communication is done by a radio link 7 (or similar) called Data Link. As a variant, if the distance between the airplane 1 and the base on the ground is greater than the range of the link and / or if obstacles obstruct the link, the link 7 can use one or more relays 8, as shown in dashed lines in Figure 1. This (or these) relay (s) can (can) be airborne (s) or arranged (s) in any suitable way. Link 7 (or 8) has a data rate which can be much lower than the data rate of sensor 3. The rate of sensor 3 can be around 1000 Mbits / s or more, while that of the link, in particular that of the downlink (towards the base on the ground) can be approximately 10 Mbits / s or even much less, which makes it possible to use conventional and inexpensive equipment for the transmission and reception of data. Of course, this is only possible if only a few (those of areas of interest) of the many filmed images of the aircraft 1 are to be transmitted in flight.

As shown in FIG. 2, the device 5 comprises a formatting device 9 connected to the sensor 3 and to the memory 4, as well as to a device 10 for lossless data compression and for sub-sampling. The memory 4 is connected by an encryption device 11 to the on-board part 12 of the Data Link transmission system. A management processor 13 is connected to elements 3, 4, 9 and 12.

The sensor 3 communicates to the device 9 the useful video data (filmed images proper) and data relating to the aiming position (geographical position of the aiming line on the ground). The formatting device 9 is connected to a display device 14 intended for the pilot or an interpreter (operator) who has taken place on board the aircraft.

The functions of the various elements in FIG. 2 are as follows. The formatting device 9 reads the digital video at the output of the sensor 3, "cropping" it if necessary (to remove redundancies due to overlapping images), as well as the associated control information ("Data Valid", frame and line synchronization), which constitutes said useful video signal. On the other hand, it sends to the memory 4 the video, formatted as specified above and including the position of the line of sight, in order to make it record there in three different formats (formats of the “7023” type). These three formats are:

-a): "full resolution" format intended to be used at base 6A after the airplane has returned to this base. This format corresponds to raw digital video, that is to say that has not undergone any manipulation (selection of area of interest for example) on board the aircraft. This formatting makes the useful video, such as coming from the sensor 3, undergo compression without loss of resolution (compression in a ratio 2) carried out using the device 10. In addition, this formatting also consists in embedding in the video digital information necessary for ground operation, for example: position of the line of sight, size of the sensor, time and date of the shots, as well as various other parameters that can be measured in flight (altitude, weather, etc.) .), and finally marks (produced in flight under the control of the pilot or an interpreter) delimiting one or more areas of interest.

-b) "full field" format, also intended for base 6A. The formatting device 9 performs a sub-sampling of the useful video arriving from the sensor 3 (the sub-sampling ratio depends on the scanning frequency of the sensor), and embeds in the digital video the same information as that relating to the "full resolution" format, as well as the marks delimiting at least one area of interest.

-c) “waterfall” format (waterfall, that is to say horizontal or vertical scrolling of the images) intended for the pilot's display device. As with the "full field" format, the formatting device 9 performs a subsampling of the useful video and embeds the same information in the digital video. In addition, the device 9 formats the video, in a manner known per se, to allow viewing thereof on board the aircraft according to a scrolling in "waterfall". The management processor 13 receives commands from the ground station 6A to activate either the reading in “full field” format or the reading in “full resolution” format of the memory 4, and / or commands determining the start addresses and the end of the video sequence to be transmitted via the Data Link. These addresses generally correspond to said marks embedded in the video on board the aircraft to delimit an area of interest.

From the device 12, the management processor receives a “report” of the operation of the Data Link (signaling it, for example, transmission interruptions, transmission errors, etc.).

On board aircraft, the processor 13 receives the different flight parameters (line of sight, time, date, altitude, etc.) so that the device 9 can embed them in the images intended to be used on the ground. In addition, the processor 13 receives from an appropriate switch, manipulated by the pilot (or by another person on board the aircraft) marking signals to be embedded in the stored images to delimit areas of interest, and it therefore controls memory 4.

The processor 13 controls the memory 4 to write the useful video therein with the above-mentioned embedded data, and to play it back-to-back or in real time. The information read in the memory 4 is, on the one hand sent by the Data Link to the station 6A, and, on the other hand, directly to the on-board display device 14. Those passing through the Data Link are only sent if this link is free and undisturbed (or not too disturbed, to allow transmission at the minimum possible rate depending on the amount of video to be transmitted, which can be relatively low, for example 10 Mbits / s or even less) , and if the pilot authorizes it. If so, the commands transmitted to the memory 4 include in particular the choice of the video format to be played ("full field" or "full resolution"), the start and end addresses of sequences to be transmitted. In the event of the Data Link link being cut (or of too poor quality), the processor 13 sends to the memory 4 read stop commands; then resume reading as soon as possible.

When the data read from memory 4 is intended for on-board display, the processor 13 sends it mode choice commands (real time or deferred) and the start and end addresses of the quences to view. In addition, the processor 13 can send, if necessary, commands to the sensor 3, for example if the latter comprises several sub-assemblies, in order to activate them all, or a part of them. The device 12, like its counterpart 15 of the station 6A, performs several functions.

On the one hand, it manages the transmission antennas (pointing, hanging, dropping, hanging up). On the other hand, it performs the coding of the signals to be transmitted and the decoding of the signals received, by implementing appropriate error correction processes. FIG. 3 shows the block diagram of the circuits of the ground station 6A. It includes a Data Link device 15, homologous to the device 12 in FIG. 2. The output of the device 15 is connected by a decryption device 16 (complementary to the encryption device 11) to a deformatting device 17 (complementary to the formatting 9), itself connected to a device 18 for decompression without loss of data, its output being connected to a device 19 for analyzing and using data. The device 17 is also connected to a mission rendering reader 20 (suitable for reading the memory 4 when the aircraft returns to the base). The device 19 is connected to a reader / recorder 21 of information coming from the Data Link and to an archiving recorder / reader 21 A (for example of the tape type). The device 19 is also connected to the device 15 (to control the reading of areas of interest from the memory 4 via the Data Link and to receive a report on the operation of the link), to the device 17 (to send it commands), to the reader 20, to the readers / recorders 21 and 21 A (to send them commands and receive operating reports) and to display screens 19A.

The functions of the formatting-deformatting device 17 are as follows. It decompresses, without loss of data, the compressed video arriving by the downlink of the Data Link, and that coming from the player 20. It plays an interface role on the one hand between the player 20 and the station 19 to provide the latter with the useful video stored in the memory 4, on the other hand between the station 19 and the recorder 21, to save the images or sequences of images used (with marking of the zones or sites identified, comments and, if necessary, adjustments to the deo, relating to these images, and finally between the player 20 and the recorder 21 for saving all of the video relating to the mission. Station 19 performs the following functions:

It supervises the device 15 to read the report on the operation of the link and to send read commands from the memory 4 (it sends it start and end addresses of the sequences selected by the operator on the ground,

• it controls the recording of the useful video and of the data coming from the Data Link or from the 21 A archiving recorder on a work support suitable for analysis, such as a buffer memory,

• it controls the reader 20 and the readers / recorders 21 and 21 A and manages their reports,

• it formats the useful video of said working medium (buffer memory), which is advantageously of the type for simultaneous writing and reading. This formatting essentially consists of sorting the images in chronological order of acquisition in flight, oversampling them (to increase the signal / noise ratio) and converting the video thus treated from the "full field" format to the " waterfall ",

• it provides display on at least one screen 19A either in "full field" format or in "full resolution" format. The "full field" format corresponds to video sub-sampled either during recording by the recorder 21 or on board the aircraft (by the trainer 9) for transmission to the ground. The “full field” display is done in continuous scrolling (waterfall), and the selected areas of interest are displayed in “full resolution”. This “full resolution” format also refers to the video transmitted via the Data Link. The display screens 19A also display the information embedded in the video by the formatting device 9. The display device controlling the screens 19A also performs other conventional functions such as stopping and resuming the scrolling of the images (rotation, mirror, zoom ...), control of image transcoding (for example if the pixels are coded by sensor 3 in 12 bits, transcoding on 8 bits, maximum generally accepted for the resolution of the eye, i.e. 256 levels of gray), calculation and adjustment of contrast and brightness (dynamically) of the images to be displayed, histogram of the images displayed (for a possible gamma correction), corrected tion of the length / width ratio of the images to be displayed, to compensate for variations in aircraft speed and distance to the ground,

• printing of the selected images when viewing the sensor images, • automatic location of the shots on a digitized card, simultaneously with the viewing

• creation of an operating file concerning: the overlay on the visualized images of comments and specific annotations of the interpreter, specific graphics, the superimposition of the images of the sensor ("full field" and "full resolution" formats) ), superimposition of the images of the sensor and of a corresponding map (or geographic landmarks) ...

• automatic extraction of known objects (road, river, buildings, etc.) using known routines to assist in the detection, recognition and identification of these objects,

• automatic contour detection,

• 3D operation

• distribution of the operating result on an appropriate network. According to a variant of the invention, several Data Link transmission channels are used in parallel to reduce the transmission error rate and / or to allow operation on the ground by several operators.

FIG. 4 shows the different stages of the processing of the images captured by the sensor 3 and sent to the ground. This sensor 3 transmits useful digital video signals (or digitized) to the formatting device 9, which performs compression (by a factor of 2) in real time. The information thus compressed is recorded in memory 4. The latter is read simultaneously (or slightly delayed) in “full field” or “full resolution” format (at the option of the operator on board or on the ground) as indicated in 22 The selected format is coded and encrypted in real time (in 1 1), transmitted to the ground by the Data Link. This link can either be recognized as good, and there is then continuation of the airplane by the antennas on the ground (at 23), or be cut or of too poor quality, and there is then “dropout” and search for plane (in 24). Advantageously, an indicator (15A) indicates the state of the link (established or interrupted). The information sent from the aircraft to the ground base arrives there or not (which is symbolized by a switch 25). If the connection is good, the information arriving on the ground is, on the one hand, recorded by the recorder 21 (with the indication of the identity of the area recorded, to guide the operator, as noted in 26). On the other hand, as indicated by switch 27, this video information or else that available on the ground is chosen by the operator for further processing. As symbolized at 28, said information available on the ground is either that read at 21, or that recorded in memory 4 and read by the reader 20 after returning to the ground of the aircraft, then archived at 21 A.

The processing, downstream of 27, consists of a real-time shaping (in 17) of the video chosen (in 27 and 28) by the operator, then in a "full field" or "full resolution" display ( operator choice in 29). For the “full resolution” option (30), the operator displays either the video arriving directly via the Data Link, or that having been recorded on the ground (choice 31). The display includes the indication of the identity of the selected area of interest, this indication being either that embedded in the video coming directly from the aircraft (32), or that (33) of the area selected from those recorded. on the ground in 21.

For the "full field" option (34), with display in "waterfall" mode, the operator can order a freeze frame and / or select an area of interest (35). If he selects an area of interest, he can order (36) the transmission, via the Data Link, of the corresponding video, in "full resolution" format, stored in memory 4, and view it on one at least 19A screens in "full resolution" format (37). Of course, as long as there is no selection (at 35) of area of interest, the “waterfall” display continues (arrow 38). Similarly, after transmission of the area of interest (36) and display (37), the display in "waterfall" can continue, if it is not completed (arrow 39). In step 40, said display device controlling the screens 19A performs the aforementioned video manipulations (rotation, mirror, zoom, transcoding, etc.).

Claims

1. Method for real-time exploitation of aerial shots obtained by a sensor (3) with high information rate and applied to a mobile (1) connected to an operating base by at least one communication link “Data Link” type (12, 15), this mobile being provided with means for storing the video thus obtained, these storing means being able to be read during their recording, characterized in that during the recording of the video, reference is made areas of interest among the shots taken, these markers being recorded with the corresponding video, and only the video corresponding to the areas of interest is read and transmitted to the base via the Data Link.

2. Method according to claim 1, characterized in that the location of the areas of interest is controlled on board the mobile.

3. Method according to claim 1, characterized in that the location of the areas of interest is controlled from the operating base.

4. Method according to one of the preceding claims, characterized in that at least one of the following information is embedded in the video: the position of the line of sight, the size of the sensor, time and date of the shots of view, mobile altitude, weather information.

5. Method according to one of the preceding claims, characterized in that the video is recorded on board the mobile in three formats: "full resolution", "full field" and "waterfall".

6. Method according to one of the preceding claims, characterized in that at least one part of the video information recorded in the storage means is displayed on board the mobile (14).

7. Method according to one of the preceding claims, characterized in that at the operating base the information coming from is recorded of the "Data Link" link, those coming directly from the storage means after the return of the mobile to the base, and the latter after operation.

8. Method according to one of the preceding claims, characterized in that at the operating base at least one of the following operations is carried out when viewing the video: rotation, mirror, zoom, mid-sampling or video subsampling, transcoding of the number of gray levels, contrast and brightness adjustment, histogram, correction of the length / width ratio of the images.

9. Device for real-time operation of aerial shots obtained by a high-speed sensor (3) arranged on a mobile (1) connected to an operating base (6A) by a “Data Link” type link (12, 15), this mobile being provided with means (4) for storing the video thus obtained, these storing means being able to be read during their recording, characterized in that the "Data Link" link has a lower transmission rate to that of the sensor, and that the mobile comprises means (13) making it possible to add to the recorded video marks at the start and end of identified sequences to be transmitted to the base.

10. Device according to claim 9, characterized in that the operating base comprises a recording / reading device (21) of information coming from the "Data Link" and a recording / reading device (21 A) d archiving.