CA2125788A1

CA2125788A1 - Buffer and frame indexing

Info

Publication number: CA2125788A1
Application number: CA002125788A
Authority: CA
Inventors: Eric Charles Peters
Original assignee: Individual
Current assignee: Avid Technology Inc
Priority date: 1991-12-13
Filing date: 1992-12-10
Publication date: 1993-06-24
Also published as: JP2987206B2; US5640601A; JPH07505001A; AU3274493A; WO1993012481A2; WO1993012481A3; US5513375A

Abstract

A data buffer that compensates the differences in data rates, between a storage device and an image compression processor. A
method and apparatus for the real time indexing of frames in a video data sequence.

Description

~VO g3/12481 2 1 2 ~ 7 8 8 PCr/US92tlO643 BU~FER AND FR~ INDE2~NG

Back~round othe In ention This invention relates to hardware desigIIs coupled with sof~ware-based algorithms for capture, compression, decompression, and playback of digital image sequences, particularly in n editing enviroIlment.
The idea of taking motion video, digitizing it, compressiDg the digital datastream, and storing it on some kind of media for later plavback is not new. RCA's Sarnoff labs began working on this ~n the early days of the video disk, seeking to create a digital rather than an aIlalog approach. ;
..... ... .
This techIIology has since become known as Digital Video I~teractive (DVI). -Anoth.er group, led by Phillips in Europe, has also worked on a digital motion video approach for a product they call CDI (ComLpact Disk `~
Interactive). Both DVI and CDI seel~ to store motion ndeo aIld sound on CD-ROM disks for playbacl~ in low cost players. In the case of DVI, the compressio~ is done in batch mode, as~d take~ a long time, but the ~ ~ -20 playbac3~ hardware is low cost. CDI is less specific about the compression approach, a~d ms~ ly provides a format for the data to be stored on the :
, di~k.
A few years ago, a standards~making body k~own as CCIIT, ba~ed in France, working in conju~ction ~nth ISO, the Intnational Standards 25 Organization, created a worl~ing group to focua on image compresshn.
Thi9 group,~ called t~e Joint Photographic E~cperts Group (JPEG) met for ma~y years to determine the mqst ef~ecti~e way to compres~ digital images. They evaluated a wide raIlge of compression schemes. includ~g vector qu ntization (the tech~lique used by DVI) aIId DCT (Discrete Cos~e 30 Transform?. After e~austive qualitati~ tests and carefi~l study, the JPEG group picked the D(~T approach, a~d also defined in detail the ;;
vanous ways this approaach could be used for jm~ge compression. 1he ; ~
. .~. .

~': ::'~
~, :, . .. .

wo 93/l2481 2 12 5 7 8 8 PCI-/US92/10643 group published a proposed ISO standard that is generally referred to as the JPEG standard. This standard is now in its final form, and is awaiting ratification by ISO, which is expected.
The JPEG standard has wide implications for image capture and storage, ~mage tra~sm~ission, and image playback. A color photograph can -be compressed by 10 to 1 with virtually no ~isible loss of quality.
Compression of 30 to 1 ca~ be achieved with loss that is so minimal that ~ ' most people cannot see the difference. Compression factors of 100 to 1 and ,-more can be achieved while maint ining image quality acceptable for a wide range of purpo~es. , The creation of the JPEG standard has spu~Ted a variety of important hardware developments. The DCT algorithm used by the .IPEG
standard is ea~tremely,comple2~. It requires converting an image from the spatial dom,ain to the frequency domain, the quantization of the various ', frequency compoIlerlts, followed by Hu~an coding of the res~lting components. ~he convercio~ from spatial to ~equency domain, the qu~rltizatio~l, and the Huf~an coding are 11 computationally intensi~e.
Ha~dware vendors have responded by building specialized integrated circuits to implement the JPEG algonthm.
One vendor, C~Cube of Sa~ Jose, California, has createt a JPEG
chip (the CL550B) t~at not only implements the JPEG standard i~
hardware, but can process n im~ge ~with a resolution of, for e~cample, 720 ~c 488 pixels (CCIRR 601 video standard) in just V30th of a seco~d. Th'is means that the JPEG algorithm can be applied to a digitized ~ndeo sequence,~a~d the restllting compres~ed data can be stored for later playbacl~. The same c~ip can be used to compre~s or decompress images or image sequences. The availability of thi~ JPEG chip has spurred computer' vendors a~d system integrators to design new products that incorporate - --the JPEG chip for tion video. However, the implementation of the chip 30 in a hardware and software en~ironm~nt capable of processing images

2~2~7~8 with a resolution of 640 x 480 pixels or greater at a rate of 30 ~rames ~er second in an editing er.vironment introduces multiple ~roblems. . - :-For high auality images, a data size of 15-40 Kbytes per rame is needed for images at 720 x 488 resolution. T~.is means that 30 frames per second video will have a data rate of 450 ~o 1200 Kby~es per second. For data coming from a dis~ storage ~. ~M
devlce, this is a high da~a rate, requiring careful atterl~lon :`.
to insure a working syst~m.
The most common approach in prior systems for sending data from a disk to a compression processor is to copy of the data . :
from disk int~ the memory of the host computer, and then to .~:
send the data to the compression processor. Tn this met~od, ~he computer memory acts as a buffer agains~ the di~lerent data ~ .
ra~es o he compression ~rocessor and the disX. T~is scheme :~as two drawbacks.; ~irst, the data is moved twice, once ~om .-~
tAe disk to the host memory, and another t;me from the hos.
memory to .he compression processor. For a data rate of 1200 .
Kbytes per second, ~his can seriously tax the~host compu.e~
allowing it to do little else but the data copying.
Furthermore, the Macintosh computer, for exampie, canno~ read ~.
da~a from the dlsk and co~y data to ~he compression pr~ocessor at Ihe same time. The ~resent invention ~rovides a com~ressed data buf~fer speciflcally designe~d so that data can be sent : ::
direc~ly rrom ~he disi~ to the buffer. .
~ Also dis~closed in the prior ar. reference EP~ 0 alo 382 is a data ~ransfer cont.oller for performing;a da~ta ~ ar.s er .~
belween 2: memory and a ~er-pheral unit, uslng a direc memcry .~.
access (~M~) me~hod. T~e daia trar.s~er con~rolier ~e~fo.ms a da~a transfer 'rom the peripheral uni. to a memory wr.erein~.ne peri~heral~uni~ operates as a serial data communlca~lon c~n~~ol uni~ ro~an ex-~ernal~Iy ~-ovided device, such.as key-~oa d ~c a .

-3~

--212S788 ~ ~

miorocom~uter. A CPU, the controller, the ~eri~heral unit, and the memcry are all interconnected to each other via an internal bus of a microcomputer. The data from the externally provided device is received by the peripheral unit and transferred to the memory by the con~roller. The memory includes -a program memory ~or storing a program to be executed by ~he CPU and a data memory for temporarily storing data. The data memory .
includes a DMA transLer destination area into which the data are transferred from the peripneral unit by a DMA transLer. - ~.
The da~a transfe- controller, according to the European reference ~P-.~0 410 382, comprises a first resister for storing adcress i~formation relative to a predetermined address of the D~ ~xans.er area, a second register for storing a numbe~ of data ~o be tra.nsferred, a D~ control uni~ for ~erforming a ..
data t ansfer between the memory and the ~eripheral unir by use~
of tne first and second registers, a tnird register for s.or ng da a used for:~access;ng the DMA trans.er area of the memory:, ar u~date- for u~dating~he data stored in:the thlrd register each time a memory acces~s is performed, and a counter unit for ~-changing the conten~ ~hereof each time the data trar.sfe_ is erformed between the memory and the peripneral unit.
ruropean reference r~P-A-O 185 924 ciscloses a bu~-~er system for detecting.errors caused by failures in 2 re`ad and/or write circui~s. The buffer system of the in~en~ion nc'udes:a ~:
s~orage~array that is addressable for read and wrlte operatio~s ;~ :.
by an:address of n-bit~s that are:su~pliec by a read add~ess :
counter~and~:a~write address counte.r, each having n + 1 ~its. .
3uring:~a write opera ioni the (~n + l).h ~i. of the~write a~dress coun~é~ is-~stored:as a pa t~of a pari~y bi~ fo the address :-:
arr~ay location.~ Durlng a:read operation the (n . l)th ~il OL ~ .
;tne rea~ address;~:coun er enters a ~arity chec~,~g function~ OL~
he word :read from the~ addressable .~

--3 2-- :

.,-,.",.,.,., ~,..
...
location. ,~n error 's signaled i the (n + l)th bit of the read address counter does not agree wi~A the (n + l)th bit of '.~
the wri~e counter at ~he time of the write operation. Thus, ~ :' the buffer sys.em ~revents reading the same entries on .he . -:,'.-.
second pass through the memory array. . .. ..
W.i~h ~ne JPEG algorithm, as with many com~ression ,~
a'gori~hms" the amount of data .hat results fr~m compressing an ;: ,.~
image depends on the image itself. An image of a lone sea gull '' ~
against a blue sky will take much less data than a ci~yscape of .. -;.
brick ~uildings wi=h lots of detail. There-rore, it becomes : ,. ';
difficult to know where a frame starts within a data 'ile that ~,.,-.
contains a sequence of frames, such as a digitized and compressed sequence of video. This creates particular problems .... ~.-'.`.'.. ;.':
in the ~laybac~ -' om many files based on edi. decisions. With .
fixea size compressi~n approaches, one can slmply index directiy:~ln~o ~ne -'lle by:multlplyl g ~he ~ , ,,'~
~ . - . .

,,., . ~
'',',' ~,'''.""", ,..:

: 2063 ::~ : ' ~3b~
. ., :

212~78~ `0 93/12481 PCI`/US92/1~643 frame number by the frame size, which results in the offset needed to start reading the desired ~ame. When the frame size ~aries, thi~ simple multiplication approach ~o longer works. One needs to have an inde~{ that stores the offset for each frame. Creating this inde~c can be time 5 consuming. The present invention pro~rides an effiaent mde.Ying method.

Sum~e Illvention The data buf~er of the i~ention compe~sates for the data rate difEerences between a storage dev~ce and the data compression processor of 10 a digital Lmage compression and playback urlit. The data buffer interfaces to a host central processing llnit, a storage device, a DMA address register, a~d a DMA li~nit regi~ter, and is mapped into the address space of the host computer bus. The data sequence is unloaded from the storage dev~ce into the data buffer, which is t~ice mapped into the address space of the 15 host computer.

Brief Descri~tion of the Drawin~
~ig. 1 is a block diagram of a ndeo image capture and playbac3 system implementi~lg data compression;
Fig~ 2 is a schematic diagram of a compressed tata bnffer according to one embodiment of the invention; snd Fig. 3 is a schematic illustration of an edited sequence of images along with two mapp~g schemes of the compressed data buf~er ~ the host system's bu DescriDtion o the PrefeITed Embodiment :
A bloc~c diagram according to a preferred embodiment of a s~stem for capture, compression, storage, decompression, and playback of images is illustrated in Fig. I.
~, "

: . '~ ~ '' . ~: : -:

~vn 93/12481 2 1 2 5 7 8 ~ PCT/US92/10643 - -As shown, an image digitizer (frame grabber) 10, captures and digiti~:es the images from an analog source, such a~. videotape. Image `~
digitizer 10 may be, for example, a TnleVision NuVista+ board. However, :;
the NuVista+ board is preferably modified and augrnented with a pi~el 5 engine as described ''Image Digitizer Including PLxel Engine" by B. Joshua Rosen et al., filed December 13, 1991, to provide better data throughput for a variety of image formats and modes of operation. Other methods of ~;
acquiring digitized video frames may be used, i.g., direct capture of digital ~ ;:
video in "D-1" or D-2" digital v~deo formats. ; ~ ~ ;
A compression processor 12 compresses l;he data acco~ding to a ~;
compressio~ algorithm. Preferably, this algorithm is the JP~:G algolithrn, i~ltroduced a1Dove. As discussed above, C^Cube produces a compression processor tCL550~3) based on the JPEG algorithm that is appropriate for ;
use as the compression processor 12. However, other em~odiments are within the scope of the i~vention. The compression processor 12 may be a ~ ~ -proces~or that impleme~ts the new MPEG tMotion Picture Ex,Derts Group) algorithm, or a processor that impleme~ts any of a vanety of other image ~ -c~Dmpression algorit~ms known to those skilled in the art. :
The compressed data from the processor 12 ~ preferably input to a 20 compressed data buffer 14 which is interfaced to a host computer 16 ;
connected to a disk 18. The compressed data buffer 14 preferably ;
implements a DMA process in order to absorb speed differences betwee~
the compression processor 12 and the disl~ 18, and further to perm~t data -trallsfer between the processor 12 and the diqk 18 wit~ a s~gle pass 25 through a CPU of the host cor~puter 16. (The details of the compressed data buf~er 14 according to the present invention will be presented .hereinbelow.) The host computer 16~ may be, for e~cample, an Apple Macintosh.

Buffer ;

, ~', .'`~, WO 93/12481 2 1 2 ~ 7 ~ 8 Pcr/U~92/10643 As discussed above, a compressed data buffer is provided to tal~e up the data rate differences between the disk 18 and the data compression processor 12. In this way, data can be sent directly ~rom the disk to the buffer, or vice ~ersa, pa~si~g through the host CPU only once. Orle thus 5 aviods copying the data from the compression hardware into the host's mai~ memory before it can be written from there to the disk storage subsystem. This scheme cuts the CPU overhead in half, dou~ling data throughput.
A detailed schematic diagr~m of the storage end of the system of ~-10 Fig. 1 is shown in Fig. 2. The compressed data buf~er 14 is addressable.
Associated with the bu~er 14 are a D~A address register 20 and a DMA
limit register 22. These regi~ter~ and the buffer ~cre seen by a CPU bus 24 of the host computer 16. Because the buffer 14 is addressable, standard file system callq can be used to request that the host computer 16 read 15 data from the disk 18 aIld seud it to the buf~er 14, or read data from the buffer 14 and send it to the disk 18. The buf~er 14 looks to the computer ~`
16 like an e~ctension of its own memory. No changes to the host computer disk read or write routines are requ~ed. For example, a si gle call to the operating system 16 of the host computer specif~ring a buf~er pointer, a .
ao length to read, and a destination of the disk will ef~ect a direct transfer of ;~ ;
data from the buffer to the disk. By looking at the DMA address at the JPEG buf~er, one can tell when the data is ready. By setting the DMA
limit, feedback throttIe~ the JPEG proces or filling the buf~er. -According to the i~Yention, the buf~er 14 i~ mapped in an address space of the host computer's bus 24 twice. Thus, t~e buffer i9 accessible in -two contiguous locations. l'hi~ has important ramifications in an edit~ng erl~nronrnent d~nng playbacl~
Fig. 3 shows n edited sequence of images and a representatio~ of a buffer that is mapped to the address space of the host computer's bus only once. Ihe sequence is longer than the buffer. Each edit point in the ".

;

Wl~ 93/12481 212 ~ 7 8 8 PCr/US92/10643 sequence represents a point at which the data must be picked up at a new - ;-place on the disk.
During playback, the sequence will be read iIltO the buffer from left -to right, and the buffer w~ll empty from leflc to right as the images are S played. In the e~ample illustrated, segments a, b, c and d fit into the buffer. Segment e does not however. For the buffer show~, therefiore, two -reads will be required to tra~sfer segment e, since part of e will go at the end of the buffer, and the rest will go at the beg~n~ing of the buffer, as the beginning empties dunng playback. It is desirable to !in~it the number of ;~
10 reads as much as possible, as reads reduce the throughput of the system.
The longer the reads, the more efficient the system.
This problem can be largely eliminated by mapping the buffer into the address space of the host computer's bus twice. As ilUustrated in Fig.
3, segment e now fits icn contiguous memory in the buf~er by overflowing 15 into the second mapping. I~ this example, thea, the double-mappillg ~as allowed a siIlgle read, where two reads would ha~e been required before.
In gerleral, for every read, you can read as m~ch a~ is empty in the buffer.
The space i~ the second mapping is only temporarily borrowed. I~
practice, the scheme is implemented by mal~i~g the address of the second 20 mapping the same as t~e address of the first except for a single bit, arld byhaving the hardware of the system ignore t~is bit. So ~hether data is ; ~ ;
wntten to the first mapping or the second, it goes to the same place in the buf~er.
Thi9 double mappi~g solves an mportant problem i~ a way that 25 would rlot be possible without the bu~er, since the computer's memory itself cannot i~ ge~eral be remapped to mimic the tec~ique.
' Frame Illde~in~
, For any data compressiosl scheme that results in compressed images with vana~le fr me size, a method of frame inde~ing i~ required for ;' :.'''.'' "` '~ '' ,~ ' '' ~."-'.~..

~ .~

wo 93/12481 2 1 2 ~i 7 ~ g PCI`/US92/10643~. .
finding frames to put together ~n edited sequence. Ihe location of any frame is preferably instantly available.
The C-Cube chip descnbed above prondes a mechanism for creating an index by allowing the user to specify that a mar~s:er code be placed at a specified location in every frame. Therefore, a marker code can be placed at the beginning or end of every ~rame. II1 prior approacheq, a program ~-has been w~itten to seguentially scan the file containing a sequence of images on a disk, and fi~ld ~nd remember the location of each marker code.
This is a post processing approach and is time consuming.
According to the ~rame inde~ng method of the invention, the image digitizer is programmed to generate an interrupt to the CPU of the host computer at every frame.l As the compression processor is putting data iII the compressed data buffer, each time the CPU detects an ~ntenupt it notes the loc:ation of the pointer in the bu~er. By keeping track of the 15 nurnber of times the pointer has been through the memory, and the n~rnber of bytes the pointer is into the memory at each interrupt, the CPU
~an keep a table in memory of the position, or more preferably, the length ;
of each fra~e. This table can be dumped to the disk at the end of the file, thereby pro~riding the location of every *ame in the file.
The table of ~ame locations does not solve all problem~, however. ~ :
Retrieving this irlformation as needed du~ng playback of an editted sequence i9 prohibi~ively t~me consuming. The solution ~s to make ollly that informatioll necessary for a gi~en edited sequence available to the -CPU. T~e required iDf rmahon is the be nning ant end of each segment , of the sequen~e.
According to the invention, a data structure representing an edited séquence is generated at human ~teraction time dunng the editing ! ' ;; ~
~ ' `' ,' : ~; :, ~, other prior approach is to use a fast processor or special purpose hardware to -recognize ~nd record the position of tile marker code on the fly.

'"~

4~1 2 1 2 5 7 8 8 PCr/~S92/10643 ~ -process. Each time a user mar~s an edit point, an item is added to the list. By including ~n the list two fields representing the locations of the begin~ing of f;rst and end of last frarnes in a segment, this information will be readily available at playback time. Since this prefetchi~g of inde~

5 value~ occurs during htLman interaction time, it does not create a bottleneck in the system. ;
The CPU can also be alerted whe~ever the frame sizes are gett~ng too large for the system to handle. Compensating mechanisms can be triggered into action. One example of such a mechanism is the quality 10 adjustment method disclosed in copending application "Quantization Table ;
Adjustment" by Eric C. Peters filed December 13, 1991. Thi9 adjustment `
reduces frame size (at the expense of quality). `
It ~1vill be clear to those skilled in the art that a buffer according to the invention can be simply designed using prog~an~mable ~ray logic and -15 memory chips.
What is cla~ed is~

: ....
:'. ' `

:. -,, :. .
-~ ~.,. -, ....

.

9 ,~

' , ~

Claims

1. A system for compensating for data rate differences between a storage device (18) and a data compression processor (12) characterized in that:
a host control processing unit (16) including a bus (24) having an address space and a central processing unit (19);
an addressable compressed data buffer (14) including a first port (13) and a second port (15), the first port (13) being coupled to the data compression processor (12) and the second port (15) being coupled to the host control processing unit (16);
a direct memory access address register (20), connected to the first port (13) of the addressable compressed data buffer (14), which stores a data address or a location within the data buffer (14);
a direct memory access limit register (22), connected to the first port the addressable compressed data buffer (14), which limits a direct memory access transfer of data between the compression processor (12) and the compressed data buffer (14);
an interface (17) linking the second port (15) of the addressable compressed data buffer (14) and the host control processing unit (16); and wherein the addressable data buffer (14) is twice mapped into the address space of the bus (24) such that a data transfer between the storage device (18) and the compressed data buffer (14) is accomplished by passing the data through the central processing unit (19) only once.

2. The system set forth in claim 1, where the addressable compressed data buffer (14) is configured to store digital video image data.

3. The system set forth in claim 2, where the compression processor (12) is configured to compress digital video image data.

4. The system set forth in claim 2, where the compression processor (12) is configured to compress digital video image data by the steps of:
computing a discrete transform of the image data to create a plurality of coefficients corresponding to frequencies, quantizing the plurality of coefficients to create a plurality of quantized coefficients; and coding the quantized coefficients to create a plurality of encoded coefficients.

5. A method for compensating for data rate differences between a storage device (18) and a data compression processor (12) and for transferring data between the storage device (18) and the data compression processor (12), the method characterized in the steps of:
providing an addressable data buffer (14) having a first port (13) and a second port (15), the first port (13) being coupled to the data compression processor (12), a host processing unit (16) couple to and between the storage device (18) and the second port (15) of addressable data buffer (14), wherein the host processing unit includes a bus (24) having an address space;
mapping the data buffer (14) twice into the address space of the bus (24) to provide a contiguous buffer memory; and transferring data between the compressed data buffer (14) and the storage device (18) by passing the data through the host processing unit (16) only once.

-11a-

6. A method for indexing frames in a sequence of compressed images with variable frame sizes, comprising the steps of:
providing an addressable data buffer and a host computer including a central processing unit and a local memory;
configuring a data digitizer to signal the central processing unit at every frame of the compressed images;
transferring the compressed images from the digitizer to the data buffer;
at each signal from the data digitizer detecting a location of the frame in the data buffer; and storing the frame location in the local memory, thereby providing a record of the location of every frame of the sequence of compressed images.

7. A method as claimed in claim 5, further comprising the step of direct memory access transferring the data from the data buffer (14) to the compression processor (12).

8. A method as claimed in claim 5, wherein the step of transferring further comprises the step of direct memory access transferring the data from the compression processor (12) to the data buffer (14).

9. A system for compensating for data rate differences between a storage device (18) and a data compression processor (12) characterized in that:
a host control processing unit (16) including a bus (24) having an address space and a central processing unit (19);
an addressable compressed data buffer (14) including a first port (13) and a second port (15), the first port (13) being coupled to the data compression processor (12) and the second port (15) being coupled to the control processing unit (16);

-11b-a direct memory access address register (20), connected lo the first port (13) or the addressable compressed data buffer (14), which stores a data address of a location within the data buffer (14);
a direct memory access limit register (22), connected to the first port (13) of the data buffer (14), which limits a direct memory access transfer of data between the compression processor (12) and the compressed data buffer (14); and wherein the data buffer (14) is twice mapped into the address space of the bus (24) such that a data transfer between the storage device (18) and the compressed data buffer (14) is accomplished by passing the data through the central processing unit (19) only once, without storing the data in a local memory (17) of the host computer.

-11c-